Merge feature/rust-r5-pipeline-core into develop
ci / rust (push) Canceled after 0s
ci / check (push) Canceled after 0s
ci / docker (push) Canceled after 0s
ci / supabase (push) Canceled after 0s
ci / rust (push) Canceled after 0s
ci / check (push) Canceled after 0s
ci / docker (push) Canceled after 0s
ci / supabase (push) Canceled after 0s
This commit is contained in:
commit
9ba2a250f4
291 files changed
+60814
-14686
No files matched your search
@@ -0,0 +1,157 @@
|
||||
# Security gate: Rust R5.3 draft pipeline (2026-10-22)
|
||||
|
||||
Mode: audit (delta list). Branch `feature/rust-r5-pipeline-core`, uncommitted working tree.
|
||||
Scope: the R5.3 change set (8 modified files, 17 new files), read whole; `--check` run once
|
||||
(read-only, exit 0, seed folder unchanged). No code was edited.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 25 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] symlink check covers the final component only — crates/core/src/bin/import_equine.rs:182-203
|
||||
- [LOW] check-then-write is not atomic (TOCTOU) — crates/core/src/bin/import_equine.rs:213-228
|
||||
- [LOW] serde_json::Map indexed with [] (panics on a missing key) — crates/core/src/bin/import_equine.rs:111
|
||||
- [LOW] INIT_CWD trusted from the environment — crates/core/src/bin/import_equine.rs:206-211
|
||||
- [LOW] seed SQL assembled by string formatting — crates/core/src/curation/draft/seed.rs:28-29,91-101
|
||||
- [LOW] new test file carries the suite's clippy allow line (convention, not a scanner suppression) — crates/core/tests/curation_draft.rs:7
|
||||
- [MEDIUM] guardrails SB-POLICY-NOT-OWNER, outside the R5.3 delta, pre-existing — supabase/migrations/20261008110000_anatomical_sheets.sql:1177
|
||||
PROOF: semgrep 725 rules on 25 files → 0 findings; checklist 9 checks → 6 findings (all LOW); guardrails scan.py --domain api,supabase → 0 FAIL, 1 WARN (out of delta), 1 MANUAL; anti-gaming → 0 new un-gated suppressions
|
||||
```
|
||||
|
||||
## Files scanned (25)
|
||||
|
||||
Modified: `crates/core/src/bin/import_equine.rs`, `crates/core/src/curation/mod.rs`,
|
||||
`crates/core/Cargo.toml`, `package.json`, `CLAUDE.md`, `CHANGELOG.md`,
|
||||
`docs/development/RUST.md`, `.claude/tasks/TODO.md`.
|
||||
New: `crates/core/src/curation/draft/{mod,schema,targets,fields,ops,rename,apply,records,migration_map,pipeline,seed,stats}.rs`,
|
||||
`crates/core/tests/curation_draft.rs`, `crates/core/tests/oracles/r5/curation-synthetic.json`,
|
||||
`packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`, `scripts/import-equine.sh`,
|
||||
`.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`.
|
||||
|
||||
## Findings in detail
|
||||
|
||||
### 1. [LOW] Symlink check covers the final component only
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:182-203`. `writable_destination` calls
|
||||
`fs::symlink_metadata(out)` and refuses when `out` itself is a symbolic link. An ancestor
|
||||
of `out` that is a symlink is not resolved: `--out <typed-parent>/link/new` with
|
||||
`link -> <elsewhere>` creates `new/legacy-import` and `new/draft` under the real target of
|
||||
`link`, and the success line prints the typed path. Because the destination must be absent
|
||||
or an empty folder, nothing existing can be overwritten or deleted; the effect is new files
|
||||
in a folder other than the one typed. Fix direction: `fs::canonicalize` the nearest existing
|
||||
ancestor of `out` and refuse when it differs from the typed ancestor, or at least print the
|
||||
canonical destination in the success line.
|
||||
|
||||
### 2. [LOW] Check-then-write is not atomic
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:213-228`. `write` validates with
|
||||
`writable_destination`, then runs `fs::create_dir_all` and `fs::write` per file. A process
|
||||
that swaps `out` (or a created subfolder) for a symbolic link between the check and the
|
||||
writes would have the writes follow it. A development tool on the owner's machine has no
|
||||
such adversary; reported for completeness since the brief names TOCTOU. Fix direction:
|
||||
when `out` is absent, claim it with `fs::create_dir(out)` (fails if something appears) and
|
||||
re-run `symlink_metadata` on the result before writing; a `std::fs::DirBuilder` without
|
||||
`recursive` for the two folder levels achieves the same.
|
||||
|
||||
### 3. [LOW] `serde_json::Map` indexed with `[]`
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:111`:
|
||||
`result.view_boxes[HEAD_VIEW_ID].as_str()`. `Index` on `serde_json::Map` panics on a
|
||||
missing key (unlike `Index` on `Value`). The legacy importer inserts both `main` and
|
||||
`head` unconditionally (`crates/core/src/curation/legacy/importer.rs:365-370`), so the
|
||||
panic is unreachable today; the same pattern was replaced by `get` in commit 6afc1e0 after
|
||||
a previous gate. Same line in `crates/core/tests/curation_draft.rs:72` (test code, allowed).
|
||||
Fix direction: `result.view_boxes.get(HEAD_VIEW_ID).and_then(Value::as_str)` with a
|
||||
typed error.
|
||||
|
||||
### 4. [LOW] `INIT_CWD` trusted from the environment
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:206-211`. `base_dir` takes `INIT_CWD` as the base
|
||||
of a relative `--out` without checking it is absolute or exists. pnpm sets it to the
|
||||
invocation folder, which is the intent; a stale or foreign value exported in the shell
|
||||
silently retargets a relative `--out`. Mitigations already present: the destination must
|
||||
be absent or empty, the resolved path is printed, an absolute `--out` ignores the base
|
||||
(`PathBuf::join` replaces it). Fix direction: accept `INIT_CWD` only when absolute, else
|
||||
fall back to `current_dir`.
|
||||
|
||||
### 5. [LOW] Seed SQL assembled by string formatting
|
||||
|
||||
`crates/core/src/curation/draft/seed.rs:28-29` (`sql_text`, doubles single quotes) and
|
||||
`:91-101` (`document_literal`, dollar-quoted with the tag collision refused). The checklist
|
||||
flags SQL built by concatenation; here the output is a committed seed file reviewed by diff
|
||||
and run by `supabase db reset`, not a runtime query. Every interpolated value is derived by
|
||||
the pipeline: the uuid sheet id, the sha256 hex and byte size of the frozen asset, the
|
||||
constant file name, the constant change note, and the Prettier-formatted document inside
|
||||
`$equine_doc$ ... $equine_doc$`. No request or user-typed value reaches it. Parity with
|
||||
`packages/anatomy/src/curation/seed.ts`. Not a parameterization case; listed so it is not
|
||||
silently dropped.
|
||||
|
||||
### 6. [LOW] Clippy allow line in the new test file
|
||||
|
||||
`crates/core/tests/curation_draft.rs:7`:
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. A new
|
||||
file, so new to the tree, but the identical line heads all 31 committed
|
||||
`crates/core/tests/*.rs` files and `crates/clippy.toml` states the rule ("tests and the
|
||||
worker binary allow these lints locally where they own I/O"). Test code, not a security
|
||||
scanner suppression. The binary's pre-existing `#![allow(clippy::disallowed_methods)]`
|
||||
(`import_equine.rs:15`) is an unchanged context line and the documented exception for the
|
||||
two files allowed `std::fs`.
|
||||
|
||||
### 7. [MEDIUM, outside the delta] Guardrails `SB-POLICY-NOT-OWNER`
|
||||
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, policy
|
||||
`species_select_active` on `species` not tied to `auth.uid()`. The scanner runs on the
|
||||
project root; this file is not part of the R5.3 change set and the policy predates it.
|
||||
`species` is shared reference data, so an owner scope is likely not the intent; confirm and
|
||||
record the exception rather than leave it to be re-flagged. `SB-RLS-LIVE` is MANUAL (not
|
||||
reported here).
|
||||
|
||||
## Considered, no finding
|
||||
|
||||
- Non-ASCII slicing: `targets.rs:121` slices at `find(':')` and `migration_map.rs:68` at
|
||||
`rfind('/')`, both ASCII bytes and therefore char boundaries; `migration_map.rs:55-59`
|
||||
(`strip`) works on UTF-16 units with `from_utf16_lossy`; `schema.rs:319` uses
|
||||
`strip_prefix`; `fields.rs:84` splits on `'.'`.
|
||||
- Indexing on parsed JSON in library code: `targets.rs:250,252,253,282,415` read
|
||||
`ids[0]`, `ids[1]`, `ids[2..]` after `parse_target` guarantees one id (collections) or at
|
||||
least two (findings); `fields.rs:92` reads `[0]` of a `split('.')` that always yields one
|
||||
element; `pipeline.rs:81-84` reads four numbers after `len() != 4` is refused;
|
||||
`migration_map.rs:185` is a slice pattern; `ops.rs:277,304,309,259` remove/insert at
|
||||
indices produced from the same list moments before (`insertion_index` refuses
|
||||
`index > len`, `u64 -> usize` saturates). No `unwrap`, `expect`, `panic!`, `unreachable!`,
|
||||
`unsafe` or `Map[...]` in `crates/core/src/curation/draft/*`.
|
||||
- Panics from a hand-written `changes.json` / `references.json`: both parsed with strict
|
||||
serde (`deny_unknown_fields`), custom validators return errors; every op failure refuses
|
||||
the whole changeset with the change id and op index (`apply.rs:103-109`); the final strict
|
||||
parse into `AnatomicalSheetDocument` and `validate_sheet_document` refuse shape and
|
||||
integrity errors (`apply.rs:128-144`); `CHG-000` reserved id refused (`pipeline.rs:127`).
|
||||
Recursion in `walk`, `touch_tree`, `strip_marks` is bounded by document depth (serde_json
|
||||
caps parsed nesting at 128; only repeated authored `add`s could deepen it; a stack overflow
|
||||
would abort a developer tool, nothing more).
|
||||
- `--check` is read-only: `fs::read`, `fs::read_dir`, `println!`/`eprintln!` only.
|
||||
Ran once from `crates/core` on the native toolchain: exit 0,
|
||||
`legacy-import: 6 files, 0 differences`, `draft: 6 files, 0 differences`, six R5.4 files
|
||||
listed as pending; `git status` shows the seed folder untouched.
|
||||
- `--out` never deletes: no `remove_*` call in the binary; refuses an empty path, a
|
||||
symbolic link, a non-folder, a non-empty folder; file names come from constants and the
|
||||
render functions (no `..`); parents created with `create_dir_all`.
|
||||
- `scripts/import-equine.sh`: `set -euo pipefail`, `"$@"` quoted, `cargo run --locked`;
|
||||
`scripts/rust-env.sh` only appends an absolute `$CARGO_HOME/bin` when `cargo` is absent.
|
||||
- TypeScript generator `packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`:
|
||||
writes only `crates/core/tests/oracles/r5/curation-synthetic.json`
|
||||
(`OUT = resolve(HERE, '../../../crates/core/tests/oracles/r5')`), and only under
|
||||
`MAKE_R5_ORACLES=1`; otherwise it reads and compares.
|
||||
- Secrets: none in the change set; no tracked `.env` or credential file; the seed SQL
|
||||
carries no credential (`creator_user_id null`, `visibility 'private'`).
|
||||
- Logging: stderr lines carry paths of the developer's own repository; a CLI for the
|
||||
developer, not an end-user surface.
|
||||
- Semgrep: 0 results, 0 errors, 0 skipped paths across the three pinned packs
|
||||
(725 code rules loaded; Rust coverage in these packs is thin, so the manual checklist
|
||||
carries the weight for `crates/`).
|
||||
- Observation outside the change set: `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md`
|
||||
appeared as modified (1 line) during the audit; not produced by `--check` (no write path).
|
||||
|
||||
REPORT_WRITTEN: /home/bchanot/Documents/faunosteo/.claude/audits/2026-10-22-rust-r5-draft-security.md
|
||||
@@ -0,0 +1,73 @@
|
||||
# Verification: Rust R5.3 draft pipeline (2026-10-22)
|
||||
|
||||
Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. Fresh verifier, report-only (persisted by the orchestrator: the agent has no write tool).
|
||||
|
||||
## Pass 1: ECARTS(1)
|
||||
|
||||
- C1 to C12 MET, each with code references and proofs run by the verifier: `cargo test --features curation --test curation_draft` 6/6, `curation_legacy` 5/5, the whole crate with `--features curation --locked` 22 suites green, clippy `-D warnings` clean, fmt clean, `import-equine --check` 12 files 0 differences (git status unchanged), `--out` into a `mktemp -d` folder then `diff -r` (legacy identical; draft identical but the three R5.4 entries), produced `document_sha256` = `27c7383c…2660` = sha256 of the produced `document.json`, 8 refusal and success probes of `--out`, anatomy 142/142, domain 262/262, `tsc --noEmit` clean.
|
||||
- C13 partial by design: `pnpm check:all` and the security gate were the orchestrator's (both green / PASS); docs present (RUST.md §11.12 and §11.13, CHANGELOG, CLAUDE.md).
|
||||
- Gap: the floor scan flagged the plan heading of §B.3.5 (`.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md:94`, pre-existing at HEAD, R5.1), whose status words read as an unfinished marker. Resolved: heading reworded to "(approved at A.5, applied at R5.6)", the status BDR-068 records.
|
||||
- Semantic fidelity (beyond the declared `set` without `value`): no reachable divergence. Four unreachable ones: (1) `id_text` printed a non-string id as JSON text; aligned on `js_string` (`String()`); (2) `trimEnd` stripped U+0085; aligned on the core's exact JavaScript white-space set (`is_js_space`); (3) an added item with a `null` child: TypeScript throws at the op, Rust refuses at the shape gate (both refuse; kept, declared in RUST.md §11.12); (4) a position beyond 2^64: both refuse "out of range", the printed number differs (kept, declared).
|
||||
- Reproduced quirks: `strip` slices eight UTF-16 units of `op.to`; `apply_record` reads `record.op` whatever the record target (declared observation, RUST.md §11.12).
|
||||
|
||||
## Pass 2: ECARTS(3)
|
||||
|
||||
- C1, C3 to C12 MET (proofs rerun: fmt, clippy, curation_draft 6 + curation_legacy 5, `--check` 12 files 0 differences, 11 `--out` probes incl. a symlinked parent printing the canonical path and a dangling symlink refused, `diff -r`, anatomy 142, domain 262).
|
||||
- C2 NOT MET: JSON `null` on the optional `scope` (set, cite), `position` (add, move) and `field` (add) parsed as absent by serde; Zod `.optional()` refused it. Not reached by the committed changeset. C13 NOT MET on the same root (undeclared). Fixed: those fields deserialised through `js::present` (absent accepted, `null` refused); three refused synthetic cases `schema-null-{scope,position,field}` added to the generator (58 cases: 9 compared, 49 refused alike).
|
||||
- Floor: the pass 1 record above quoted the flagged words; reworded.
|
||||
- Fidelity details, outcome-identical: re-homing candidates on a `Vec` where TS used a `Set` (aligned: deduplicated); `annotation_names` reads a non-string name as empty, duplicate ids first-wins (kept: the legacy import writes strings and unique ids); unlisted counter keys keep insertion order (kept: enum strings); `parseViewBox` error quoting (kept: both refuse).
|
||||
- Doc line count corrected (2,689).
|
||||
|
||||
## Pass 3: ECARTS(4) (first attempt stopped by an API credit error before any check; rerun on Opus)
|
||||
|
||||
- C1, C4 to C10, C12 MET (fmt, clippy, curation_draft 6 + curation_legacy 5, `--check` 12/0, `--out` + `diff -r`, anatomy 142, floor clean, a TypeScript probe of 14 changesets and 2 references files).
|
||||
- C2: three accept/refuse splits at parse, undeclared: (a) a repeated key outside `ops` refused by serde's derive where `JSON.parse` keeps the last value; (d) a lone surrogate escape; (e) nesting deeper than 128. Fixed (a): both files parsed through `js::parse_json` then read into the typed shape (test `a_repeated_key_keeps_its_last_value_as_json_parse_does`); declared (d), (e) in RUST.md §11.12.
|
||||
- C3: (b) a field path ending in `__proto__`: TypeScript swallowed the write, Rust refuses at the gate: declared. (c) an authored `"\u0000touched"` key: TypeScript refused at the gate, Rust took it for a touch mark and the cleanup dropped it silently: a defect, fixed by refusing any NUL-prefixed key in the changeset at parse (test `a_reserved_touch_mark_key_in_an_authored_value_is_refused`).
|
||||
- C11: `--out <symlink>/` (trailing slash) followed the link into an empty folder: fixed, the destination is rebuilt from its components before the symlink check (probed: `alink/` and `alink//` refused, rc 2, target left empty).
|
||||
- C13: the splits above undeclared, and §11.13 claimed a leftover mark would be refused by the gate while the cleanup removed it: both corrected.
|
||||
- The verifier asked the orchestrator to run a probe harness it had written in the scratchpad (its own run was refused by the permission layer); not run: the cases are covered by the tests above and the declared table.
|
||||
|
||||
## Pass 4: ECARTS(1), on C13 only
|
||||
|
||||
- C1 to C12 MET (fmt, clippy, curation_draft 8 + curation_legacy 5, `--check` 12/0, `--out` + `diff -r`, refusal probes incl. `symlink/` and `symlink/.` rc 2, `pnpm r5:import-equine --check` rc 0, anatomy 142, floor clean).
|
||||
- C13: undeclared (a) invalid UTF-8 in changes.json / references.json: TS lossy, Rust refused. Fixed: `read_text` lossy, as R5.2 kept for the catalog. Declared: (b) `1e400` in a discarded value, (c) an inherited `Object.prototype` member in a field path; corrected (c') the NUL-key row, (d) test and line counts (8 tests, 2,718 lines).
|
||||
- Notes kept: synthetic refusals assert `is_err` only (the contract asks no reason); `scripts/import-equine.sh` called directly without INIT_CWD resolves a relative `--out` from the repository root (pnpm sets INIT_CWD).
|
||||
- Not re-verified by a fresh verifier after these corrections (docs plus the one-line lossy read); `pnpm check:all` rerun green on the final code (00:10:03), `--check` 12 files 0 differences.
|
||||
|
||||
## Final round on 8654905 (owner 2026-10-23: fresh verifier on C1 to C13 + fresh security)
|
||||
|
||||
- Verifier ECARTS(2): C3 and C13, one defect. A `set` field path whose segment starts with NUL (`name.\u0000touched`, any settable holder, self or subtree) was written and then stripped with the touch marks: author value lost, a record claiming the write; TypeScript refused it at the gate. C1, C2, C4 to C12 MET (15 + 200 tests, `--check` 0/12, 14 destination probes, floor clean).
|
||||
- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final.md`): MEDIUM-1 a 20,000-segment field path (40 KB changes.json) overflowed the stack; MEDIUM-2 `--out` writes not exclusive after the claim; LOW-1 the NUL field path (same as the verifier); LOW-2 `..` and parents created before a failed claim; LOW-3 a destination inside the seed accepted; LOW-4 INIT_CWD and direct-run base; LOW-5 non-UTF-8 argument panic; LOW-6 clippy disallowed list partial (pre-existing); LOW-7 the test allow line (convention).
|
||||
- Fixed: NUL segment refused at the op (`reserved key`); field paths capped at 8 segments; document depth capped at 64 levels after every op (iterative walk); `--out`: `..` refused, parent must exist, destination inside the seed refused, destination printed before writing, every folder `create_dir`, every file `create_new`; `args_os`. Documented: direct-run base (LOW-4). Kept: LOW-6 (pre-existing lint scope, out of R5.3), LOW-7 (test convention). Tests added: field-path refusals and 20,000 segments refused without crash, 40 nested adds refused at the bound, 8 `--out` refusals against the built binary with nothing written.
|
||||
- Next: a fresh verifier and a fresh security audit on the fixed commit.
|
||||
|
||||
## Second final round on 314cf04
|
||||
|
||||
- Verifier ECARTS(3): C2, C3 (one family) and C13. serde's derives accept a struct written as a positional array and a unit enum written as a one-key map, in changes.json, references.json and the curated document (gate through the R2 derives); Zod refused all. 55,000-changeset random differential (TS generator, Rust probe): 7,436 accepted on both sides byte-identical, 335 mismatches, all this family but one message (`document has no "categories"` vs `document has no list "categories"`). C1, C4 to C12 MET (18 tests, 203 crate tests, `--check` 0/12, refusal probes, floor clean). Note: the same derives sit in the R2 report and archive contracts (unverified there).
|
||||
- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final2.md`): MEDIUM memory amplification (no cap); LOW quadratic depth walk, `--check` and reads following symlinks, residual TOCTOU, `write_exclusive` any component, lossy normalisation, test aimed at the real seed, suppressions context, lint list gaps.
|
||||
- Fixed: `draft/shape.rs` (typed reading serialised back, compared by shape) on both authored files and the gate; 10 changesets + 2 references files frozen from TS (all refused by TS, refused alike); missing-list message aligned; reads: regular file, 16 MiB cap; `listed` does not follow links; `write_exclusive` plain names, root re-checked; seed guard as pure `inside` + unit test, integration test no longer targets the seed. Kept, declared in §11.13: memory amplification, quadratic walk, single-user TOCTOU residual, lossy UTF-8 and repeated keys. Raised to the owner: the R2 derive leniency in the gated readers.
|
||||
|
||||
## Third final round on 42bbc48
|
||||
|
||||
- Verifier ECARTS(2), C3 + C13, one family: a null or primitive among the children of an added item. TS: TypeError where walked (refused), skipped by a reference rename's citation walk (accepted if later removed), gate if kept. Rust: tolerated by touch/walk (accepted), refused by rename's item_at_mut. Three splits; §11.12 claimed "both refuse" for the null child, false. Differential: 82,000 random changesets, 13,758 accepted identical (document, records, stats, map), 63 schema + 34 references probes agree. C1, C2, C4 to C12 MET (19 tests).
|
||||
- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final3.md`), LOW only: reads follow folder links out of the seed; shape comparison ignores array length; a folder appearing after the claim accepted; seed guard wiring untested; `set name.fr.x` records `before: null`; lint allows (context).
|
||||
- Fixed: non-object children refused at the add (3 TS-frozen cases, 1 declared refusal); reads confined to the canonical seed; array length compared; only folders this run created; `claim_destination(typed, seed)` + fixture unit test. Declared: replaced-string `before` (TS same), memory factor ~2x TS.
|
||||
|
||||
## Fourth final round on 6c0fe88
|
||||
|
||||
- Verifier ECARTS(2), C3 + C13: records differ (document identical) on array index keys in authored values or field segments (JSON.parse puts them first) and on integers beyond 2^53 (JSON.parse rounds; a Rust record held 9007199254740993 while document.json printed ...992); §11.12 declared-case count wrong. C1, C2, C4 to C12 MET; 25,000 random changesets: 0 accept/refuse splits, 4,331 accepted identical.
|
||||
- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final4.md`): MEDIUM a present non-list `children` replaced by `[]` (author data lost); LOW `--check` lists through a top-level folder link, mount alias of the seed, walk cost understated (8.2 MB file ~36 s, cap-bound worst case ~45 min), fixture in shared temp, context items.
|
||||
- Fixed: `text::js_normalize` (key order + big integers at parse) and `text::js_insert` (field path writes); non-list `children`/`categories` refused at add, `children_path` never replaces; `--check` requires each generated folder to be the seed's own; `inside_by_identity` (dev, ino of every ancestor); `MAX_DOCUMENT_VALUES` 200,000; fixture under target/. Cases frozen from TS: 2 key-order, 1 integer, 1 declared non-list children; Rust tests: big integer from raw text, value bound.
|
||||
|
||||
## Fifth final round on 3ece1a6
|
||||
|
||||
- Verifier ECARTS(2), C3 + C13: a `__proto__` segment in a field path: at the end, TS records `before: {}` (setter ignored the primitive) where Rust recorded null and kept a plain key; in the middle, TS polluted Object.prototype and refused, Rust accepted when overwritten; RUST.md claimed the inherited-member family refused by TS at the op, false for `__proto__`; test count stale (15 vs 13). C1, C2, C4 to C12 MET; 120,000 random changesets: 40,494 accepted identical, no other undeclared split.
|
||||
- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final5.md`): LOW identity guard covers the seed folder only (a subfolder alias passes), quadratic id comparisons (parse_references 31 s at 160,000), seed SQL formatting (checked), test allow (context).
|
||||
- Fixed: `__proto__` segment refused at the op (reserved key), declared case frozen from TS (the non-polluting one; the polluting one tested in Rust only, it would corrupt the generator's process); identity guard over every folder of the seed tree; HashSet in assert_declared, distinct, duplicates; docs rows and counts.
|
||||
|
||||
## Sixth final round on c6e0d85: CONFORME 13/13 (R5.3 closed)
|
||||
|
||||
- Verifier CONFORME: C1 to C13 MET on the committed state c6e0d85 (23 curation tests, `--check` 12/0 via cargo and pnpm, `--out` + `diff -r`, 21 destination probes, anatomy 142, floor clean; TS-vs-Rust differential on 70,073 changesets: 9,397 accepted identical, 0 Rust-only accepts, TS-only accepts all declared families).
|
||||
- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final6.md`), LOW only: destination folder itself not identity-checked (needs an empty seed folder + a hand mount; none exists), one linear membership in cite (doc precision), seed SQL formatting (checked), test allow (context).
|
||||
- check:all green 10:54:50 on c6e0d85.
|
||||
- Carried into R5.4 (not applied to the verified state): destination identity check, cite membership hashed, "seed folder unreadable" message, stale test comment, doc nits (inherited row: remove form too; 200,000-value row in the table; line count 2,908; set-without-value counted among schema refusals).
|
||||
@@ -0,0 +1,237 @@
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 31 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] unbounded-nesting-stack-overflow — crates/core/src/curation/draft/fields.rs:111
|
||||
- [MEDIUM] out-writes-not-exclusive-toctou — crates/core/src/bin/import_equine.rs:232
|
||||
- [LOW] reserved-nul-key-bypass-via-field-path — crates/core/src/curation/draft/fields.rs:83
|
||||
- [LOW] out-dotdot-and-parent-creation — crates/core/src/bin/import_equine.rs:192
|
||||
- [LOW] out-inside-seed-not-refused — crates/core/src/bin/import_equine.rs:186
|
||||
- [LOW] init-cwd-base-resolution — crates/core/src/bin/import_equine.rs:223
|
||||
- [LOW] args-non-utf8-panic — crates/core/src/bin/import_equine.rs:282
|
||||
- [LOW] core-no-io-lint-partial (pre-existing, context) — clippy.toml:4
|
||||
- [LOW] suppressions (context) — crates/core/tests/curation_draft.rs:7 (new), crates/core/src/bin/import_equine.rs:15 (pre-existing)
|
||||
PROOF: semgrep 163 rules on 31 files → 0 findings; checklist 18 checks → 9 findings (0 CRITICAL, 0 HIGH, 2 MEDIUM, 7 LOW)
|
||||
```
|
||||
|
||||
Security gate, fresh run, 2026-10-23. Scope: `git diff 5e114da..HEAD` (HEAD 8654905, branch
|
||||
`feature/rust-r5-pipeline-core`), 31 changed files, all present. Product files read whole:
|
||||
`crates/core/src/curation/draft/*.rs` (11), `crates/core/src/bin/import_equine.rs`,
|
||||
`crates/core/src/curation/mod.rs`, `crates/core/Cargo.toml`, `crates/core/tests/curation_draft.rs`,
|
||||
`packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`, `scripts/import-equine.sh`,
|
||||
`package.json`. Docs and `.claude` files grepped for secrets only.
|
||||
|
||||
Semgrep's packs carry few Rust rules, so the Rust review below is manual. Two behaviours were
|
||||
confirmed by a scratch probe (a throwaway binary in the session scratchpad linking
|
||||
`faunosteo-core` with `curation`, run on the mini baseline of `oracles/r5/curation-synthetic.json`;
|
||||
nothing in the repository was touched). `import-equine --out` was never run: its behaviour is
|
||||
traced by reading. `import-equine --check` was run as the brief allows: `legacy-import: 6 files,
|
||||
0 differences`, `draft: 6 files, 0 differences`, 6 pending R5.4 files listed, exit 0, `git status`
|
||||
clean afterwards.
|
||||
|
||||
## Checklist
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Hardcoded secret, token, key, auth URL (diff incl. docs, `.claude`, oracle JSON) | none |
|
||||
| 2 | SQL by interpolation (`seed.rs`) | none, see note A |
|
||||
| 3 | Unsanitized render / eval | n/a (no web output; the TS test has no eval) |
|
||||
| 4 | Sensitive endpoint without authz | n/a (no endpoint; seed SQL design pre-existing) |
|
||||
| 5 | Stack trace / internal path to an end user | none (dev CLI, stderr to the developer) |
|
||||
| 6 | Secret / PII in a log | none |
|
||||
| 7 | Tracked `.env` / credential file | none in the diff (repo tracks `.env.example` only) |
|
||||
| 8 | Guardrails scanner (`api,supabase`) | 0 FAIL; 1 WARN `SB-POLICY-NOT-OWNER` in `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff, dropped (gate rule) |
|
||||
| 9 | Panics from `changes.json` / `references.json`: indexing, unwrap, byte slicing | none, see note B |
|
||||
| 10 | Unbounded recursion, depth and resource caps | MEDIUM-1 |
|
||||
| 11 | Reserved NUL-key refusal | LOW-1 |
|
||||
| 12 | `--out` claim: symlinks, trailing `/`, `.` / `..`, TOCTOU | MEDIUM-2, LOW-2, LOW-3 |
|
||||
| 13 | `INIT_CWD` | LOW-4 |
|
||||
| 14 | `--check` writes nothing | verified (run + read: no write call on that path) |
|
||||
| 15 | Generator writes outside `crates/core/tests/oracles/r5/` | none: one constant path, `curation-synthetic.json`, only under `MAKE_R5_ORACLES=1` |
|
||||
| 16 | Core has no fs / env / clock access | holds by inspection; lint partial, LOW-6 |
|
||||
| 17 | CLI argument handling | LOW-5 |
|
||||
| 18 | New suppressions (anti-gaming) | LOW-7 (context, non-blocking) |
|
||||
|
||||
## Findings
|
||||
|
||||
### MEDIUM-1: nesting built by ops is unbounded, a small `changes.json` aborts the tool
|
||||
|
||||
- Files: `crates/core/src/curation/draft/fields.rs:83-89` (`split_field`, no segment cap) and
|
||||
`:111-126` (`write_field` creates one object per segment); first recursive walker hit
|
||||
`crates/core/src/curation/draft/records.rs:114-123` (`strip_marks`, called at
|
||||
`apply.rs:114`); same class in `records.rs:88-105` (`touch_tree`), `targets.rs:298-322`
|
||||
(`walk`), `apply.rs:129` (`doc.clone()` + `from_value`), and the recursive `Value` drop.
|
||||
- serde_json's 128-level parse limit (no `unbounded_depth` feature) caps each authored value,
|
||||
but the ops build depth past it. A `set` whose `field` is `name` followed by N `.a` segments
|
||||
creates N nested objects in one op.
|
||||
- Probe, release build (the profile `scripts/import-equine.sh` uses): N = 10,000 is refused cleanly
|
||||
(`curated document is not shape-valid: unknown field 'a'`). N = 20,000 (a ~40 KB
|
||||
`changes.json`) gives `thread 'main' has overflowed its stack`, `fatal runtime error: stack
|
||||
overflow, aborting`, exit 134. The debug build (`cargo run` without `--release`) has larger
|
||||
frames and a lower threshold.
|
||||
- The finding tree has the same problem, by analysis only (not run): repeated `add` ops (each up to
|
||||
~63 finding levels under the parse limit) or `move` chains deepen the tree without limit.
|
||||
`walk` clones the growing path for every entry, so memory grows with depth² before the stack
|
||||
overflows.
|
||||
- Impact: the process aborts where it should give a typed refusal. It still fails closed: the
|
||||
pipeline (`folders()`) runs before `claim_destination`, so no folder is created and nothing is
|
||||
written. The input is a committed, owner-authored file. Robustness of a dev tool, and a gap in
|
||||
the "depth and resource caps kept" requirement. Not exploitable beyond a crash.
|
||||
- Fix: cap `split_field` segments, refusing beyond the deepest settable path (`name.fr`,
|
||||
`count_range.min`, `geometry.cx`: 2 segments, so a cap of 4 is generous). Cap the finding tree
|
||||
depth with an iterative check after each `add` and `move`, e.g. a `MAX_FINDING_DEPTH` constant
|
||||
that refuses with the change id and op index.
|
||||
|
||||
### MEDIUM-2: `--out` writes are not exclusive, so the claim can be raced
|
||||
|
||||
- File: `crates/core/src/bin/import_equine.rs:232-247` (`write`: `fs::create_dir_all` at :239,
|
||||
`fs::write` at :242), together with `:186-219` (`claim_destination`).
|
||||
- The claim checks the folder once: absent then `mkdir`, or existing and empty. The writes then
|
||||
use `create_dir_all`, which follows an existing symlink, and `fs::write`, which is `O_CREAT |
|
||||
O_TRUNC` and follows symlinks. Anything planted between the claim and the writes is followed or
|
||||
truncated. That could be `<out>/draft` turned into a symlink to the real seed `draft/`, or a
|
||||
file created early. The claimed path could also be swapped for a symlink before `canonicalize`
|
||||
(:218). An existing empty folder is accepted whatever its owner and mode. The cross-user vector
|
||||
is an empty, world-writable folder pre-created by another account in a shared location.
|
||||
- The doc comment at :182-185 says "created here, so nothing can appear between the check and the
|
||||
writes". That holds against a symlink at the final component (`mkdir` gives `EEXIST`), but not
|
||||
against entries created inside the folder afterwards.
|
||||
- Impact: a concurrent same-user process, or a hostile local account on a multi-user host, can
|
||||
redirect or overwrite files. On the single-user dev machine this needs an attacker who already
|
||||
has the user's rights. It is still an overstatement of the "never overwrites" guarantee the
|
||||
owner asked for, and the std-only fix is cheap.
|
||||
- Fix: create every subfolder under the root with `fs::create_dir`, never `_all` (fails on any
|
||||
existing entry and does not follow a final symlink). Write every file with
|
||||
`OpenOptions::new().write(true).create_new(true)` (`O_EXCL`: refuses an existing file or
|
||||
symlink). Optionally refuse an existing claimed folder that is not owned by the current uid or
|
||||
is group/world-writable (`std::os::unix::fs::MetadataExt`). Correct the comment.
|
||||
|
||||
### LOW-1: the reserved NUL-key refusal does not cover dotted `field` segments
|
||||
|
||||
- Files: `crates/core/src/curation/draft/schema.rs:389-410` (`reserved_key` scans JSON object keys
|
||||
only), `fields.rs:83-89` (`split_field` accepts any non-empty segment), `records.rs:16`
|
||||
(`TOUCH_MARK = "\u{0}touched"`), `records.rs:114-123` (`strip_marks` removes that key at every
|
||||
level).
|
||||
- Probe: `{"op":"set","target":"finding:hip/hip-dysfunctions/mobility","field":"name.\u0000touched","value":"author text"}`
|
||||
is accepted. The document's `name` stays `{"fr":"mobility fr","en":"mobility en"}` and a record
|
||||
`field="name.\0touched" after="author text"` is still emitted. The authored value is dropped
|
||||
silently while the trace (the R5.4 CHANGELOG) claims the write. Any other NUL-prefixed segment,
|
||||
e.g. `name.\u0000other`, is refused by the strict shape gate. The TypeScript kept the key and
|
||||
refused at that gate.
|
||||
- No review-rule bypass: the first segment must be settable, so the mark cannot be set at item
|
||||
level. Every authored JSON key holding the mark is refused at parse.
|
||||
- Fix: refuse a `field` segment that starts with `\0` (in `split_field` or the `SetOp` parse),
|
||||
with the same "reserved key" message.
|
||||
|
||||
### LOW-2: `..` is kept and resolved after symlinks, and a failed claim leaves created folders
|
||||
|
||||
- File: `crates/core/src/bin/import_equine.rs:192-200`.
|
||||
- Rebuilding the path from `components()` drops a trailing `/` and inner `.` components, which is
|
||||
correct: a final symlink is then refused by `symlink_metadata`, dangling links included.
|
||||
`..` is kept, though, and the kernel resolves it after any symlinked component.
|
||||
`--out link/../new` creates `<parent of link target>/new`, not the lexical `./new`.
|
||||
`--out link/..` is always refused (the parent of the target is never empty).
|
||||
- In the absent branch, `create_dir_all(parent)` runs before the final `create_dir`. A claim that
|
||||
then fails, e.g. `--out link/new/..` (`mkdir ..` gives `EEXIST`), leaves `<link target>/new`
|
||||
behind, created through the symlink, while the tool prints "nothing written".
|
||||
- Parent symlinks are followed by design (doc comment), but the resolved destination is printed
|
||||
only after the files are written.
|
||||
- Fix: refuse any `Component::ParentDir`. Create only the final component, refusing a missing
|
||||
parent, or document the parent creation. Print the canonical destination before the first
|
||||
write.
|
||||
|
||||
### LOW-3: a destination inside the official seed tree is not refused
|
||||
|
||||
- File: `crates/core/src/bin/import_equine.rs:186-219`.
|
||||
- `--out supabase/seed/anatomical-sheets/equine/x` passes the claim, being absent, and writes
|
||||
`x/legacy-import/...` and `x/draft/...` inside the seed. Committed files are never overwritten,
|
||||
and the populated folders (`legacy-import/`, `draft/`) are refused as not empty. The owner
|
||||
requirement "never modifies the official seed automatically" holds, since this needs an
|
||||
explicitly typed path. But generated folders can land untracked in the seed, and `--check` then
|
||||
reports them as "not produced".
|
||||
- Fix: refuse when the canonical destination is under `fs::canonicalize(SEED)`, or under the
|
||||
repository root.
|
||||
|
||||
### LOW-4: `INIT_CWD` and the base of a relative `--out`
|
||||
|
||||
- Files: `crates/core/src/bin/import_equine.rs:223-229`, `scripts/import-equine.sh:12`.
|
||||
- Any absolute `INIT_CWD` in the environment is trusted, including one inherited or left stale by
|
||||
an outer pnpm. Without pnpm, the script `cd`s to the repository root, so a relative `--out` from
|
||||
`bash scripts/import-equine.sh` resolves against the repository root, not the caller's folder.
|
||||
The doc says "else the current one". If `current_dir()` fails, the base becomes empty and the
|
||||
path stays relative to a cwd that no longer exists, which fails harmlessly. Not dangerous with
|
||||
the claim rules (new or empty folder only), but surprising.
|
||||
- Fix: print the resolved destination before claiming, and state in the doc that direct runs
|
||||
resolve from the repository root. Or refuse a relative `--out` when `INIT_CWD` is unset.
|
||||
|
||||
### LOW-5: a non-UTF-8 argument panics
|
||||
|
||||
- File: `crates/core/src/bin/import_equine.rs:282` (`env::args()` panics on non-Unicode argv).
|
||||
Self-inflicted, CLI only, not reachable from `changes.json`.
|
||||
- Fix: `env::args_os()` and a usage error on a non-UTF-8 argument.
|
||||
|
||||
### LOW-6 (pre-existing, context): the core's no-I/O guarantee is only partly lint-enforced
|
||||
|
||||
- File: `clippy.toml:4-14` (not in this diff). `disallowed-methods` lists `fs::read`,
|
||||
`read_to_string`, `write`, `File::open`, `File::create`, `env::var`, `var_os` and the clocks.
|
||||
It does not list `fs::create_dir(_all)`, `remove_file`, `remove_dir(_all)`, `read_dir`,
|
||||
`rename`, `copy`, `metadata`, `symlink_metadata`, `canonicalize`, `OpenOptions::open`,
|
||||
`env::current_dir`, `env::args(_os)` or `process::Command::new`.
|
||||
- A grep of `crates/core/src/curation/` finds no `std::fs`, `std::env`, `std::process`,
|
||||
`OpenOptions` or `unsafe`, and no `allow(clippy` in the draft modules. The claim holds today by
|
||||
inspection, not by lint.
|
||||
- Fix: extend the list so a future library change cannot add file system access unnoticed.
|
||||
|
||||
### LOW-7 (context): suppressions
|
||||
|
||||
- New: `crates/core/tests/curation_draft.rs:7`
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. This is a
|
||||
test target, the established pattern of the crate's integration tests (`anatomy_*`,
|
||||
`archive_rules`, `curation_format`, `curation_legacy` and others), and the pattern `clippy.toml`
|
||||
itself documents ("Tests ... allow these lints locally where they own I/O"). It allows a
|
||||
robustness/architecture lint, not a security scanner (`nosemgrep`, `nosec`, `eslint-disable
|
||||
security`), so it is not counted as an un-gated suppression.
|
||||
- Pre-existing: `crates/core/src/bin/import_equine.rs:15` `#![allow(clippy::disallowed_methods)]`
|
||||
(present at 5e114da, line 11).
|
||||
- No `nosemgrep`, `nosec`, `eslint-disable`, `ts-ignore` or `noqa` was added in product files.
|
||||
|
||||
## Notes (verified, no finding)
|
||||
|
||||
A. Seed SQL (`seed.rs`). Every single-quoted value is a constant or derived from one:
|
||||
`sheet_id_for(EQUINE_SPECIES_ID, EQUINE_SHEET_VERSION)`, sha256 hex, the uuid asset id, `usize`,
|
||||
`horse.svg`, the description and the change note. Quotes are doubled, and backslashes are literal
|
||||
under `standard_conforming_strings`. Author data reaches only the dollar-quoted document literal,
|
||||
which is refused when the JSON contains `$equine_doc$`. The close cannot overlap the body's tail,
|
||||
because the body is a pretty-printed JSON object trimmed to its final `}`, so no suffix of it is a
|
||||
prefix of the tag. Ported verbatim from `seed.ts`.
|
||||
|
||||
B. Panics. Library unwrap/expect are denied by workspace lints. Every index is guarded:
|
||||
`numbers[0..3]` after `len == 4`, `ids[0]`/`[1]`/`[2..]` after the `ids.len() >= 2` check for
|
||||
findings, and `split_field(..)?[0]` on a non-empty split. Byte slices cut on ASCII `:` and `/`
|
||||
(char boundaries). `Vec::insert` is bounded by `insertion_index` (checked `<= len`, `After` gives
|
||||
at most `len`). `Vec::remove` uses an index `locate` just resolved. `move` re-locates the
|
||||
destination after the removal. `u64` positions saturate and are refused by the range check.
|
||||
The only aborts found are the stack overflows of MEDIUM-1 (`panic = "abort"` in release; a stack
|
||||
overflow aborts in any profile).
|
||||
|
||||
C. Strict validation. `deny_unknown_fields` on every authored struct; the `op` tag is removed
|
||||
before strict parsing of the variant. `.optional()` semantics: `null` is refused, absent is
|
||||
accepted. `CHG-\d{3,}` ids, duplicate change and reference ids, level C without a question, the
|
||||
reserved `CHG-000` and empty ops or citations are all refused. Repeated JSON keys keep the last
|
||||
value, as `JSON.parse` does. `references.json` is parsed into the strict `Reference`.
|
||||
|
||||
D. `--check` only reads: `fs::read`, `read_dir`, `is_dir`. `listed` follows directory symlinks in
|
||||
the seed. A loop ends in an `ELOOP` error and a symlinked file is only compared, never printed.
|
||||
These are trusted, committed contents.
|
||||
|
||||
E. The oracle generator writes only
|
||||
`crates/core/tests/oracles/r5/curation-synthetic.json` (`resolve(HERE, '../../../crates/core/tests/oracles/r5')`,
|
||||
a constant), only when `MAKE_R5_ORACLES === '1'`, and otherwise just compares.
|
||||
`crates/core/tests/curation_draft.rs` only reads.
|
||||
|
||||
F. Secrets: none in the 31 files (diff grep for key, token, password, bearer, `service_role`, JWT,
|
||||
PEM, cloud key prefixes and `user:pass@` URLs; semgrep `p/secrets` 0). The references use example
|
||||
URLs without credentials.
|
||||
@@ -0,0 +1,211 @@
|
||||
# Security gate (fresh): R5.3 draft port, final 2
|
||||
|
||||
Branch `feature/rust-r5-pipeline-core`, HEAD `314cf04`, scope `git diff 5e114da..HEAD`
|
||||
(32 files), focus on `9f11499`. Fresh run: no earlier verdict consulted.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 32 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] resource-cap/memory-amplification — crates/core/src/curation/draft/records.rs:68
|
||||
- [LOW] resource-cap/cpu-quadratic — crates/core/src/curation/draft/apply.rs:109
|
||||
- [LOW] check-follows-symlinks — crates/core/src/bin/import_equine.rs:148
|
||||
- [LOW] out-toctou-residual — crates/core/src/bin/import_equine.rs:203
|
||||
- [LOW] write-exclusive-components — crates/core/src/bin/import_equine.rs:246
|
||||
- [LOW] silent-input-normalisation — crates/core/src/bin/import_equine.rs:66
|
||||
- [LOW] test-targets-real-seed — crates/core/tests/curation_draft.rs:681
|
||||
- [LOW] suppression-context (new, test crate, documented policy) — crates/core/tests/curation_draft.rs:7
|
||||
- [LOW] suppression-context (pre-existing) + lint list gaps — crates/core/src/bin/import_equine.rs:15
|
||||
PROOF: semgrep 129 rules on 32 files → 0 findings; checklist 10 checks → 9 findings
|
||||
```
|
||||
|
||||
## Tool notes
|
||||
|
||||
- semgrep 1.168.0, pinned packs only, `--metrics=off`. The first run died inside semgrep-core
|
||||
(`io_uring_queue_init: Cannot allocate memory`, sandbox memlock limit). The rerun with
|
||||
`EIO_BACKEND=posix` scanned all 32 files: 0 errors, 129 rules run (163 loaded), 0 findings.
|
||||
- These packs carry almost no Rust rules. The Rust analysis below therefore rests on a full
|
||||
read of every changed product file plus runtime probes, not on semgrep.
|
||||
- Guardrails scanner (`api,supabase`): 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER` in
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql`, outside the diff, dropped).
|
||||
|
||||
## Checklist
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Hardcoded secret, token, key, auth URL (diff incl. docs, `.claude`, oracle JSON) | none (only `https://example.org/a`) |
|
||||
| 2 | SQL by concatenation / interpolation | none: `seed.rs` interpolates constants only, `sql_text` doubles quotes, the document goes into `$equine_doc$…$equine_doc$`, and a document holding the tag is refused. The trimmed document always ends with `}`, so no prefix of the tag can join the closing tag. `config.toml` `sql_paths` lists literal files, so an `--out` under `supabase/seed/` is never executed by `db reset` |
|
||||
| 3 | Unsanitised render / `eval` | none (no rendering; the TS oracle test has no sink) |
|
||||
| 4 | Sensitive endpoint without authz | N/A (dev tool, no endpoint) |
|
||||
| 5 | Stack trace / internal path to end users | none (operator CLI only) |
|
||||
| 6 | Secret / PII in a log | none |
|
||||
| 7 | Tracked `.env` / credential file | none (`.env.example` only, outside the diff) |
|
||||
| 8 | Guardrails scanner | 0 findings in diff files |
|
||||
| 9 | New suppressions (anti-gaming) | no `nosemgrep` / `nosec` / `eslint-disable` / `ts-ignore` / `noqa`; one clippy allow (L7) |
|
||||
| 10 | Brief-specific: seed protection, `--check` read-only, `--out` semantics, caps, strict validation, author data | M1, L1 to L6 |
|
||||
|
||||
## Findings
|
||||
|
||||
### M1 [MEDIUM] No memory cap: hostile `changes.json` amplifies about 10,000× (non-blocking)
|
||||
|
||||
`records.rs:68-75`: every record deep-clones the whole `Op`, value included, plus
|
||||
`before`/`after`. `ops.rs:145-164`: a subtree `set` writes the value into every entry and
|
||||
records it once per entry. `ops.rs:193,197`: a `rename` records the whole item twice, subtree
|
||||
included. `import_equine.rs:60-68`: input reads have no size bound. The depth and segment
|
||||
caps of `9f11499` bound the stack. Nothing bounds memory, the document size or the record
|
||||
volume.
|
||||
|
||||
Measured with the release probe under `ulimit -v 3000000`:
|
||||
|
||||
| input | size | peak RSS | outcome |
|
||||
|---|---|---|---|
|
||||
| real `changes.json` | 227 KB | 14 MB | OK, 825 records |
|
||||
| add 1,000 nodes + subtree `set` of a 100 KB description | 154 KB | 1.10 GB | OK, accepted and rendered |
|
||||
| same, 2,000 nodes | 211 KB | > 2.38 GB | abort: `memory allocation of 804175406 bytes failed` |
|
||||
| 5,000-node category + 200 rename ping-pongs | 300 KB | 2.13 GB | OK |
|
||||
| same, 500 renames | 326 KB | > 3 GB | abort |
|
||||
|
||||
Growth is linear in (ops × item size): one 90-byte rename op on that category costs about
|
||||
10 MB. A 1 MB hostile file asks for tens of GB. This host has 14 GB of RAM with its 3 GB of
|
||||
swap already full, so the result is an OOM kill or thrashing, during `--check` as well. No
|
||||
data is lost and nothing is written. The input is an authored repository file, which can come
|
||||
from a contributor PR on an MIT project. The TypeScript cloned the same way, so this is a
|
||||
parity issue. It still falls short of the owner's explicit "depth and resource caps"
|
||||
requirement.
|
||||
|
||||
Fix direction:
|
||||
- Cap the input files in the binary (for example `changes.json` ≤ 4 MiB; the real file is
|
||||
227 KB).
|
||||
- Bound the curated document next to `assert_depth`: node count or serialised size, plus
|
||||
string length.
|
||||
- Optionally, records reference the op (index or `Rc<Op>`) instead of cloning it.
|
||||
|
||||
### L1 [LOW] Quadratic CPU: whole-document depth walk after every op
|
||||
|
||||
`apply.rs:109,125-147`: `depth_of` (new in `9f11499`) walks the whole document after each op.
|
||||
A 2.4 MB input (20,000-node added subtree, then 10,000 one-field `set` ops) takes 9.9 s. The
|
||||
cost is O(ops × document). Fix direction: the input cap of M1. Otherwise compute depth
|
||||
incrementally (the depth of the touched item's path plus the depth of the written or inserted
|
||||
value).
|
||||
|
||||
### L2 [LOW] `--check` and input reads follow symbolic links of the committed seed
|
||||
|
||||
`import_equine.rs:148`: `listed` decides recursion with `entry.path().is_dir()`, which follows
|
||||
links. `import_equine.rs:60-62,163`: `fs::read` follows links and has no size bound. Two
|
||||
consequences:
|
||||
- A committed `draft/x -> /` makes `--check` walk the filesystem and print file names to
|
||||
stderr, bounded only by ELOOP and time.
|
||||
- `curation/changes.json -> /dev/zero`, or a FIFO, ends in OOM or a hang.
|
||||
|
||||
Both stay read-only and local, with no exfiltration. Traced by reading: no symlink was planted
|
||||
in the repository. Fix direction: use `entry.file_type()` (no follow) and report a symlink as a
|
||||
difference; read inputs only after a `symlink_metadata` regular-file check and under a size
|
||||
cap.
|
||||
|
||||
### L3 [LOW] Residual check-then-use window in `--out`
|
||||
|
||||
The path is canonicalised at `import_equine.rs:203`, then the folder is created at `:216`.
|
||||
Each intermediate folder is checked with `symlink_metadata` at `:252`, then the file is opened
|
||||
at `:257-261`. All of these are path-based. A concurrent local actor with write access to the
|
||||
destination's parent, or the owner of a pre-existing empty destination, could swap a
|
||||
component for a symlink between check and use. The impact is bounded: every file is
|
||||
`create_new` (O_EXCL) with a fixed name, so nothing is overwritten, truncated or deleted. The
|
||||
worst case is new fixed-name files in another folder the user can write. Fix direction, only
|
||||
if wanted: hold the claimed root as a directory fd and create children with
|
||||
`openat`/`mkdirat` + `O_NOFOLLOW` (rustix or cap-std). Otherwise document the single-user
|
||||
machine assumption.
|
||||
|
||||
### L4 [LOW] `write_exclusive` accepts any path component
|
||||
|
||||
`import_equine.rs:244-255` pushes each component of `relative` without refusing `RootDir`,
|
||||
`ParentDir` or `Prefix`. Today every name is a compile-time constant (`render_artifacts`,
|
||||
`render_draft_artifacts`, `assets/Horse.svg`), so this is not exploitable. It becomes a
|
||||
traversal if R5.4 derives a file name from data, for example review drawings named after view
|
||||
ids. Fix direction: refuse any component other than `Component::Normal`.
|
||||
|
||||
### L5 [LOW] Silent input normalisation, against "author data never dropped silently"
|
||||
|
||||
- `import_equine.rs:66-68`: invalid UTF-8 in `changes.json` or `references.json` silently
|
||||
becomes U+FFFD.
|
||||
- `schema.rs:382-384` (`js::parse_json`): a duplicated key keeps its last value, and the first
|
||||
is dropped without notice.
|
||||
|
||||
Both are declared parity with `readFile(..., 'utf8')` and `JSON.parse`. Fix direction: refuse
|
||||
invalid UTF-8 (`String::from_utf8`) and duplicate keys in the authored files, or warn.
|
||||
|
||||
### L6 [LOW] A test runs the real binary against the official seed path
|
||||
|
||||
`curation_draft.rs:676-701` runs `import-equine --out <seed>/new-folder`. If the seed guard
|
||||
regressed, the test itself would create `new-folder/` with 12 files inside the official seed
|
||||
before its assertion fails (no overwrite). For the same reason this gate did not run
|
||||
`cargo test --test curation_draft`: the brief forbids `--out` against a repository path. Fix
|
||||
direction: assert the seed refusal through the pure claim logic with an injectable seed root,
|
||||
or keep it and accept the regression risk knowingly.
|
||||
|
||||
### L7 [LOW, context] New lint allow in a test crate: not a security suppression
|
||||
|
||||
`curation_draft.rs:7` adds
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. It sits on an
|
||||
integration-test crate, uses the same line as 10 existing test files, and follows the policy
|
||||
written in `clippy.toml` ("Tests … allow these lints locally where they own I/O"). It silences
|
||||
no security scanner and no security finding, so it is not blocking.
|
||||
|
||||
### L8 [LOW, context] Pre-existing allow, and gaps in the I/O ban list
|
||||
|
||||
`import_equine.rs:15` `#![allow(clippy::disallowed_methods)]` was already present at
|
||||
`5e114da`. Separately, `clippy.toml` bans only `fs::read`, `read_to_string`, `write`,
|
||||
`File::open`/`create`, `env::var`/`var_os` and the clocks. `fs::create_dir*`,
|
||||
`OpenOptions::open`, `read_dir`, `canonicalize`, `remove_*`, `env::current_dir`,
|
||||
`env::args_os` and `process::Command` would pass the lint inside the library. A grep confirms
|
||||
that `crates/core/src` outside `src/bin` uses none of them today. Fix direction: extend the
|
||||
list.
|
||||
|
||||
## Verified, no finding
|
||||
|
||||
Stack and validation. The probe linked `faunosteo-core` with the `curation` feature and ran
|
||||
in debug on a 2 MiB thread, in debug on the main thread, and in release:
|
||||
- a 20,000-segment field and a 9-segment field are refused; the 8-segment field is refused
|
||||
later by the strict shape check;
|
||||
- `name.\0touched` is refused; a `\0touched` key inside a value is refused at parse;
|
||||
- a value at serde's 128 nesting limit is refused with "recursion limit exceeded";
|
||||
- a 122-deep object value, a 58-level children chain and 40 chained adds (refused at op 29)
|
||||
all end with "nests more than 64 levels";
|
||||
- moving an item into itself is refused;
|
||||
- no crash, no panic in any mode.
|
||||
|
||||
Margins on real data: document depth 10 against the cap of 64; field segments at most 2
|
||||
against the cap of 8.
|
||||
|
||||
`--check`, as briefed: `legacy-import` 6 files, 0 differences; `draft` 6 files, 0 differences
|
||||
(6 pending R5.4). `git status` clean afterwards.
|
||||
|
||||
`--out`, 20 cases in a fresh `mktemp -d`. A copy of the binary was built there; the
|
||||
repository stayed clean.
|
||||
- Written:
|
||||
- an absent folder: 12 files, `document.json` byte-identical to the seed;
|
||||
- an existing empty folder;
|
||||
- a parent symlink: the files land at the canonical path;
|
||||
- an interior `./`;
|
||||
- a relative path with an absolute `INIT_CWD`; a relative `INIT_CWD` is ignored.
|
||||
- Refused with exit code 2 and nothing created:
|
||||
- an existing non-empty folder; a file;
|
||||
- a symlink, `symlink/`, `symlink/.`, a dangling symlink (its target is not created);
|
||||
- a missing parent; `..`; `""`; `/`; an extra argument; a non-UTF-8 argument.
|
||||
|
||||
Scratch total: 74 files = 6 × 12 + 2 fixtures.
|
||||
|
||||
Seed protection, traced by reading and not run against the repository:
|
||||
- the parent is canonicalised, then compared with `starts_with` component by component
|
||||
against the canonical seed;
|
||||
- a root equal to the seed, or to any ancestor, is non-empty and refused;
|
||||
- a final-component symlink is refused;
|
||||
- `folders()` is computed before the claim, so a refused changeset creates nothing.
|
||||
|
||||
The library performs no file system, environment or process access. The Rust ownership model
|
||||
rules out reference cycles from `move`.
|
||||
|
||||
Probe artefacts (scratch only):
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/secfinal2.SbMNbq`
|
||||
@@ -0,0 +1,188 @@
|
||||
# Security gate (fresh, audit mode): R5.3 curated draft, third final round
|
||||
|
||||
Branch `feature/rust-r5-pipeline-core`, HEAD `42bbc48`. Scope: `git diff 5e114da..HEAD`, 34 files,
|
||||
with the fix commit `776c709` read whole (shape.rs, schema.rs, apply.rs, targets.rs,
|
||||
bin/import_equine.rs, tests/curation_draft.rs), plus every other changed product file under
|
||||
`crates/core/src/curation/draft/` and `scripts/import-equine.sh`.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 34 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] reads-follow-directory-symlinks — crates/core/src/bin/import_equine.rs:74
|
||||
- [LOW] shape-array-length-not-enforced — crates/core/src/curation/draft/shape.rs:47
|
||||
- [LOW] write-exclusive-accepts-a-new-folder — crates/core/src/bin/import_equine.rs:282
|
||||
- [LOW] seed-guard-wiring-untested — crates/core/src/bin/import_equine.rs:223
|
||||
- [LOW] write-field-before-null-on-replaced-holder — crates/core/src/curation/draft/fields.rs:131
|
||||
- [LOW] clippy-allow-new-in-test (context) — crates/core/tests/curation_draft.rs:7
|
||||
- [LOW] clippy-allow-pre-existing (context) — crates/core/src/bin/import_equine.rs:15
|
||||
PROOF: semgrep 163 rules on 34 files → 0 findings; checklist 14 checks → 7 findings
|
||||
```
|
||||
|
||||
## Method
|
||||
|
||||
- semgrep 1.168.0, the three pinned packs, `--metrics=off --quiet --json`, on the 34 existing
|
||||
files of the diff (16 `.rs`, 13 `.md`, 2 `.json`, 1 `.ts`, 1 `.sh`, 1 `.toml`). Exit 0, 0
|
||||
errors, 0 results; the `--time` rerun counts 163 rules. No `EIO_BACKEND=posix` was needed.
|
||||
These packs carry few Rust rules, so the verdict rests on reading the code and on the probes.
|
||||
- Checklist (14 checks): hardcoded secrets, SQL built by formatting, unsafe render, authz,
|
||||
errors to the user, secrets in logs, tracked `.env` / credentials, guardrails scanner,
|
||||
new suppressions, `--out` destination containment, read containment, shape and strictness
|
||||
gate, depth and resource caps, author data loss and audit trail.
|
||||
- Probes, all under
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/probe.m1yh/`:
|
||||
a copy of the seed (`seedroot/sheets/equine`), a throwaway crate linking `faunosteo-core`
|
||||
with `curation`: `ie` = `import_equine.rs` with only the `SEED` constant pointed at the
|
||||
copy (diff = those lines only), and `shape` = direct calls to `parse_changeset` and
|
||||
`assert_same_shape`. No `--out` ever targeted the repository. Nothing was deleted; the
|
||||
scratch artifacts stay in place.
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| `ie --check` on the unmodified copy | legacy-import 6 files 0 differences, draft 6 files 0 differences, 6 pending R5.4, exit 0 |
|
||||
| `--out` parent = symlink into the seed (`seedlink/new`, `//seedlink//new`) | refused, inside the seed folder |
|
||||
| `--out` = symlink to the seed (`seedlink`, `seedlink/`) | refused, a symbolic link |
|
||||
| `--out` = seed itself / empty folder inside seed / new folder inside seed / `sheets/./equine/new` | refused (not empty / inside the seed folder) |
|
||||
| `--out /`, `--out ""` | refused |
|
||||
| `INIT_CWD=<seed>` with a relative `--out` | refused, inside the seed folder |
|
||||
| FIFO at a committed file name in `draft/` | not opened, reported, exit 1, no hang |
|
||||
| 17 MiB sparse input in `legacy-sources/` | refused, larger than 16777216 bytes, exit 2 |
|
||||
| `add` of a value nested 118 levels; `add` of a 60-level children chain (level B) | refused, nests more than 64 levels, exit 2, no stack overflow |
|
||||
| changeset value nested 120 / 140 levels | parsed / refused by serde's recursion limit |
|
||||
| title as array, level as map, scope as map, change as array, position as array | all refused (shape gate or untagged enum) |
|
||||
| `draft/assets` and `curation` replaced by directory symlinks to outside folders | see L1: read through both links |
|
||||
|
||||
## Findings (all LOW, non-blocking)
|
||||
|
||||
**L1. Reads follow a symbolic link in any folder component. `import_equine.rs:64-75`
|
||||
(read at :74), callers :84, :96-97, :123-127, :178.**
|
||||
`read` checks the final component with `symlink_metadata` (:67), then `fs::read(path)` (:74)
|
||||
resolves the whole path again. A directory symlink inside the seed is followed. Measured: with
|
||||
`draft/assets` and `curation` swapped for directory symlinks to folders outside the seed copy,
|
||||
`--out` wrote its 12 files, exit 0, no warning, built from the changeset and the cleaned drawing
|
||||
read through the links. `--check` read `assets/horse.svg` through the link without calling it
|
||||
missing, and only flagged `not produced: assets`, exit 1. Between :67 and :74 a swap of the
|
||||
final component is also followed, and a FIFO swapped in would block the read. Impact is small:
|
||||
read-only, regular files under 16 MiB, output only to a fresh folder or a comparison, and the
|
||||
symlink must already sit in the user's own repository. But the docstring at :64 ("never a
|
||||
symbolic link") and RUST.md:3017 ("every read a regular file under 16 MiB, never a link")
|
||||
claim more than the code does, and the owner requirement says no unintended access through
|
||||
symbolic links. The §11.13 single-user residual names `--out` only.
|
||||
Fix: open once and check the handle. Either canonicalise the path and require it to equal the
|
||||
lexical path under the canonical seed (no link in any component), then `File::open`,
|
||||
`file.metadata()` for `is_file` and length, and read through `take(MAX_FILE_BYTES + 1)`. Or
|
||||
open with `O_NOFOLLOW` per component. Otherwise, correct :64 and RUST.md:3017 to "final
|
||||
component only" and add reads to the §11.13 residual.
|
||||
|
||||
**L2. The shape gate does not enforce array length. `shape.rs:47-53`.**
|
||||
When two arrays differ in length, the length guard on the array arm (:47) fails, and the
|
||||
fall-through at :52 returns `None` because both are arrays. Probe:
|
||||
`assert_same_shape({"a":[1,2,3]}, {"a":[1]})` returns `Ok(())`. The module doc (:5) and
|
||||
RUST.md:2969 promise "an array an array of the same length". This is unreachable today. No
|
||||
deserializer in `draft/schema.rs` or `domain/sheet/schema.rs` changes an array's length:
|
||||
`non_empty_strings`, `stable_ids` and `at_least_three` keep every element, and `Point` reads
|
||||
an exact 2-tuple. So it is a latent gap: a future filtering or deduplicating deserializer
|
||||
would drop elements without the gate noticing.
|
||||
Fix: `(Value::Array(w), Value::Array(t)) if w.len() != t.len() => Some(format!("expected
|
||||
{} elements at {path}, received {}", t.len(), w.len()))` ahead of the equal-length arm.
|
||||
|
||||
**L3. `write_exclusive` accepts a folder that appeared after the claim.
|
||||
`import_equine.rs:280-284`, docstring :261-263, RUST.md:2902.**
|
||||
The docs say an entry that appeared since the claim, "file, folder or symbolic link, is
|
||||
refused". Line 282 accepts any existing real directory. It has to, for the folders this run
|
||||
created itself (`legacy-import/` for its second file, `assets/`). This is harmless: it stays
|
||||
inside the claimed root, a symlink is still refused (`symlink_metadata`), and files stay
|
||||
`create_new`, so nothing is overwritten. It falls within the declared single-user residual.
|
||||
Fix: correct the docs to "a file or a symbolic link". Or record the folders this run created
|
||||
and refuse any other one that already exists.
|
||||
|
||||
**L4. The seed guard is tested as a predicate, never through its call site.
|
||||
`import_equine.rs:223` (wiring), :365-380 (test).**
|
||||
The unit test pins `inside()` on literal paths. No test runs `claim_destination` against a
|
||||
seed. A regression at the call site would pass the whole suite: comparing the lexical `out`
|
||||
instead of the canonical `parent`, or dropping :222-225. That guard backs the owner's first
|
||||
requirement. The probes above show today's wiring is correct.
|
||||
Fix: make the seed a parameter of `claim_destination` (the binary passes `SEED`). Then test it
|
||||
on a temporary "seed" folder: a parent symlinked into it, the seed itself, a new folder under
|
||||
it. This never touches the real seed.
|
||||
|
||||
**L5. Replacing a non-object intermediate records `before: null`.
|
||||
`fields.rs:126-139` (replace at :131-133), `read_field` :111-122, used at `ops.rs:116-118`.**
|
||||
`set name.fr.x` turns the string `name.fr` into `{x: …}`. `read_field` returns `None` for
|
||||
that path, so the record says `before: null` and the original string appears in no record.
|
||||
The TypeScript does the same (`fields.ts:62-80`). The final document is safe: the strict gate
|
||||
refuses the stray object unless a later op overwrites the field. In that case the CHANGELOG
|
||||
trail lacks the original value. This is the family of the declared inherited-member row
|
||||
(§11.12) and is not declared.
|
||||
Fix: declare it in the §11.12 table, or refuse a write through an existing non-object
|
||||
intermediate (always an author error).
|
||||
|
||||
**L6. New lint allow in a test (context). `crates/core/tests/curation_draft.rs:7`.**
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` on an
|
||||
integration test that reads fixtures and runs the built binary. This is not a security-scanner
|
||||
suppression, and product code keeps the lint. No `nosemgrep`, `nosec`, `eslint-disable`,
|
||||
`ts-ignore` or `noqa` was added anywhere in the diff.
|
||||
|
||||
**L7. Pre-existing lint allow (context). `crates/core/src/bin/import_equine.rs:15`.**
|
||||
`#![allow(clippy::disallowed_methods)]` was already present at `5e114da` (line 11). This file
|
||||
is the one place of the tool allowed the file system.
|
||||
|
||||
## Checklist results
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Hardcoded secret, token, key, auth URL (diff incl. docs, `.claude`, oracles) | none |
|
||||
| 2 | SQL by formatting | `seed.rs`: fixed and pipeline values through `sql_text` (quotes doubled), the document in `$equine_doc$` with the tag refused (:91); no injection path. Nothing new |
|
||||
| 3 | innerHTML / eval / unsafe render | none in code (prose in memory files only) |
|
||||
| 4 | Endpoint without authz | N/A (local tool, no endpoint) |
|
||||
| 5 | Internals surfaced to a user | N/A (the operator's own CLI) |
|
||||
| 6 | Secret / PII in a log | none |
|
||||
| 7 | Tracked `.env` / credentials | `.env.example` only |
|
||||
| 8 | guardrails `scan.py --domain api,supabase` | 0 FAIL; 1 WARN `SB-POLICY-NOT-OWNER` at `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the delta, not counted; 1 MANUAL |
|
||||
| 9 | New suppressions (anti-gaming) | none un-gated; L6, L7 context |
|
||||
| 10 | `--out` containment | passes every probe; L3, L4 |
|
||||
| 11 | Read containment | L1 |
|
||||
| 12 | Shape and strictness gate | Zod's two refusals reproduced; L2 latent |
|
||||
| 13 | Depth and resource caps | 16 MiB reads, serde depth 128, 8 field segments, document depth 64: all hold under probes |
|
||||
| 14 | Author data loss, audit trail | NUL keys refused, marks stripped; L5 |
|
||||
|
||||
## Owner requirements
|
||||
|
||||
- Official seed never modified automatically: met. `--out` under or at the seed is refused
|
||||
on canonical paths (8 probe paths), `--check` writes nothing, the binary test aims only at
|
||||
`CARGO_TARGET_TMPDIR`, and the TypeScript generator writes oracles only under
|
||||
`MAKE_R5_ORACLES=1`. Test gap: L4.
|
||||
- `--check` writes nothing: met. Its code path holds only `symlink_metadata`, `read`,
|
||||
`read_dir`.
|
||||
- `--out` never deletes or overwrites: met. No remove call; `create_dir` and `create_new`.
|
||||
- Dangerous paths and invalid destinations refused: met (empty, `..`, missing parent, `/`,
|
||||
symlink with `/` or `/.`, file, non-empty folder, inside the seed).
|
||||
- No unintended access through symbolic links: met on the write side (outside the declared
|
||||
residual window). Partly met on the read side: L1.
|
||||
- Strict changeset validation: met. L2 is latent.
|
||||
- Depth and resource caps: met.
|
||||
- Author data never dropped silently: met for the document. L5 concerns the record trail only.
|
||||
|
||||
## Declared residuals (RUST.md §11.13) checked
|
||||
|
||||
- Memory: no new cap. One correction: the Rust records clone the whole op per record
|
||||
(`records.rs:70`), while the TypeScript shared `ctx.op` (`types.ts:41-48`). A subtree `set`
|
||||
therefore holds its value twice per record, against once in the TypeScript: about twice the
|
||||
TypeScript, so "as the TypeScript's clones would" (RUST.md:3032) is low by about 2x. The
|
||||
conclusion (unbounded, over 2 GB) stands. Not material.
|
||||
- Depth walk after every op (`apply.rs:110`, :126-139): matches the code. Not re-timed.
|
||||
- `--out` check-then-use window: consistent. Reads have the same window (L1).
|
||||
- Lossy UTF-8 (`import_equine.rs:79-81`) and repeated keys (`schema.rs:387-389`):
|
||||
consistent.
|
||||
|
||||
## Permission note
|
||||
|
||||
One probe command was refused by the permission system. It was a compound command: write the
|
||||
shape probe, `cd` into the scratch crate, export `CARGO_TARGET_DIR`, source
|
||||
`scripts/rust-env.sh`, then `cargo build`. No rule text came back. I did not retry that form
|
||||
and did not source the env file. The probe file was written alone. The build then ran as a
|
||||
plain `cargo build --manifest-path <scratch crate>`, with `cargo` already on PATH, and the
|
||||
permission system allowed it.
|
||||
@@ -0,0 +1,170 @@
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 35 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] author-data-silently-replaced (non-array `children` of an added item) — crates/core/src/curation/draft/ops.rs:252 + crates/core/src/curation/draft/targets.rs:246
|
||||
- [LOW] check-lists-through-top-level-folder-link — crates/core/src/bin/import_equine.rs:193 (listed, :164)
|
||||
- [LOW] seed-containment-is-path-based (bind-mount alias) — crates/core/src/bin/import_equine.rs:230 (inside, :253)
|
||||
- [LOW] declared-depth-walk-cost-understates-cap-bound-worst-case — crates/core/src/curation/draft/apply.rs:110 / docs/development/RUST.md:3046
|
||||
- [LOW] unit-test-fixture-in-shared-temp-with-fixed-name — crates/core/src/bin/import_equine.rs:383
|
||||
- [LOW] context: clippy purity lint allowed in a new test file (project convention, not a scanner suppression) — crates/core/tests/curation_draft.rs:7
|
||||
- [LOW] context: seed SQL rendered by string formatting from pipeline values — crates/core/src/curation/draft/seed.rs:29
|
||||
PROOF: semgrep 129 rules on 35 files → 0 findings; checklist 9 checks → 7 findings (1 MEDIUM, 6 LOW)
|
||||
```
|
||||
|
||||
# R5.3 draft, security gate (fresh, final 4)
|
||||
|
||||
Branch `feature/rust-r5-pipeline-core`, HEAD `6c0fe88`, scope `git diff 5e114da..HEAD`
|
||||
(35 files, 11 commits), with attention to `86273b6`. Read whole: `bin/import_equine.rs`,
|
||||
`curation/draft/{shape,ops,apply,schema,targets,fields,records,rename,stats,migration_map,seed,pipeline,mod}.rs`,
|
||||
`curation/mod.rs`, `scripts/import-equine.sh`, the `package.json` line, the binary and
|
||||
integration tests that touch the file system, the oracle generator's write guard
|
||||
(`MAKE_R5_ORACLES=1` only). Every probe ran in
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/sec4.j5fliv`
|
||||
on a copy of the seed, through a probe binary that is `import_equine.rs` byte for byte except
|
||||
the `SEED` constant (pointed at the copy). Nothing ran `--out` against the repository, nothing
|
||||
was deleted, `git status` is clean after the run.
|
||||
|
||||
## Semgrep
|
||||
|
||||
`semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten
|
||||
--metrics=off` on the 35 existing files of the diff: 129 rules run, 0 findings, 0 errors.
|
||||
The rulesets carry almost no Rust rules; the Rust code was covered by reading and probing.
|
||||
|
||||
## Checklist
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Hardcoded secret / token / key / auth URL | none (grep over the 35 files) |
|
||||
| SQL by string interpolation | `seed.rs` renders the seed file by formatting: LOW context below, not a runtime query |
|
||||
| Unsanitized render (innerHTML, eval…) | not applicable (Rust CLI and library, no DOM) |
|
||||
| Sensitive endpoint without authz | not applicable (no endpoint) |
|
||||
| Stack trace / internal path to an end user | not applicable (errors go to the local developer's stderr) |
|
||||
| Secret / PII in a log | none (stdout/stderr carry paths, counts and refusals) |
|
||||
| Tracked `.env` / credential file | only `.env.example` tracked |
|
||||
| Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff, dropped), 1 MANUAL |
|
||||
| New suppressions (anti-gaming) | no `nosemgrep`, `nosec`, `eslint-disable`, `ts-ignore`, `noqa`; one clippy allow in a new test file (LOW context below) |
|
||||
|
||||
## Owner requirements, verified
|
||||
|
||||
- Seed never modified automatically: every read goes through `read()` (canonical path inside
|
||||
the canonical seed, regular file, 16 MiB cap). Probed: `curation/` as a folder link out of
|
||||
the seed and `changes.json` as a file link out of it are both refused (`leads outside the
|
||||
seed folder`); a folder link that stays inside the seed is accepted.
|
||||
- `--check` writes nothing: only `canonicalize`, `metadata`, `read`, `read_dir` on that path.
|
||||
Probe copy: 12 files, 0 differences, nothing new under the fixture seed (`find -newer`).
|
||||
- `--out` refusals, probed on the fixture seed: a link to the seed and to `draft/` (as the
|
||||
destination and as its parent), a link chain `p/q` ending in a seed link, the seed path
|
||||
itself plus `/new`, `legacy-import/new`, `INIT_CWD` set inside the seed, `..`, an empty path,
|
||||
`/`: all refused, nothing created. A relative `INIT_CWD` is ignored. A valid destination
|
||||
gets 12 files identical to the seed (`diff -rq`), a second run on it is refused `not empty`.
|
||||
Files are `create_new`, folders `create_dir` tracked in `created`, so an entry that appears
|
||||
after the claim is refused. File names are pipeline constants.
|
||||
- Strict changeset validation, depth and resource caps: `deny_unknown_fields`, the `op`
|
||||
discriminator, non-empty strings, the shape comparison (now with array length), reserved
|
||||
NUL keys at parse and in field paths, 8 field segments, serde_json's 128 parse depth, 64
|
||||
document levels after each op (iterative walk), 16 MiB per file. Recursive walks
|
||||
(`reserved_key`, `first_mismatch`, `strip_marks`, `walk`, `touch_tree`, `js_deep_equal`) are
|
||||
all bounded by the 128 parse depth or the 64-level check.
|
||||
- `86273b6`: `assert_object_children` refuses a non-object entry of a `children` or
|
||||
`categories` list (explicit stack); the length arm of `first_mismatch` sits before the
|
||||
element arm; `claim_destination(typed, seed)` with its fixture test refuses before any
|
||||
`create_dir`. All correct. One gap left next to the first: MEDIUM below.
|
||||
|
||||
## Findings
|
||||
|
||||
### MEDIUM: a non-array `children` on an added item is silently replaced by `[]`
|
||||
|
||||
`crates/core/src/curation/draft/ops.rs:252` (`assert_object_children` skips a `children`
|
||||
value that is not an array) and `crates/core/src/curation/draft/targets.rs:246`
|
||||
(`children_path` overwrites any present non-array `children` with `[]`). An `add` whose value
|
||||
carries `"children"` as an object, a string or `null`, followed by an `add` (or a `move`)
|
||||
into that item, is accepted: the strict gate never sees the author's value, which is gone
|
||||
from `document.json` and the migration map. Probed on the real equine changeset plus one
|
||||
level-A change: `"children": {"author-note": {"id": "kept", ...}}` (a whole nested finding
|
||||
written as an object instead of an array), `"children": "author text"` and
|
||||
`"children": null` all give `probe-node.children = [probe-child]`, exit 0, 12 files written.
|
||||
Only the add record's `after` keeps the value; no R5.3 output shows it (the CHANGELOG is
|
||||
R5.4). Parity with the TypeScript `childrenOf`, and not declared: §11.12 says "a lean leaf
|
||||
gets `children: []` appended on demand", which covers an absent key, not a present value
|
||||
replaced. This breaks the owner's "author data never dropped silently" through a plausible
|
||||
typo (object in place of array). Non-blocking: no security boundary is crossed.
|
||||
Hint: in `assert_object_children`, refuse a present `children` or `categories` key whose
|
||||
value is not an array (same message family, same place as the entry check), or make
|
||||
`children_path` refuse a present non-array value and append `[]` only when the key is absent.
|
||||
Declare the split in the §11.12 table with a frozen TypeScript case.
|
||||
|
||||
### LOW: `--check` lists through a top-level folder link
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:193` calls `listed(&target, ...)` on
|
||||
`SEED/legacy-import` or `SEED/draft` without checking the folder itself: `read_dir` follows
|
||||
it when it is a link. Probed with `legacy-import` pointing out of the seed: the reads are
|
||||
refused (`missing: ...`), but `not produced: private-name.txt` and `not produced:
|
||||
sub/deep.txt` print the names of files outside the seed, subfolders included. Read-only,
|
||||
names only, same user. §11.12's "`--check` lists a link without walking into it" holds for
|
||||
entries, not for the generated folder itself.
|
||||
Hint: canonicalize the target and require `inside(&canonical, &seed)` before listing, or
|
||||
refuse a target whose `symlink_metadata` is a link.
|
||||
|
||||
### LOW: the seed guard is path-based, a mount alias passes it
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:230` compares canonical paths (`starts_with`).
|
||||
`canonicalize` resolves links, not mounts: if the seed (or the repository) is reachable
|
||||
through a second path (bind mount, the same NAS share mounted twice), `--out
|
||||
<alias>/.../equine/new` passes `inside` and creates a new folder inside the official seed.
|
||||
Only new files (never an overwrite, never a delete); the user must type the alias. Reasoned,
|
||||
not probed (a mount needs privileges).
|
||||
Hint: also compare `(dev, ino)` (`std::os::unix::fs::MetadataExt`) of the canonical parent and
|
||||
each of its ancestors with the seed folder's; a bind mount root shares the source's pair.
|
||||
|
||||
### LOW: the declared depth-walk cost understates the cap-bound worst case
|
||||
|
||||
`crates/core/src/curation/draft/apply.rs:110` runs `assert_depth` over the whole document
|
||||
after every op; RUST.md:3046 declares it with one sample ("a 2.4 MB changeset takes about
|
||||
10 s"). The sample is right (probe: 2.3 MB, one 1 M-node add then 2,000 sets, 12.4 s,
|
||||
455 MB). The worst case inside the 16 MiB read cap is two orders larger: 8.2 MB, one 4 M-node
|
||||
add then 1,000 sets, 35.7 s and 1.7 GB, so about 30 ms per op, and the remaining 8 MB holds
|
||||
about 94,000 more ops: roughly 45 minutes of CPU before the final gate refuses. Twenty
|
||||
thousand plain sets on the real document take 0.9 s, so ordinary changesets are not affected.
|
||||
Local developer tool, interruptible. The declaration is not wrong, it is incomplete.
|
||||
Hint: state the cap-bound figure in §11.13, or bound the work: a total node cap on the
|
||||
document after each op (also bounds memory), or a depth check limited to the subtree the op
|
||||
wrote.
|
||||
|
||||
### LOW: unit-test fixture at a fixed name in the shared temp folder
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:383`: `env::temp_dir().join("faunosteo-import-equine-claim")`
|
||||
with `create_dir_all`. On a multi-user host another account can pre-create or link that
|
||||
name. The test only runs refusals, so the worst outcome is a failed test or an empty folder
|
||||
created where the link points. The integration test already uses the private
|
||||
`CARGO_TARGET_TMPDIR`.
|
||||
Hint: put the fixture under the target folder (for example
|
||||
`concat!(env!("CARGO_MANIFEST_DIR"), "/../../target/...")`) or a per-run name created
|
||||
exclusively.
|
||||
|
||||
### LOW (context): clippy purity lint allowed in a new test file
|
||||
|
||||
`crates/core/tests/curation_draft.rs:7` `#![allow(clippy::disallowed_methods,
|
||||
clippy::unwrap_used, clippy::expect_used)]`. A determinism and robustness lint (no I/O, no
|
||||
clock in the core), not a security scanner suppression. The same line heads the ten other
|
||||
core test files. The binary's `#![allow(clippy::disallowed_methods)]` was already present at
|
||||
`5e114da`. Not blocking.
|
||||
|
||||
### LOW (context): seed SQL rendered by string formatting
|
||||
|
||||
`crates/core/src/curation/draft/seed.rs:29` (`sql_text`) and `:90` (`document_literal`). Every
|
||||
interpolated value is a pipeline constant, a sha256 hex, a derived uuid or a byte count;
|
||||
quotes are doubled (correct with `standard_conforming_strings` on, the PostgreSQL default);
|
||||
the document is embedded in a `$equine_doc$` literal and refused if it holds the tag; the
|
||||
formatted JSON always ends with `}`, so no tag can form at the closing boundary. This
|
||||
generates a reviewed seed file, it is not a runtime query. No action.
|
||||
|
||||
## Residual risks declared in RUST.md §11.13
|
||||
|
||||
Memory without a cap beyond 16 MiB per file: confirmed (1.7 GB from an 8.2 MB file), in line
|
||||
with the declaration. Single-user check-then-use window on reads and `--out`: confirmed as
|
||||
described, every file stays `create_new`. Lossy UTF-8 and repeated keys: as declared. The
|
||||
quadratic walk: LOW above.
|
||||
@@ -0,0 +1,141 @@
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 37 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] seed-guard-identity-coverage — crates/core/src/bin/import_equine.rs:263-273 (claim at :236; claim in docs/development/RUST.md §11.13)
|
||||
- [LOW] quadratic-id-comparisons — crates/core/src/curation/draft/ops.rs:355-383, :404-408; schema.rs:369-380
|
||||
- [LOW] sql-built-by-formatting (checked, not injectable) — crates/core/src/curation/draft/seed.rs:29-135
|
||||
- [LOW] new-lint-allow-in-test (context, not a scanner suppression) — crates/core/tests/curation_draft.rs:7
|
||||
PROOF: semgrep 129 rules on 37 files → 0 findings; checklist 14 checks → 4 findings
|
||||
|
||||
# Security gate, R5.3 curated draft, fifth final round (fresh, blind)
|
||||
|
||||
- Repository: /home/bchanot/Documents/faunosteo, branch `feature/rust-r5-pipeline-core`, HEAD `3ece1a6`.
|
||||
- Scope: `git diff 5e114da..HEAD` (37 files, all present), focus on `07d6307`.
|
||||
- Product files read in full: `crates/core/src/bin/import_equine.rs`, `crates/core/src/curation/text.rs`,
|
||||
`crates/core/src/curation/draft/{apply,fields,migration_map,mod,ops,pipeline,records,rename,schema,seed,shape,stats,targets}.rs`,
|
||||
`crates/core/src/curation/mod.rs` (diff), `crates/core/Cargo.toml`, `package.json`, `scripts/import-equine.sh`;
|
||||
test files and the TypeScript oracle generator inspected for file system and process calls.
|
||||
|
||||
## Semgrep
|
||||
|
||||
`EIO_BACKEND=posix semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten --metrics=off --json <37 files>`:
|
||||
129 rules run (multilang 41, json 3, ts 84, bash 1), 37 targets, 0 findings, 0 errors. These packs carry no Rust
|
||||
rule: the Rust code is covered by the checklist and the probes below, not by semgrep.
|
||||
|
||||
## Checklist (14 checks)
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Hardcoded secret, token, key, auth URL (diff incl. docs, `.claude`, oracle JSON) | none (oracle holds only `https://example.org/a`) |
|
||||
| 2 | SQL built by string formatting | seed.rs generator, quoting verified: LOW 3 |
|
||||
| 3 | Unsanitized render, `eval`, `unsafe` | none (no web render, no `unsafe` in scope) |
|
||||
| 4 | Sensitive endpoint without authz | not applicable (no endpoint; dev CLI and library) |
|
||||
| 5 | Stack trace, internal path, DB error to an end user | none (CLI messages go to the developer running the tool) |
|
||||
| 6 | Secret or PII in a log | none (stdout/stderr carry paths, counts, refusals) |
|
||||
| 7 | Tracked `.env` or credential file | none in the diff (`.env.example` only, pre-existing) |
|
||||
| 8 | Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`) outside the diff, filtered out; 1 MANUAL not reported |
|
||||
| 9 | New suppressions (anti-gaming) | no `nosemgrep`, `nosec`, `eslint-disable`, `ts-ignore`, `noqa`; one clippy allow in a test file: LOW 4 |
|
||||
| 10 | `--check` writes nothing | proven: strace of the built binary, 28 `openat`, none with `O_WRONLY`/`O_RDWR`/`O_CREAT`/`O_TRUNC`, no `mkdir`/`unlink`/`rename`/`truncate`/`symlink`/`link`/`chmod`; seed file list, sizes and mtimes identical before and after; `git status` clean; output 12 files, 0 differences |
|
||||
| 11 | `--out` destinations | 14 probes in a fresh `mktemp -d` under the scratchpad, all as designed (below) |
|
||||
| 12 | Seed guard of `--out` (canonical path + device/inode) | covers the seed folder and its aliases; not an alias of a seed subfolder: LOW 1 |
|
||||
| 13 | Resource caps (depth, value bound, non-list children, memory, CPU) | depth, value bound and children refusals hold; memory matches the §11.13 declaration; CPU has an undeclared quadratic term: LOW 2 |
|
||||
| 14 | `js_insert` / `js_normalize` | behave as `JSON.parse`/assignment (index keys ascending first, `"01"` and `"4294967295"` named, existing key keeps its place, integers past 2^53 rounded, past u64 already f64); recursion bounded by the parser's 128 levels; only called on parsed values and on field-path writes |
|
||||
|
||||
### Probes run (scratchpad only, never `--out` into the repository, no recursive delete)
|
||||
|
||||
Binary built with `CARGO_TARGET_DIR` in the scratchpad (`cargo build --release --locked --offline -p faunosteo-core --features curation --bin import-equine`);
|
||||
a throwaway library probe linking `faunosteo-core` with `curation`.
|
||||
|
||||
- `--out`: absent folder (written, 12 files); same folder again (`not empty, nothing written`); empty folder with trailing `/` (written);
|
||||
parent through a symlink to another scratch folder (parent resolved, canonical path printed before the first write); final symlink,
|
||||
with `/` and as a dangling link (`a symbolic link, nothing written`); `./new/.` (normalised, written); `""` (usage error); `/`
|
||||
(`not a destination folder`); relative path with `INIT_CWD` unset, relative (ignored) and absolute (typed from it). No entry outside
|
||||
the probe folder appeared.
|
||||
- Non-list `children` / `categories` in an added value: object, `null`, string, nested object, nested `null` entry, each refused at the add.
|
||||
- Depth: an added or set value nested 60 to 122 levels is refused by the 64-level bound after the op; 122+ is refused by the parser's
|
||||
limit; a 58-level `children` chain (the `touch_tree` recursion) refused by the bound, no stack overflow.
|
||||
- Memory: one `set` of a 976 KiB array peaks at 156 MB RSS, a 3.9 MiB array at 610 MB (about 156 times the input, refused by the
|
||||
200,000-value bound after the op). A 16 MiB changeset therefore asks for about 2.5 GB in one op, in line with the §11.13 row
|
||||
("memory in proportion to ops × items × value size", "more than 2 GB"): declaration accurate, not reported.
|
||||
|
||||
## Findings (all non-blocking)
|
||||
|
||||
### LOW 1: the identity guard covers the seed folder, not an alias of one of its subfolders
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:263-273` (`inside_by_identity`, called at `:236`). The guard compares the (device, inode) of
|
||||
the seed folder with every ancestor of the canonical parent. A bind mount of the seed itself (or of an ancestor) is refused, as
|
||||
intended. A bind mount of a seed subfolder, for example `draft/` mounted at `/mnt/d`, has the inode of `draft/`, not of the
|
||||
seed: `--out /mnt/d/x` passes both checks and creates `draft/x/legacy-import/…` and `draft/x/draft/…` inside the official seed.
|
||||
The same applies to a network share mounted twice with distinct superblocks (distinct `st_dev`). Traced by reading; not probed
|
||||
(a mount needs privileges and is the user's to run).
|
||||
|
||||
Why only LOW: it needs a mount the owner made by hand; every write is `create_dir`/`create_new` into a new or empty folder, so
|
||||
nothing is overwritten or deleted, and the stray folder shows up in `--check` as `not produced`. What is materially off is
|
||||
the wording: the code comment says "a share mounted twice is caught too" and RUST.md §11.13 says "a second mount path of
|
||||
the seed is refused too". Both hold for the seed folder only.
|
||||
|
||||
Suggested fix: collect the (device, inode) of every folder of the seed tree once (a few dozen) and refuse a parent whose
|
||||
ancestor matches any of them; or keep the code and narrow the comment and §11.13 to "a second mount path of the seed folder or
|
||||
of an ancestor".
|
||||
|
||||
### LOW 2: id comparisons are quadratic, a CPU cost the residual-risk declaration does not name
|
||||
|
||||
- `ops.rs:355-372` `assert_declared`: `declared.contains` on a `Vec` costs O(ids × declared), per cite op and per change
|
||||
(`apply.rs:104`). The TypeScript used a `Set` here (`ops.ts:185`).
|
||||
- `ops.rs:375-383` `distinct`: O(n²), called once per scope entry inside the cite loop (`:404`). The TypeScript used `new Set`.
|
||||
- `ops.rs:406` `before.contains` per added id: O(ids × existing), same as the TypeScript's `includes`.
|
||||
- `schema.rs:369-380` `duplicates`: O(n²) on change and reference ids, same order as the TypeScript's `indexOf`.
|
||||
|
||||
Measured (release build): `parse_references` takes 0.24 s at 20,000 references, 1.9 s at 40,000, 6.0 s at 80,000 and 31 s at
|
||||
160,000 (10.5 MiB). A subtree cite of every reference on 5 items takes 1.16 s at 16,000 ids and 4.7 s at 32,000, where the
|
||||
200,000-value bound then refuses. The value bound caps the apply phase near 30,000 references, so a hostile 16 MiB pair of
|
||||
files stays in minutes, the same order as the declared "a 16 MiB changeset of tiny ops could take minutes". The §11.13
|
||||
CPU sentence names only the bound walk.
|
||||
|
||||
Suggested fix: a `HashSet<&str>` for `assert_declared`, `distinct` and the `duplicates` scan (output order kept by walking
|
||||
the original list). Or add one clause to §11.13 ("id comparisons are quadratic, minutes at the 16 MiB cap").
|
||||
|
||||
### LOW 3: seed SQL built by string formatting, checked and not injectable
|
||||
|
||||
`crates/core/src/curation/draft/seed.rs:29-135`. A checklist hit on "SQL built by interpolation", but this code writes a
|
||||
generated seed file (run by `supabase db reset` and the PGlite harness). It is not a runtime query, and no untrusted text
|
||||
reaches it in an injectable form:
|
||||
|
||||
- the `sql_text` values are pipeline constants, the sheet id from `sheet_id_for(EQUINE_SPECIES_ID, EQUINE_SHEET_VERSION)`,
|
||||
sha256 hex, a uuid from the sha256 and the fixed `horse.svg`, all quoted by doubling `'` (`standard_conforming_strings` is on
|
||||
by default in both runners);
|
||||
- the authored document is dollar-quoted, and a text holding `$equine_doc$` is refused (`:91`). The text is `format_json`
|
||||
of an object, so it starts with `{` and ends with `}` after `trimEnd`: no tag can form at a junction, and a `$`
|
||||
escape is decoded and re-emitted as a literal `$`, then caught.
|
||||
|
||||
Same construction as the TypeScript. No action needed. If touched later, keep the tag check on the final text.
|
||||
|
||||
### LOW 4 (context): new lint allow in an integration test
|
||||
|
||||
`crates/core/tests/curation_draft.rs:7` `#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`.
|
||||
This allows the project's no-I/O/no-clock lint and the unwrap lints in a test file, as every other core test file does
|
||||
(`anatomy_*`, `pdf_*`, `curation_format.rs`, `curation_legacy.rs`, …). It does not silence a security scanner, so it is not
|
||||
a gated suppression. The binary's `#![allow(clippy::disallowed_methods)]` (`import_equine.rs:15`) was already present at `5e114da`.
|
||||
|
||||
## Owner requirements, checked
|
||||
|
||||
- The official seed is never modified automatically: `--check` is proven read-only. The `--out` seed guard holds for the seed
|
||||
folder and its aliases (exception: LOW 1).
|
||||
- `--out` never deletes (no remove call in the binary) and never overwrites (`create_dir`, `create_new`, the claimed folder
|
||||
re-checked before each file). Refusals probed for an empty path, `..`, a missing parent, a symlink (trailing `/`, `/.`,
|
||||
dangling), a non-folder and a non-empty folder.
|
||||
- Strict changeset validation: strict shapes plus the shape comparison; NUL keys refused at parse and in field segments;
|
||||
non-list or non-object children refused at the add, never replaced (`targets.rs:245-256`).
|
||||
- Depth and resource caps: parser 128, field path 8 segments, document 64 levels and 200,000 values after each op, files
|
||||
under 16 MiB. Memory matches the declaration; the undeclared CPU term is LOW 2.
|
||||
- Author data is never dropped silently in the final document: an overwritten value is recorded as `before`. A path through
|
||||
a non-object replaces it (`before: null`), which matches the TypeScript (`isItem` excludes arrays too) and is declared in §11.12.
|
||||
|
||||
## Not done
|
||||
|
||||
- Semgrep has no Rust rules in the pinned packs. Rust coverage comes from the reading, the checklist and the probes.
|
||||
- Mount aliases (LOW 1) were traced by reading, not reproduced.
|
||||
- The test suites were not run (the verifier's job); only the release binary and the probe were built, outside the repository.
|
||||
@@ -0,0 +1,151 @@
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 38 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] seed-guard-destination-identity — crates/core/src/bin/import_equine.rs:236 (destination branch :243-253)
|
||||
- [LOW] cite-membership-linear (doc precision, RUST.md §11.13) — crates/core/src/curation/draft/ops.rs:401-405
|
||||
- [LOW] sql-built-by-formatting (checked, not injectable) — crates/core/src/curation/draft/seed.rs:29-135
|
||||
- [LOW] new-lint-allow-in-test (context, not a scanner suppression) — crates/core/tests/curation_draft.rs:7
|
||||
PROOF: semgrep 163 rules on 38 files → 0 findings; checklist 14 checks → 4 findings
|
||||
|
||||
# Security gate, R5.3 curated draft, sixth final round (fresh, blind)
|
||||
|
||||
- Repository: /home/bchanot/Documents/faunosteo, branch `feature/rust-r5-pipeline-core`, HEAD `c6e0d85`.
|
||||
- Scope: `git diff 5e114da..HEAD`, 38 files, all present. Focus on `f245b3e`.
|
||||
- Product files read in full: `crates/core/src/bin/import_equine.rs`,
|
||||
`crates/core/src/curation/draft/{apply,fields,migration_map,mod,ops,pipeline,records,rename,schema,seed,shape,stats,targets}.rs`,
|
||||
`crates/core/src/curation/{mod,text}.rs`, `scripts/import-equine.sh`, and the diffs of `crates/core/Cargo.toml` and
|
||||
`package.json`. The test files and the TypeScript oracle generator were checked for file system and process calls.
|
||||
|
||||
## Semgrep
|
||||
|
||||
`EIO_BACKEND=posix semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten --metrics=off --quiet --json <38 files>`:
|
||||
163 rules run (javascript 73, generic 57, typescript 11, python 6, package managers 5, json 3, java 3, scala 3, bash 1,
|
||||
ruby 1), 38 files scanned, 0 findings, 0 errors. The pinned packs hold no Rust rule, so the Rust code is covered by the
|
||||
reading, the checklist and the probes below.
|
||||
|
||||
## Checklist (14 checks)
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Hardcoded secret, token, key, auth URL (whole diff incl. docs, `.claude`, oracle JSON) | none |
|
||||
| 2 | SQL built by string formatting | `seed.rs` generator, quoting verified: LOW 3 |
|
||||
| 3 | Unsanitized render, `eval`, `unsafe`, process spawn | none in scope |
|
||||
| 4 | Sensitive endpoint without authz | not applicable (dev CLI and library, no endpoint) |
|
||||
| 5 | Stack trace, internal path, DB error to an end user | none (CLI messages go to the developer running the tool) |
|
||||
| 6 | Secret or PII in a log | none (stdout and stderr carry paths, counts, refusals) |
|
||||
| 7 | Tracked `.env` or credential file | none (`.env.example` only, pre-existing, outside the diff) |
|
||||
| 8 | Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN `SB-POLICY-NOT-OWNER` at `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff, filtered out; 1 MANUAL not reported |
|
||||
| 9 | New suppressions (anti-gaming) | no `nosemgrep`, `nosec`, `eslint-disable`, `ts-ignore`, `noqa`; one clippy allow in a test file: LOW 4 |
|
||||
| 10 | `--check` writes nothing | built binary run: 12 files, 0 differences, 6 R5.4 files listed as pending; seed file list and mtimes identical before and after; `git status` clean |
|
||||
| 11 | `--out` destinations | 11 probes in a fresh `mktemp -d` under the scratchpad, all as designed (below) |
|
||||
| 12 | Seed guard of `--out` (canonical path, then device/inode over the seed tree) | holds for every parent path; the destination folder itself is not identity-checked: LOW 1 |
|
||||
| 13 | `__proto__` field segments | refused at every position, before any write (probe below) |
|
||||
| 14 | Resource caps and id comparisons | parser depth, 8 segments, 64 levels, 200,000 values hold; hash sets in `distinct`, `assert_declared`, `duplicates` keep the old semantics; one linear membership left in `cite`: LOW 2 |
|
||||
|
||||
### Probes (scratchpad only; no `--out` into the repository; no recursive delete)
|
||||
|
||||
The binary was built with `CARGO_TARGET_DIR` in the scratchpad (`cargo build --release --locked --offline -p faunosteo-core --features curation --bin import-equine`).
|
||||
Two throwaway probes ran there: one linking `faunosteo-core` with `curation`, and one compiling lines 213-293 of
|
||||
`import_equine.rs` verbatim against a scratch seed tree.
|
||||
|
||||
- `split_field`: `__proto__`, `name.__proto__`, `name.__proto__.x` and `name.fr.__proto__` are all refused with
|
||||
`reserved key "__proto__"`. `name.__proto__ ` (trailing space), `name.__PROTO__`, `name.constructor` and `name.prototype` are
|
||||
plain keys, as in JavaScript on a plain object. A NUL segment, 9 segments and an empty segment are refused.
|
||||
`apply_changeset` with `set name.__proto__…` followed by an overwriting `set name` refuses at `CHG-001 op 0` in all four
|
||||
variants. `is_settable` calls `split_field` before `item_at_mut`, so nothing is touched or written first.
|
||||
- Identity guard (verbatim copy, scratch tree `seed/{a/b/c,d}` plus a link to `/` and a link loop inside it): the walk
|
||||
records 5 identities and follows neither link. `seed/a/b/new`, `seed/new`, a parent link into `seed/a/b` and a parent link
|
||||
to `seed` are all refused as `inside the seed folder`, with nothing created. A final link, a trailing `/` and a trailing
|
||||
`/.` are refused as `a symbolic link`. A sibling folder is accepted. With one seed folder made unreadable (`chmod 000`,
|
||||
restored afterwards), every claim is refused: the walk fails closed.
|
||||
- Real binary `--out`: absent folder (12 files written, then `diff -r` against the seed shows only the 6 R5.4 files);
|
||||
the same folder again (`not empty`, exit 2); an empty folder (written); a regular file (`not a folder`); a final link, with
|
||||
`/` and `/.` (`a symbolic link`, exit 2, link target left empty); `empty/../x` (`..` refused); a missing parent; `""`
|
||||
(usage error); a relative path with `INIT_CWD` unset (typed from the current folder, the canonical path printed before the
|
||||
first write). No entry appeared outside the probe folder.
|
||||
- Cite cost (release build): citing 28,000 declared ids twice on one finding takes 0.92 s, four times 2.7 s. The value bound
|
||||
then refuses a larger registry plus its citations.
|
||||
|
||||
## Findings (all non-blocking)
|
||||
|
||||
### LOW 1: the destination folder itself is never identity-checked
|
||||
|
||||
`crates/core/src/bin/import_equine.rs:236` runs `inside_by_identity(&parent, seed)`, which checks the ancestors of the
|
||||
canonical parent. When the destination already exists (`:247-253`), only its emptiness is checked, never its own
|
||||
(device, inode). Suppose an existing empty folder is a second mount path (a bind mount, or a share mounted twice on the
|
||||
same device) of an empty folder of the seed. The claim accepts it, and `write` then creates `legacy-import/…` and
|
||||
`draft/…` inside the seed tree. A mount alias of a non-empty seed folder is refused as `not empty`.
|
||||
|
||||
Why only LOW: this needs an empty folder under the seed (none in the checkout: `find … -type d -empty` returns nothing,
|
||||
and git tracks no empty folder) plus a mount made by hand at the destination. Every write is `create_dir` or `create_new`,
|
||||
so nothing is overwritten or deleted. Also, the §11.13 sentence "a second mount path of the seed or of any of its folders
|
||||
on the same device is refused too" holds for every parent, but not for the destination itself. Traced by reading, not
|
||||
reproduced: a mount needs privileges and is the user's to run.
|
||||
|
||||
Suggested fix: in the existing-folder branch, also refuse when `inside_by_identity(&root, seed)` is true (`root.ancestors()`
|
||||
starts with `root` itself). Alternatively, run the identity check on `root` once it exists, created or not. Or narrow the
|
||||
§11.13 sentence to the ancestors of the destination.
|
||||
|
||||
### LOW 2: one cite comparison is still linear, while §11.13 says cite uses hash sets
|
||||
|
||||
`crates/core/src/curation/draft/ops.rs:401-405`: for each scope entry, every distinct cited id is checked with
|
||||
`before.contains(&Value::from(id))` on the item's existing citations. That costs O(distinct × existing) per item. `distinct`
|
||||
and `assert_declared` are hashed now, but this membership is not, and §11.13 reads "id comparisons use hash sets (cite,
|
||||
declared references, duplicate ids)". Measured: about 0.9 s per repeated op at the largest registry the 200,000-value bound
|
||||
admits (about 28,000 references, each serialised as 6 values). Spread over a subtree, the worst case is about twice that.
|
||||
A 16 MiB changeset therefore stays in minutes, the same order as the declared CPU residual. So this is a precision issue
|
||||
in the declaration, not a new risk.
|
||||
|
||||
Suggested fix: build a `HashSet<&str>` of the string entries of `before` once per item and keep the output order by
|
||||
walking `distinct`. Or reword §11.13 ("cite deduplication and the declared check are hashed; the check against existing
|
||||
citations is linear, bounded by the value bound").
|
||||
|
||||
### LOW 3: seed SQL built by string formatting, checked and not injectable
|
||||
|
||||
`crates/core/src/curation/draft/seed.rs:29-135`. This is a checklist hit on "SQL built by interpolation", but the code
|
||||
writes a generated seed file, not a runtime query. The `sql_text` values are pipeline values: the sheet id from the legacy
|
||||
import, sha256 hex, the asset id derived from it, the fixed file name and the constant change note. All of them are quoted
|
||||
by doubling `'`. The authored document goes into a `$equine_doc$` dollar-quoted literal, and a document text that holds the
|
||||
tag is refused (`:91`). The text is `format_json` of an object, so it begins with `{` and ends with `}` after `trimEnd`, and
|
||||
no tag can form at a junction. Same construction as the TypeScript. No action needed. If this is touched later, keep the
|
||||
tag check on the final text.
|
||||
|
||||
### LOW 4 (context): lint allow in a new integration test
|
||||
|
||||
`crates/core/tests/curation_draft.rs:7` `#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`.
|
||||
Every other core test file carries the same allow, and `clippy.toml` documents it ("Tests and the worker binary allow these
|
||||
lints locally where they own I/O"). It silences no security scanner, so it is not a gated suppression. The binary's
|
||||
`#![allow(clippy::disallowed_methods)]` (`import_equine.rs:15`) was already present at `5e114da`. The core library itself has
|
||||
no file system, environment, clock or process call (grep over `crates/core/src` outside `bin/`).
|
||||
|
||||
## Owner requirements, checked
|
||||
|
||||
- The official seed is never modified automatically. `--check` was run and is read-only. `--out` refuses every parent
|
||||
inside the seed by canonical path and by identity over the whole seed tree (exception: LOW 1, the destination itself).
|
||||
- `--out` never deletes (the binary has no remove call) and never overwrites (`create_dir`, `create_new`, the claimed
|
||||
folder re-checked before each file). It refuses an empty path, `..`, a missing parent, a symbolic link (trailing `/` and
|
||||
`/.` included), a non-folder and a non-empty folder.
|
||||
- Strict changeset validation: strict shapes plus the shape comparison; NUL keys refused at parse and in field segments;
|
||||
`__proto__` segments refused at the op; non-list or non-object children refused at the add.
|
||||
- Depth and resource caps: parser 128 levels, field path 8 segments, document 64 levels and 200,000 values after each op,
|
||||
source files under 16 MiB. The seed walk is iterative and follows no link.
|
||||
- Author data is not dropped silently. An overwritten value is recorded as `before`, and the cases with no faithful reading
|
||||
(`__proto__`, NUL keys, non-list children) refuse.
|
||||
- The residual risks declared in §11.13 (record memory, the per-op bound walk, the single-user check-then-use window, lossy
|
||||
UTF-8 and repeated keys, a share mounted twice on distinct devices) match the code. The exceptions are the two wording
|
||||
points in LOW 1 and LOW 2.
|
||||
|
||||
## Notes (no severity)
|
||||
|
||||
- When the seed walk fails, the refusal reads `inside the seed folder` (`import_equine.rs:287-289`). That fails closed, but
|
||||
the message is misleading. Something like "seed folder unreadable" would point at the real cause.
|
||||
- The doc comment of `crates/core/tests/curation_draft.rs:520-526` lists `__proto__` among the ordinary keys, then says it
|
||||
is refused. It is stale wording in a test.
|
||||
|
||||
## Not done
|
||||
|
||||
- The pinned semgrep packs have no Rust rules: Rust coverage comes from the reading, the checklist and the probes.
|
||||
- LOW 1 was traced by reading. No mount was made.
|
||||
@@ -0,0 +1,125 @@
|
||||
SECURITY — VERDICT: BLOCK(1)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 29 files
|
||||
BLOCKING:
|
||||
1. [HIGH] review-drawings-amplification — crates/core/src/curation/draft/pipeline.rs:209 (bound at :205, projection at :211; amplifier reached through crates/core/src/curation/draft/app_review.rs:33 → crates/core/src/anatomy/layer.rs:108-118) — the 64 MiB review-drawing bound projects only the cleaned drawing repeated per view; every unnamed detail trigger copies the opened view's fr and en names four times into review/app-<view>.svg, so T triggers × a view name of S bytes write 4·T·S bytes unchecked: a 485,140-byte changes.json produced a 131,713,082-byte app-main.svg (no refusal), a 1,202,892-byte changes.json made the run abort on a 1 GiB allocation under a 3 GB cap, about 44 GiB at the 16 MiB read cap — hint: project each view's layer bytes before render_layer (per annotation: fixed overhead + geometry text + 4 × the name it carries, the opened view's names for an unnamed trigger) and refuse past MAX_REVIEW_DRAWINGS_BYTES together with the cleaned-drawing projection; add a test with unnamed triggers opening a long-named view; correct the RUST.md §11.14 row
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] report-tables-bound-per-table — crates/core/src/curation/markdown.rs:162 (and changelog.rs:118)
|
||||
- [LOW] report-guard-stale-doc — crates/core/src/curation/report_guard.rs:14
|
||||
- [LOW] record-weight-omits-position — crates/core/src/curation/draft/records.rs:187
|
||||
- [LOW] test-lint-relaxation (convention, not a scanner suppression) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
PROOF: semgrep 162 rules on 29 files → 0 findings; checklist 12 checks → 5 findings
|
||||
|
||||
## Scope and method
|
||||
|
||||
Range `c6e0d85..HEAD` (HEAD 29eb499), paths `crates/core`, `packages/anatomy/tests`,
|
||||
`docs/development/RUST.md`: 29 added or modified files, all scanned by semgrep (162 rules
|
||||
loaded from the three pinned packs, 0 results, 0 errors).
|
||||
|
||||
Checklist, in order: hardcoded secrets (none), SQL by string building (none in the diff; the
|
||||
seed SQL writer is unchanged R5.3 code), unsanitised render (Markdown reports and SVG drawings,
|
||||
below), sensitive endpoint authz (none: a local CLI and a library), internals surfaced to a user
|
||||
(CLI stderr on a developer machine, not applicable), secrets in logs (none), tracked `.env` or
|
||||
credential (only `.env.example`, outside the diff), guardrails scanner (`--domain api,supabase`:
|
||||
0 FAIL, 1 WARN in `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the
|
||||
diff, dropped), new suppressions (two test headers, LOW below), then the brief's threat model:
|
||||
Markdown parity against Prettier 3.8.3, resource bounds (records, tables, drawings), CLI file
|
||||
system confinement, SVG well-formedness.
|
||||
|
||||
Probes live in the session scratchpad, never in the repository:
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/probe`
|
||||
(a crate with a path dependency on `crates/core`, feature `curation`; `src/main.rs` wraps
|
||||
`report_guard::format_report`, `src/bin/amplify.rs` runs the draft pipeline in memory on the
|
||||
committed seed plus one extra change), `.../scratchpad/scripts/fmt.mjs` (the repository's
|
||||
Prettier 3.8.3 with the options of `packages/anatomy/src/format.ts`) and
|
||||
`.../scratchpad/scripts/{gen,gen2,widths,strong,diff}.py` (case generators and the
|
||||
differential). Nothing was written to the seed or any tracked file.
|
||||
|
||||
## Finding 1 (HIGH): the review-drawing bound does not cover the markers
|
||||
|
||||
`pipeline::review_drawings` refuses before rendering when
|
||||
`cleaned_svg.len() * 2 * views > MAX_REVIEW_DRAWINGS_BYTES` (64 MiB). RUST.md §11.14 states the
|
||||
guarantee as "review drawings past 64 MiB in all, projected before any is rendered … refused
|
||||
before rendering". The projection counts the cleaned drawing only. The application drawing of a
|
||||
view (`app_review::render_app_review_drawing` → `anatomy::layer::render_layer`) writes, for each
|
||||
detail trigger without a name of its own, the opened view's French and English names into
|
||||
`data-anno-name-fr`, `data-anno-name-en` and the `<title>` text `fr / en`: four copies of the
|
||||
view name per trigger. Nothing caps a view name (the sheet contract has no label length) or the
|
||||
number of triggers opening one view (integrity only requires the trigger to sit in the opened
|
||||
view's parent).
|
||||
|
||||
An authored change of one `set view:head name {fr: S bytes, en: S bytes}` and T `add document
|
||||
annotations` of unnamed detail triggers in `main` opening `head` passes the applier, the records
|
||||
bound, the 200,000-value document bound and the integrity gate, then writes about 4·T·S bytes in
|
||||
`review/app-main.svg`:
|
||||
|
||||
| S (per language) | T | changes.json | result |
|
||||
|---|---|---|---|
|
||||
| 1,024 | 10 | 166,322 B | 2.87 MB in all (baseline) |
|
||||
| 32,768 | 1,000 | 485,140 B | `review/app-main.svg` 131,713,082 B written, no refusal, peak RSS 429 MB |
|
||||
| 262,144 | 2,000 | 1,202,892 B | `memory allocation of 1073741824 bytes failed`, SIGABRT under `ulimit -v 3000000` (peak RSS 2.65 GB) |
|
||||
|
||||
At the read cap the records bound allows about S = 2 MiB and T = 5,600 (the set records 4·S
|
||||
units, each add about 1,500), so the tool would try to build about 44 GiB from a 5.4 MB
|
||||
changeset. `pnpm r5:import-equine --check` and `--out` both build every file before comparing or
|
||||
writing, so either mode dies on such a changeset. The TypeScript has the same amplification;
|
||||
the defect is that the port's declared bound, the control the round-8 fix introduced for this
|
||||
class, does not hold.
|
||||
|
||||
Reproduce: `cd .../scratchpad && (ulimit -v 4000000; ./probe/target/release/amplify 32768 1000)`.
|
||||
|
||||
Fix direction: project each view's drawing bytes before `render_layer`: the cleaned drawing,
|
||||
plus, per annotation of the view, a fixed overhead, its geometry text and four times the name
|
||||
bytes it carries (own name, or the opened view's names through `localized_names` for an
|
||||
unnamed trigger), plus the overlay's markers; refuse the sum past the bound. Freeze it with a
|
||||
test of unnamed triggers opening a long-named view, and restate the §11.14 row.
|
||||
|
||||
## LOW findings
|
||||
|
||||
- `markdown.rs:162` / `changelog.rs:118`: `MAX_TABLE_COLUMNS_TOTAL` bounds each table, not the
|
||||
report. The three level tables of the CHANGELOG can each approach 32 Mi columns: a synthetic
|
||||
raw report with three 6,000-row tables and one 5,300-byte cell each was admitted and aligned
|
||||
into 97,356,707 bytes (peak RSS 272 MB). Reaching it through the applier takes about 18,000
|
||||
records, at the edge of the 16 Mi-unit records bound. Bounded, consistent with the RUST.md
|
||||
wording ("an aligned report table"); a whole-report bound would make the worst case explicit.
|
||||
- `report_guard.rs:14-17`: the module doc says a code span admits "anything but a carriage
|
||||
return or a tab"; `generated_code` (`:186-193`) admits only `[A-Za-z0-9_:/. -]`, as RUST.md
|
||||
states. The code is the stricter one; the doc of the control is stale.
|
||||
- `records.rs:187-197`: `op_weight` does not charge the `position` anchor of an `add` or a
|
||||
`move` (`{"after": …}` / `{"before": …}`), which each record clones. One record per such op
|
||||
and the anchor must name an existing sibling, so the total stays under the changeset size:
|
||||
no amplification, the weight model is only incomplete.
|
||||
- `tests/curation_markdown.rs:6`, `tests/curation_review.rs:6`:
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` in two new
|
||||
integration-test files. The same header is on every test file of the crate, and `clippy.toml`
|
||||
says "Tests and the worker binary allow these lints locally where they own I/O". The library's
|
||||
I/O and clock ban stays enforced in `src/`. This is not a scanner-finding suppression, so it
|
||||
does not block.
|
||||
|
||||
## What held
|
||||
|
||||
- Markdown guard against Prettier 3.8.3: 189,394 generated reports in the templates of
|
||||
`changelog.rs` and `review_report.rs` (list items, bibliography, vet-flag lines, eight-column
|
||||
rows with `cell()` escaping). The cases cover random token mixes, admitted-only mixes, every
|
||||
code point from U+00A0 to U+33FF plus one in three up to U+2FFFF, in a table cell and in a
|
||||
list line, and `**word**` over the same letters. Result: 99,447 admitted, every admitted one
|
||||
byte for byte equal to Prettier, 0 silent differences. Structure forging (headings, list
|
||||
items, links, HTML, autolinks, references, code spans from authored text) is refused at the
|
||||
guard or at `assert_single_line` / `assert_authored`.
|
||||
- Records: every producer goes through `charge` (`ops.rs`, `rename.rs`, `apply.rs`
|
||||
marking). Raw report rows are bounded with the records. Aligned tables are refused before
|
||||
they are built. The guard and the aligner are linear per line.
|
||||
- SVG: element and attribute names are the generator's. Values and texts are escaped (`&`,
|
||||
`<`, `>`, `"`, tab, LF, CR). `first_unwritable` refuses non-XML-1.0 code points and CR. No
|
||||
author text can open a script or an href.
|
||||
- `--check`: byte identity, and the tree fallback only for `review/app-*` with the declared rule,
|
||||
after the committed file rebuilds from its tree. The committed drawing goes through the core
|
||||
reader first (depth cap), and `roxmltree` keeps its DTD refusal. Reads are confined to the
|
||||
canonical seed and capped at 16 MiB. A symlinked folder is never walked.
|
||||
- `--out`: no `..`, the parent is canonicalised, the final symlink is refused, the destination
|
||||
must be absent or empty, and it is refused inside the seed by path and by (dev, inode) over
|
||||
the whole seed tree. Writes use `create_dir` / `create_new`, never truncate, never delete.
|
||||
The check-to-write swap is the single-user residual already declared (§11.13).
|
||||
- Oracle generators (`packages/anatomy/tests/r5.*.test.ts`) write only under
|
||||
`MAKE_R5_ORACLES=1`, into `crates/core/tests/oracles/r5`. The Rust tests write only under
|
||||
the target folder.
|
||||
@@ -0,0 +1,153 @@
|
||||
# Security gate R5.4 (round 11): review files in Rust
|
||||
|
||||
Mode: audit (delta). Repository `/home/bchanot/Documents/faunosteo`, branch
|
||||
`feature/rust-r5-pipeline-core`, HEAD `411e105`. Scope:
|
||||
`git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`
|
||||
(29 files, 21 commits). Blind run: no prior verdict or finding list consulted.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 29 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] anti-gaming/new-lint-allow — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
- [LOW] report-guard/bare-url-autolink — crates/core/src/curation/report_guard.rs:119
|
||||
PROOF: semgrep 128 rules on 29 files → 0 findings; checklist 14 checks → 2 findings
|
||||
```
|
||||
|
||||
## Semgrep
|
||||
|
||||
`semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten
|
||||
--metrics=off --quiet --json <29 files>`: 0 results, 0 errors, 29 targets scanned, 128 rules
|
||||
run (most of the 725 loaded rules do not apply to Rust). Ruleset content is fetched at run
|
||||
time, so the rule count can change on unchanged code.
|
||||
|
||||
## Checklist (14 checks)
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Hardcoded secret, token, key, auth-bearing URL (added lines and the two oracle JSON files) | 0 |
|
||||
| 2 | SQL built by concatenation | 0 (no SQL in the diff; `seed.rs` untouched) |
|
||||
| 3 | Unsanitized render, `eval`, process spawn | 0 |
|
||||
| 4 | Sensitive endpoint without authz | n/a (no endpoint, offline CLI and library) |
|
||||
| 5 | Stack trace or internal path to an end user | 0 (the CLI prints paths to the local operator only) |
|
||||
| 6 | Secret or PII in a log | 0 |
|
||||
| 7 | Tracked `.env` or credential file | 0 (only `.env.example`, outside the diff) |
|
||||
| 8 | Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`), outside the diff, filtered |
|
||||
| 9 | New suppressions (anti-gaming) | 2 new lint allows in test files: LOW, see L1 |
|
||||
| 10 | Markdown structure forging (sections, list items, code spans, links, HTML) | 0 forging path; one LOW note, see L2 |
|
||||
| 11 | Silent difference from Prettier 3.8.3 on admitted text (differential probe) | 0 |
|
||||
| 12 | Non-well-formed review SVG | 0 |
|
||||
| 13 | Resource bounds (records, tables, drawings, quadratic scans) | 0 new |
|
||||
| 14 | CLI file-system confinement (`--check`, `--out`) | 0 |
|
||||
|
||||
### 10. Report structure forging
|
||||
|
||||
Every authored channel that reaches CHANGELOG.md or REVIEW-REPORT.md was traced:
|
||||
|
||||
- Line breaks: list-item fields (title, rationale, question, citation, URL, DOI, vet-flag
|
||||
values) refuse any control character (`review_report.rs:364`). Table cells turn `\n` into a
|
||||
space (`markdown.rs:47`), and the guard refuses `\r` and `\t` (`report_guard.rs:198`).
|
||||
No authored text starts a line: every template line begins with generated text (`- `,
|
||||
`N. `, `| `). Line-start constructs (headings, setext underlines, link reference
|
||||
definitions, footnote definitions, blockquotes, tables) are therefore out of reach.
|
||||
- Code spans: authored texts refuse backticks (`assert_authored`). Ids are validated at the op
|
||||
(`ops.rs` `assert_stable_id` on add, nested children and rename). Change ids match
|
||||
`^CHG-\d{3,}$`. Change reference ids must be declared references (`assert_declared`), whose ids
|
||||
are stable ids. Field segments are limited to `[A-Za-z0-9_-]` (`fields.rs`) and ids are not
|
||||
settable. Targets resolve to existing items. The code-span alphabet check
|
||||
(`[A-Za-z0-9_:/. -]`) and the per-line backtick parity hold as a second layer.
|
||||
- Inline constructs: links, images, references, HTML tags, autolinks `<…>`, character
|
||||
references, liquid and padded wiki brackets are refused. Cross-field pairs were checked:
|
||||
`**` opened in the title and closed in the rationale is refused by `outside_strong`, and `](`
|
||||
across fields is refused by `rewritten_construct`.
|
||||
- Invisible or bidi characters (U+202E, U+200B, NBSP) are neither alphanumeric nor in
|
||||
`PUNCTUATION`: refused.
|
||||
|
||||
### 11. Differential probe against Prettier 3.8.3
|
||||
|
||||
Probe: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/diff.mjs`
|
||||
and `.../diff2.mjs`. A Rust binary calls `report_guard::format_report`, and the repository's
|
||||
`prettier@3.8.3` runs on the same raw report with the `format.ts` options. Random authored
|
||||
texts (letters, accents, CJK, every admitted punctuation sign, `_`, `\`, `*`, `**word**`, URL
|
||||
starts, `&name=`, `&`, `&#`, brackets, list and heading markers) are placed in four
|
||||
templates: a list-item title and rationale, a table row (cell escape as `cell()` does), a
|
||||
bibliography citation and URL, and a trailing question. Eight seeds produced 146,000 cases.
|
||||
44,330 were admitted by the guard and 0 differed from Prettier byte for byte.
|
||||
|
||||
### 12. Review SVG well-formedness
|
||||
|
||||
Drawings are built as trees and printed by `anatomy::svg::serialize`. Attributes escape
|
||||
`& < > "` and `\t \n \r`, and text escapes `& < >`. `drawing::first_unwritable` refuses any
|
||||
code point outside the XML 1.0 `Char` production, and any carriage return, in every attribute
|
||||
and text node before writing. Element and attribute names are fixed. Colours come from the
|
||||
strict document (`#rrggbb`), and no `href`, `on*` or `style` value is author-controlled.
|
||||
|
||||
### 13. Resource bounds
|
||||
|
||||
- Records: `records::charge` runs before the change and op are cloned. It is called from every
|
||||
`record` site (set, subtree set, add, remove, move, cite, rename links and citations, B/C
|
||||
marking). Each record's weight includes the change texts, the op, and the before and after
|
||||
values.
|
||||
- Tables: the three CHANGELOG level tables are summed in `assert_table_bound` from the cell
|
||||
bytes before `table()` joins them. Bytes are never fewer than the display width: the first
|
||||
width-2 code point is U+1100, which takes 3 bytes. Every other table is checked in `align`
|
||||
before padding.
|
||||
- Review drawings: per-annotation projection. A probe binary
|
||||
(`.../scratchpad/probe/src/bin/drawings.rs`) compared projected and written bytes on hostile
|
||||
documents built from the equine draft: 40 points with 6,000-character `&` names (ratio
|
||||
0.405), `"<&` names (0.262), 6,000-byte `&"<` paths (0.170), unnamed triggers of a long view
|
||||
name (0.120), equine (0.406). The projection stays above what is written. The projection loop
|
||||
itself (views × annotations) took 1.2 s on a 7,000-view, 6,000-annotation document, which
|
||||
the bound then refuses.
|
||||
- Guard and aligner scans are linear per line. `underscore_pairs_later` only rescans after
|
||||
authored plain parts, and generated separators hold no `_`. The `&` and `<` splits are
|
||||
segment-local.
|
||||
- Already declared and not re-reported: the 65× parse of `changes.json`, the per-op bound walk
|
||||
(§11.13), and the linear trigger lookup in `--check`.
|
||||
|
||||
### 14. CLI confinement (`crates/core/src/bin/import_equine.rs`)
|
||||
|
||||
Every read is canonicalized and must lie inside the canonical seed folder. Reads accept only
|
||||
regular files and are capped at 16 MiB. Output file names come from stable view ids, and
|
||||
`write_exclusive` accepts only `Normal` components and `create_new`. Exercised in the
|
||||
scratchpad (fresh folders only, nothing in the repository):
|
||||
|
||||
- `--out` into an absent folder wrote 18 files. `diff -r` against the seed matched except
|
||||
`review/app-main.svg`, the declared R3 difference.
|
||||
- A second run into the same folder was refused: `not empty, nothing written`.
|
||||
- A symbolic link was refused with and without a trailing `/`.
|
||||
- A path holding `..` was refused, and so was an empty path.
|
||||
|
||||
`--check` against the repository reported `legacy-import: 6 files, 0 differences` and
|
||||
`draft: 12 files, 0 differences` (one `declared:` line), with `git status` clean afterwards.
|
||||
The `--out` refusal of a destination inside the seed was verified by reading the code and the
|
||||
unit test. It was not run against the real seed.
|
||||
|
||||
Tests run: `cargo test -p faunosteo-core --features curation --release` on `curation_review`
|
||||
(11), `curation_markdown` (2), `curation_draft` (18), `curation_format` (7) and the
|
||||
`import-equine` binary tests (3): all green.
|
||||
|
||||
## Reported findings (non-blocking)
|
||||
|
||||
### L1. [LOW] New lint allows in two new test files
|
||||
|
||||
`crates/core/tests/curation_markdown.rs:6` and `crates/core/tests/curation_review.rs:6`
|
||||
both add `#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`.
|
||||
The lint allowed here is the core's no-I/O and no-clock rule, which `clippy.toml` documents
|
||||
as allowed locally in tests. The same line heads the ten existing integration test files.
|
||||
It is a test-harness convention and does not suppress a security scanner (no `nosemgrep`,
|
||||
`nosec` or eslint-security in the diff), so it does not block. Context only.
|
||||
|
||||
### L2. [LOW] Bare URLs in any authored field render as autolinks
|
||||
|
||||
`report_guard.rs` admits bare URLs (`url_tokens`, line 119; the `&name=` query rule) in every
|
||||
authored text, not only in the reference `url` and `doi` fields. In a GFM viewer (Gitea), a
|
||||
URL typed in a title, rationale or question becomes a clickable link. This does not forge or
|
||||
disguise anything: the visible text is the target, and the output is the TypeScript's byte
|
||||
for byte. The link cannot swallow generated text either, because every authored field is
|
||||
followed by a space, ` |`, a line end, or a trailing `,` or `)` that GFM excludes. Owner
|
||||
choice: keep it, or confine URLs to the bibliography fields if links in rationales are
|
||||
unwanted.
|
||||
@@ -0,0 +1,129 @@
|
||||
# Security gate R5.4 (round 12): review files in Rust
|
||||
|
||||
Date: 2026-10-23. Mode: audit (fresh, blind). Branch `feature/rust-r5-pipeline-core`, HEAD
|
||||
`c4e717d`. Scope: `git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`
|
||||
(29 files, none deleted).
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 29 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] quadratic-scan (latent, unreachable today) — crates/core/src/curation/report_guard.rs:216
|
||||
- [LOW] quadratic-scan (bounded, ~2 s at the document bound) — crates/core/src/curation/draft/pipeline.rs:256
|
||||
- [LOW] new lint allowance (context, documented convention) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
- [LOW] declaration precision (R5.3 code, outside the diff) — crates/core/src/curation/draft/schema.rs:430
|
||||
PROOF: semgrep 128 rules on 29 files → 0 findings; checklist 9 checks → 0 findings (guardrails 0 FAIL, 1 WARN outside the diff); differential probes 38,378 admitted template reports + 898 admitted real-pipeline reports → 0 silent differences; display width 1,112,032 code points → 0 differences
|
||||
```
|
||||
|
||||
## Tool run
|
||||
|
||||
- `semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten --metrics=off`
|
||||
on the 29 scope files: 128 rules applicable, 0 results, 0 errors.
|
||||
- Guardrails scanner (`--domain api,supabase`, project root): 0 FAIL, 1 WARN
|
||||
(`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, not in
|
||||
the diff, dropped), 1 MANUAL (not reported).
|
||||
|
||||
## Checklist (9 checks)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| hardcoded secret, token, key, auth URL in added lines | none |
|
||||
| SQL by concatenation | none in the diff (`seed.rs` untouched) |
|
||||
| unsanitized render, `eval`, shell | none; SVG output escaped (below) |
|
||||
| sensitive endpoint without authz | N/A (offline CLI, no endpoint) |
|
||||
| stack trace or DB error to an end user | N/A (operator CLI; refusals print `{:?}`, control characters escaped) |
|
||||
| secret or PII in a log | none |
|
||||
| tracked `.env` or credential file | none (`git ls-files`) |
|
||||
| guardrails scanner | run, nothing in the diff |
|
||||
| new suppression (anti-gaming) | no `nosemgrep`, `nosec`, `eslint-disable`; two lint allowances in test files (LOW 3) |
|
||||
|
||||
## What was verified, by probe or by reading
|
||||
|
||||
Markdown guard (`report_guard.rs`) against Prettier 3.8.3 with the repository options, from a
|
||||
scratch harness (Prettier imported from the repository's `node_modules`, the Rust guard through a
|
||||
scratch crate depending on `faunosteo-core` with `curation`):
|
||||
|
||||
- template reports reproducing the ten list-item and cell positions of the two reports (title,
|
||||
rationale, question, citation, URL, DOI, vet-flag before/after, label cell): 1,197 hand probes
|
||||
(reference shapes `[a] [b]`, `[a] (b)`, `\|` outside tables, `+`/`>`/`1.`/`#` mid-line, URL
|
||||
trailing punctuation and `&name=`, `_` near URLs, `**word**`), 27,000 random single-field, 10,000
|
||||
random cross-field and 36,000 URL-centric cases: 38,378 admitted, 0 differ from Prettier;
|
||||
- the real generators: 7,200 random changesets run through the TypeScript (`applyChangeset`,
|
||||
`renderChangelog`, `renderReviewReport`, `formatMarkdown`) and through the Rust
|
||||
(`apply_changeset`, `render_changelog`, `render_review_report`, `format_report`): 898 admitted,
|
||||
6,302 refused, 0 raw-text differences, 0 silent formatted differences;
|
||||
- `text::display_width` against Prettier's `util.getStringWidth` for `"a" + c`, every scalar value
|
||||
from U+0020 to U+10FFFF: 0 differences wherever the port measures (it refuses the rest).
|
||||
|
||||
Report structure: authored texts carry no backtick (`assert_authored` on every authored field
|
||||
of both reports), list-item fields refuse control characters, cells turn `\n` into a space and
|
||||
escape `|`; ids written by `add`/`rename` are stable ids or region codes, field segments
|
||||
`[A-Za-z0-9_-]`, targets resolve only to existing stable ids, so no author byte starts a line or
|
||||
writes a code span. Email autolinks need `@`, which the allowlist refuses.
|
||||
|
||||
SVG drawings: the serializer escapes `& < >` in text and `& < > " \t \n \r` in attributes
|
||||
(`]]>` cannot appear); `drawing::first_unwritable` refuses non-XML characters and CR; element and
|
||||
attribute names are constants; colours are `#rrggbb` by schema; view ids (file names) are stable
|
||||
ids. Drawings projection: names copied at most four times per annotation, worst escape `&` → 5
|
||||
bytes per JSON byte, so `24 × (json + longest name)` holds.
|
||||
|
||||
Records bound: every record site goes through `Recorder::record` or `mark_to_confirm`, both
|
||||
charged before the clone. Aligned tables: CHANGELOG projected from cell bytes (bytes ≥ display
|
||||
width, numbers ≤ 25), every table rechecked in `align` before padding.
|
||||
|
||||
CLI `import_equine.rs`: reads only canonical regular files inside the canonical seed, 16 MiB
|
||||
cap; `--check` refuses a generated folder that is not the seed's own, lists without following
|
||||
links, compares bytes (app drawings only through the strict rebuild rule); `--out` refuses `..`,
|
||||
symlinks, non-empty folders and any path inside the seed by prefix or by (dev, inode), writes
|
||||
with `create_dir`/`create_new`, deletes nothing; file names from views are stable ids and
|
||||
`write_exclusive` accepts only normal components. The check/use race is declared (§11.13).
|
||||
|
||||
Timing of the guard on adversarial lines (linear): 400,000 `& ` in 7.9 ms, 400 KB of `[[x`,
|
||||
`x ]]`, `**ab**`, `<`, URL `&a=1` runs under 10 ms, a 1.4 MB 100,000-column table in 95 ms.
|
||||
|
||||
## Findings (non-blocking)
|
||||
|
||||
### LOW 1: `underscore_pairs_later` is quadratic, latent
|
||||
|
||||
`crates/core/src/curation/report_guard.rs:216`. For every plain part holding `_`, the function
|
||||
rescans all later code parts and re-collects each into a `Vec<char>`. A synthetic line of 100,000
|
||||
`` `ab` x_y `` (900 KB) took 53.7 s. Not reachable today: authored text holds no backtick, so the
|
||||
generator alone interleaves code spans and plain parts, and on every line of both reports at most
|
||||
two plain parts can carry an authored `_` (title with rationale, question), which keeps the scan
|
||||
linear. A future line that puts authored text between generated spans would turn it quadratic.
|
||||
Hint: scan the parts right to left once, keeping a "a later code part closes" flag.
|
||||
|
||||
### LOW 2: drawings projection walks views × annotations before the bound
|
||||
|
||||
`crates/core/src/curation/draft/pipeline.rs:256`. `review_drawings_bytes` runs `layer_bytes` over
|
||||
every annotation for every view, and the view count is not yet bounded there (only by the
|
||||
200,000-value document bound). Measured on a synthetic `DraftResult`: 10,000 × 10,000 in 1.8 s,
|
||||
20,000 × 20,000 in 7.6 s; the document bound allows about 11,000 of each, so about 2 s. Hint:
|
||||
group annotations by `view_id` once, or stop as soon as the running total passes the bound.
|
||||
|
||||
### LOW 3: lint allowances in the two new test files (context)
|
||||
|
||||
`crates/core/tests/curation_markdown.rs:6`, `crates/core/tests/curation_review.rs:6`:
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. The lint is the
|
||||
core's no-clock, no-file-system rule; `clippy.toml` documents that tests allow it locally, and
|
||||
every existing integration test file carries the same line. Not a security-rule suppression.
|
||||
|
||||
### LOW 4: §11.15 names `changes.json` alone for the parse amplification
|
||||
|
||||
`crates/core/src/curation/draft/schema.rs:430`. `parse_references` goes through the same
|
||||
`json_parse` as `parse_changeset`, so `references.json` under its 16 MiB cap should hold the same
|
||||
order of memory (factor not measured here). R5.3 code, outside this diff; the residual line could
|
||||
name both files.
|
||||
|
||||
## Notes
|
||||
|
||||
- The working tree changed during the run (comment and doc lines only, by another agent):
|
||||
`.claude/audits/2026-10-23-rust-r5-review-verify.md`, `docs/development/RUST.md`,
|
||||
`packages/anatomy/tests/r5.review-synthetic-oracles.test.ts`. HEAD stayed `c4e717d`; this gate
|
||||
judges the committed range.
|
||||
- Declared residuals (§11.15, including the ~65× `changes.json` parse) not re-reported.
|
||||
- Probes and harnesses live in the session scratchpad only; no tracked file touched, the
|
||||
`import-equine` binary never run; the scratch crates read only the committed sweep oracle.
|
||||
@@ -0,0 +1,195 @@
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 32 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] markdown-guard-bypass-via-ids — crates/core/src/curation/draft/review_report.rs:367 (also markdown.rs:149, ops.rs:286, ops.rs:197, changelog.rs:81)
|
||||
- [MEDIUM] uncapped-recursive-parse — crates/core/src/curation/draft/svg_compare.rs:108 (called at :129)
|
||||
- [MEDIUM] silent-prettier-difference-and-wrong-declaration — crates/core/src/curation/markdown.rs:115 (docs/development/RUST.md §11.14, §11.15)
|
||||
- [LOW] overlay-non-finite-number — crates/core/src/curation/draft/overlay.rs:163
|
||||
- [LOW] new-lint-allow-in-tests (context) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
PROOF: semgrep 128 rules on 32 files → 0 findings; checklist 9 checks (8 defaults + anti-gaming) → 1 finding (LOW); targeted probes (68 authored changesets through the Rust renderers, 136 report comparisons against Prettier 3.8.3; compare_drawing at depths 60 to 100000; one read-only --check) → 4 findings
|
||||
```
|
||||
|
||||
Gate on `git diff c6e0d85..HEAD` (HEAD 68dec25, branch feature/rust-r5-pipeline-core), focus on fix commit 94f8b35.
|
||||
No HIGH or CRITICAL. Three MEDIUMs: one makes a declared guard leak, one regresses a depth cap
|
||||
the project documents as an invariant, one shows the RUST.md parity declarations are wrong. None is
|
||||
reachable from outside the repository: every input is authored in the owner's own seed folder.
|
||||
|
||||
## Method
|
||||
|
||||
- semgrep 1.168.0 (`EIO_BACKEND=posix`, `--metrics=off`, the three pinned packs) on the 32 diff
|
||||
files that still exist: 128 rules, 0 findings, 0 errors.
|
||||
- Checklist (CLAUDE.md defaults) over the diff: secrets, SQL concatenation, unsanitized render,
|
||||
authz, error exposure, secrets in logs, tracked `.env`, guardrails scanner; anti-gaming suppressions.
|
||||
- Guardrails `scan.py --domain api,supabase`: 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`,
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`). That file is not in the diff, so the WARN is filtered out.
|
||||
- I read whole: `markdown.rs`, `review_report.rs`, `svg_compare.rs`, `drawing.rs`, `overlay.rs`,
|
||||
`app_review.rs`, `changelog.rs`, `labels.rs`, `stats.rs`, `pipeline.rs`, `bin/import_equine.rs`.
|
||||
I also read the parts of `ops.rs`, `apply.rs`, `schema.rs`, `targets.rs`, `fields.rs`,
|
||||
`anatomy/svg/{serialize,tree}.rs` and roxmltree 0.21.1 `tokenizer.rs` that these files rely on.
|
||||
- Probes, all in the scratchpad (`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/`):
|
||||
a throwaway crate linking `faunosteo-core` (feature `curation`). It feeds 68 authored changesets
|
||||
on the mini sheet of `oracles/r5/markdown-sweep.json` through `apply_changeset`,
|
||||
`render_changelog`, `render_review_report` and `format_markdown`. A Node script then runs the
|
||||
repository's Prettier 3.8.3 (the options of `packages/anatomy/src/format.ts`) on the same raw
|
||||
texts and compares (136 comparisons: 118 identical, 4 refused, 14 silent differences). A second
|
||||
entry point calls `compare_drawing` on nested input. Finally `import-equine --check` ran once
|
||||
(target dir in the scratchpad, `--locked`): `legacy-import: 6 files, 0 differences`,
|
||||
`draft: declared: review/app-main.svg`, `draft: 12 files, 0 differences`. The working tree was
|
||||
clean before and after. `--out` was never run.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. [MEDIUM] Markdown guard bypassed through ids: forged headings, sections and tables, raw CR in table rows
|
||||
|
||||
Where: `review_report.rs:367` (`assert_list_items_single_line`), `markdown.rs:149` (code parts
|
||||
checked only for emptiness and outer spaces), `ops.rs:286` (`add` takes any string id),
|
||||
`ops.rs:197` (`rename` takes any string id), `fields.rs:115` (only the first field segment is
|
||||
checked), `changelog.rs:81` (`cell()` turns `\n` into a space but leaves `\r`).
|
||||
|
||||
The single-line check covers titles, rationales, questions, citations, URLs, DOIs and vet-flag
|
||||
values. It misses the other authored texts the review report writes into list items. These are:
|
||||
|
||||
- the record subjects (op target plus field), at lines 97, 167 and 181;
|
||||
- the change `reference_ids`, at lines 98 and 339.
|
||||
|
||||
Ids are only checked as stable ids at the end of the changeset (`apply.rs:210`,
|
||||
`curated_issues`). So an item added with an arbitrary id and then removed, or renamed away, leaves
|
||||
its raw id in the records while the final document stays valid. The same holds for a field path
|
||||
`name.<anything>` set on such an item. A reference added and removed the same way can be cited in
|
||||
`reference_ids`, because `assert_declared` runs before the ops.
|
||||
|
||||
`guard_line` checks line by line. A newline inside a code span, with one backtick on each side,
|
||||
therefore yields lines that each pass. Measured, all passing the whole guard (`format_markdown`
|
||||
Ok on both reports):
|
||||
|
||||
- Add then remove annotation `x`␊`## 11. Section forgée`␊`y` (written with a backtick after `x`
|
||||
and before `y`). REVIEW-REPORT §2 and §3 then read:
|
||||
```
|
||||
- `CHG-001` Titre (niveau A, Correction, confirmation requise : non). Raison. Objets : `annotation:x`
|
||||
## 11. Section forgée
|
||||
`y`. Références : aucune.
|
||||
```
|
||||
The report gets an eleventh section. A `kind: translation` change forges an H1 in §7. A
|
||||
reference id forges a heading in the `Références :` part. Prettier puts blank lines around the
|
||||
heading, so the Rust file also differs silently from the TypeScript's.
|
||||
- An id carrying the lines `| Forgé | Tableau |`, `| --- | --- |`, `| a | b |` produces a table
|
||||
inside REVIEW-REPORT, which `format_markdown` aligns. Prettier keeps it indented inside the list
|
||||
item, so this is a silent difference again.
|
||||
- Id `x\ry`. CR is refused in plain text but not inside a code span. The CHANGELOG row
|
||||
`| `CHG-001` Titre | `annotation:x\ry` | ...` keeps a raw CR. GFM (goldmark on Gitea) reads it
|
||||
as a line ending, which breaks the table. Prettier normalizes it to LF, splitting the row in its
|
||||
own output. The same raw CR shows up in the REVIEW-REPORT list item.
|
||||
|
||||
Impact: the reports, the CHANGELOG and the review document the osteopath signs off on, can carry
|
||||
structure that never came from the generator. The owner's contract ("refused, never a report that
|
||||
differs silently") does not hold for this channel. It takes a deliberately crafted changeset in
|
||||
the repository (add then remove, or a rename chain). No accidental path was found.
|
||||
|
||||
Fix direction:
|
||||
- At the root: check ids as stable ids when the op runs. That covers the added item's `id` (and
|
||||
the key field), `rename.id`, and every target id segment and field segment. The final document
|
||||
requires stable ids anyway, so refusing earlier loses nothing that could be published.
|
||||
- Defence in depth: `guard_text` refuses any control character inside a code part, and
|
||||
`assert_list_items_single_line` also checks record subjects and change `reference_ids`.
|
||||
- Add these probes to the sweep, then declare the refusal in RUST.md.
|
||||
|
||||
### 2. [MEDIUM] Uncapped recursive XML parse of the committed drawing, ahead of the prescan
|
||||
|
||||
Where: `svg_compare.rs:108`. `outside_root` calls `roxmltree::Document::parse` with default
|
||||
options. Its caller is line 129, before `parse_svg`, which is the function that runs
|
||||
`prescan(SVG_CAPS)`. This ordering is new in 94f8b35.
|
||||
|
||||
roxmltree's `parse_element` → `parse_content` → `parse_element` recurses once per nesting level.
|
||||
The workspace manifest itself says the core prescan must cap depth before any parse. Measured with
|
||||
`compare_drawing(produced, committed, Some(rule))`: depths 60, 70 and 1000 are refused cleanly.
|
||||
At depth 100000 (a 300 KB committed text, far below the 16 MiB read cap) the process dies with
|
||||
`thread 'main' has overflowed its stack`, exit 134. In `--check` this happens whenever a
|
||||
`review/app-*.svg` differs from the produced bytes.
|
||||
|
||||
Impact: an abort, not a pass. It fails closed, writes nothing and exits non-zero. Still, it breaks
|
||||
the "depth and resource caps" requirement and the project's own prescan-first invariant.
|
||||
`drawing.rs` orders the same two calls correctly (`read_svg` first).
|
||||
|
||||
Fix direction: call `parse_svg(committed)` before `outside_root`, or run `prescan(text, &SVG_CAPS)`
|
||||
at the top of `outside_root`. Add a nested-input case to `curation_review.rs`.
|
||||
|
||||
### 3. [MEDIUM] Authored plain text that passes the guard but which Prettier rewrites; RUST.md §11.14 and §11.15 overstate the guarantee
|
||||
|
||||
Where: `markdown.rs:115` (the `_` rule tests `is_alphanumeric` on both neighbours), and the guard
|
||||
having no rule for link syntax.
|
||||
|
||||
Measured, each passing the guard, then differing from Prettier with no refusal:
|
||||
|
||||
- `中_中`, `ア_ア`, `한_한` in a title, rationale or changed label. Prettier's text splitter makes
|
||||
each CJK, kana or Hangul character its own word, so the `_` sits on a word boundary. In tables
|
||||
it becomes `中\_中`. In a review list item that holds two such texts, Prettier reads
|
||||
`_中 … 中_` as emphasis and prints `中*中 … 中*中`. Rust writes the raw text.
|
||||
- `[a]( https://x.org )` in a changed label: Prettier prints `[a](https://x.org)`, Rust keeps the
|
||||
spaces.
|
||||
|
||||
These do not break the Rust output's structure. They break parity: TypeScript-era files and Rust
|
||||
files differ without a refusal. Two declarations are therefore wrong as written:
|
||||
- §11.14 says the port "refuses ... every text Prettier would rewrite, never writing a report that
|
||||
differs silently". That holds for the 220 sweep probes only.
|
||||
- §11.15 says "links in authored text are kept as the TypeScript kept them". The TypeScript output
|
||||
was Prettier's, which normalizes this link form.
|
||||
|
||||
Fix direction:
|
||||
- Refuse `_` when either neighbour is a character Prettier's splitter isolates (CJK, kana,
|
||||
Hangul), or more simply whenever a neighbour is not an ASCII or Latin letter or digit.
|
||||
- Refuse a link destination with surrounding whitespace, or refuse `](` outright, since the equine
|
||||
reports use none.
|
||||
- Add these probes to the sweep and correct both declarations.
|
||||
|
||||
### 4. [LOW] Overlay prints an overflowed coordinate as `null`
|
||||
|
||||
Where: `overlay.rs:163`. `text_of` special-cases NaN only. `js_number` prints a non-finite value
|
||||
as `null`, where `String(Infinity)` printed `Infinity`. It is reachable with finite but extreme
|
||||
document values (a view box `min_y` near `f64::MAX`, or polygon points around `1e308`, whose sum
|
||||
overflows). The XML stays well-formed and nothing is injected. It is a silent parity gap only.
|
||||
Fix: print `Infinity` / `-Infinity` in `text_of`.
|
||||
|
||||
### 5. [LOW] New lint allows in the two new test files (context, anti-gaming)
|
||||
|
||||
Where: `curation_markdown.rs:6` and `curation_review.rs:6`, both
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. This is the
|
||||
pattern `clippy.toml` documents for tests that do their own I/O; 34 existing test files carry it.
|
||||
It is not a security suppression. There is no `nosemgrep`, `nosec`, `eslint-disable`,
|
||||
`ts-ignore` or new allow in library code.
|
||||
|
||||
## Checked and clean
|
||||
|
||||
- Strict SVG comparison logic (apart from finding 2). The fallback accepts only when
|
||||
`produced == prolog + serialize(rule(parse(committed))) + epilog` and the committed file
|
||||
rebuilds byte for byte from its own tree. So the produced file can differ from the committed one
|
||||
only where the rule changed the tree. The rule's values come from the produced document's view
|
||||
names, and `document.json` is compared byte for byte. A changed French name makes the rule
|
||||
change nothing, which is refused. A DTD is refused by roxmltree's default options. Comments,
|
||||
PIs, entity forms and other encodings fail the rebuild check. I found no way to get an
|
||||
unauthorized difference accepted.
|
||||
- Markup injection in the drawings: the serializer escapes `& < > "`, plus `\t \n \r` in
|
||||
attributes and `& < > \r` in text. Element and attribute names are fixed by code.
|
||||
`drawing_text` refuses non-XML characters and CR before writing.
|
||||
- Table cells: `|` is always escaped. An authored `\|`, a trailing backslash and a backslash before
|
||||
a space are refused. Pipes and a trailing backslash inside code spans format exactly like
|
||||
Prettier. The per-cell backtick check and the cell-count check in `align` hold.
|
||||
- Control characters (NUL, FF, VT, NEL, U+2028), NBSP, BOM, U+3000, zero-width space, `#`, `-`,
|
||||
`+`, `1.`, `>`, `---`, `~x~`, `~~x~~`, backslash escapes before punctuation, entities, autolinks,
|
||||
images, footnote syntax and raw HTML at the start of a cell: all identical to Prettier.
|
||||
- Panics on authored input: none found in the new library code. There is no `unwrap`, `expect` or
|
||||
unchecked index, and the slices fall on ASCII or roxmltree boundaries. Tree recursion
|
||||
(`first_difference`, `TriggerNames::apply`, `serialize`) runs on trees from `parse_svg`, which
|
||||
is depth-capped at 64. `first_unwritable` uses an explicit stack.
|
||||
- Resources: the guard and the alignment are linear per line. Renderers are indexed (subjects
|
||||
grouped, citation counts in one walk, hashed previous markers). Record memory is a declared
|
||||
residual (§11.13).
|
||||
- `--check`: read-only, as declared. Reads go through the canonical-path, regular-file and
|
||||
16 MiB checks. A file that is missing, not produced or different fails. `declared:` is printed
|
||||
only after the strict match. `--out` claim logic: the only change is that an existing
|
||||
destination is now checked against the seed by identity. Sound.
|
||||
- Secrets, SQL built from strings, authz, error exposure, logging of secrets or PII, tracked
|
||||
`.env`: nothing in the diff (only `.env.example` is tracked).
|
||||
@@ -0,0 +1,182 @@
|
||||
# Security gate (fresh, round 3): R5.4 review files, `c6e0d85..077d91b`
|
||||
|
||||
Branch `feature/rust-r5-pipeline-core`, HEAD `077d91b`, attention on `df82424`. Blind run: no
|
||||
earlier verdict or finding list was read.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: BLOCK(2)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 34 files
|
||||
BLOCKING:
|
||||
1. [HIGH] markdown-guard/strong-span — crates/core/src/curation/markdown.rs:79-106 (outside_strong), :157-163 — the `**strong**` model accepts pairings Prettier reads or escapes differently: a report differs from the TypeScript+Prettier output with no refusal — hint: treat `**` as a text boundary (refuse `_` or `\` next to a delimiter, a `*` run longer than 2, strong content starting or ending with a non-ASCII-space whitespace, Cf or Cc character); add these classes and a seeded random set to the TS sweep
|
||||
2. [HIGH] markdown-guard/underscore-before-autolink-literal — crates/core/src/curation/markdown.rs:157-163, :109-145 — `_` between a letter and `http://`, `https://` or `www.` passes the neighbour test, Prettier escapes it (`Voir_https://…` → `Voir\_https://…`): silent difference — hint: refuse `_` directly before or after a bare URL literal (`http://`, `https://`, `www.`), add the probe to the sweep
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] field-segment/nul-mid-segment — crates/core/src/curation/draft/fields.rs:104
|
||||
- [LOW] markdown-guard/trailing-space-latent — crates/core/src/curation/markdown.rs:169
|
||||
- [LOW] anti-gaming/test-allow (context) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
PROOF: semgrep 128 rules on 34 files → 0 findings; checklist 9 checks → 5 findings
|
||||
```
|
||||
|
||||
## Method
|
||||
|
||||
- semgrep 1.168.0, `EIO_BACKEND=posix`, the three pinned packs, on the 34 files of
|
||||
`git diff --name-only c6e0d85..HEAD` that exist: 128 rules ran (41 multilang on 34 files, 84
|
||||
TypeScript on 2, 3 JSON on 2), 0 findings, 0 errors. The packs carry no Rust rule: the Rust
|
||||
product files were read whole and probed by hand.
|
||||
- Differential probing, all under the fresh
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/probe.x3u0rG/`:
|
||||
a throwaway binary linking `faunosteo-core` (feature `curation`) calls `format_markdown` and
|
||||
the real renderers; a Node script runs the repository's Prettier 3.8.3 with the options of
|
||||
`packages/anatomy/src/format.ts`; a `tsx` script runs the TypeScript pipeline
|
||||
(`applyChangeset`, `renderChangelog`, `renderReviewReport`, `formatMarkdown`) on the same
|
||||
changesets. About 52,000 report texts compared (targeted sweeps plus two seeded random
|
||||
fuzzes), then 9 changesets end to end through both pipelines.
|
||||
- `import-equine --check` rebuilt in the scratchpad target dir and run (read-only). `--out` was
|
||||
not run; its write path was read. `git status` clean after every run.
|
||||
- Test suites `curation_draft` (16), `curation_markdown` (1), `curation_review` (9): pass.
|
||||
|
||||
## Blocking findings
|
||||
|
||||
### 1. [HIGH] The strong-span model of the Markdown guard lets silent differences through
|
||||
|
||||
`outside_strong` (markdown.rs:79-106) removes every `**…**` pair it judges well formed and
|
||||
`plain_rewrite` (:147-173) then checks the text left. Four input classes pass the guard while
|
||||
Prettier 3.8.3 (remark-parse 7 underneath) rewrites them:
|
||||
|
||||
| Class | Example authored text | Rust writes | TypeScript + Prettier writes |
|
||||
|---|---|---|---|
|
||||
| content made only of whitespace, format or control characters other than U+0020 | `** **` (also U+1680, U+2000-200A, U+2028/2029, U+202F, U+205F, U+3000, U+FEFF, U+200B-200D, U+2060, U+00AD, U+180E, C0/C1 controls) | `** **` | `\*\* \*\*` |
|
||||
| a backslash right before either delimiter | `\**a**`, `x\**a**`, `**1&é.\**-` | as written | `\*\*a\*\*` |
|
||||
| touching pairs | `**a****b**` | as written | `**a\*\***b\*\*` |
|
||||
| underscore next to a delimiter, judged on the text with the delimiters removed (`x_a`) | `x_**a**`, `**a**_x` | as written | `x\_**a**`, `**a**\_x` |
|
||||
|
||||
Every channel that writes plain text is affected: title, rationale, question, citation, URL,
|
||||
DOI, vet-flag values in the review list items, and label, description and value texts in the
|
||||
CHANGELOG cells, where the table widths change as well.
|
||||
|
||||
End-to-end proof on the mini sheet of `oracles/r5/markdown-sweep.json`
|
||||
(`probe.x3u0rG/e2e_in.json`, results `e2e_out.json` for Rust and `ts_out.json` for the
|
||||
TypeScript, comparator `js/cmp_e2e.py`): the raw texts are identical in all 8 changesets. In 7
|
||||
of them Rust accepts a final CHANGELOG different from the TypeScript's, and in 6 a different
|
||||
REVIEW-REPORT. Excerpt (changeset 2, rationale `Raison x_**important**.`):
|
||||
|
||||
```
|
||||
rust: …confirmation requise : oui). Raison x_**important**. Objets…
|
||||
ts: …confirmation requise : oui). Raison x\_**important**. Objets…
|
||||
```
|
||||
|
||||
Impact: no structure is forged. The Rust output is the authored raw text, which equals the
|
||||
TypeScript raw text: no line, item or cell is created. Rendered meaning is mostly unchanged.
|
||||
The broken property is the owner's requirement for this tool: "byte-identical or loudly
|
||||
refused". The guarantee written in RUST.md §11.14 (line 3158 "never writing a report that
|
||||
differs silently", line 3174 "0 silent difference") holds only for the 247 sweep probes, so
|
||||
that declaration is wrong. Today `--check` would catch such a change against the committed
|
||||
TypeScript output. After the switch, the report is written and committed silently, and a later
|
||||
`pnpm format` (`prettier --write .`, the draft folder is not in `.prettierignore`) rewrites it,
|
||||
so `--check` then fails far from the cause.
|
||||
|
||||
Why HIGH: this guard is the only thing that enforces an explicit owner integrity requirement,
|
||||
the documented guarantee is false, and inputs reach it through ordinary authoring. Beyond that
|
||||
requirement there is no exploit.
|
||||
|
||||
Fix direction: treat each `**` as the edge of a text node. Refuse `_` or `\` next to a
|
||||
delimiter, a run of three or more `*`, and strong content whose first or last character is a
|
||||
Unicode whitespace, a format (Cf) or a control (Cc) character. Then grow the TypeScript sweep
|
||||
with these four classes plus a seeded random combination set. The random fuzz here found them
|
||||
within a minute: the sweep's hand-picked probe list is the weak point.
|
||||
|
||||
### 2. [HIGH] An underscore glued to a bare URL passes the guard and is escaped by Prettier
|
||||
|
||||
The underscore rule (markdown.rs:157-163) accepts `_` between two ASCII letters or digits.
|
||||
Before `http://`, `https://` or `www.`, the next letter belongs to a link that remark-parse 7
|
||||
opens at the URL. The underscore then ends a text node, and Prettier escapes it.
|
||||
`rewritten_construct` (:109-145) has no rule for bare URLs.
|
||||
|
||||
| Authored text | Rust writes | TypeScript + Prettier writes |
|
||||
|---|---|---|
|
||||
| `Voir_https://example.org/x` (title) | `Voir_https://example.org/x` | `Voir\_https://example.org/x` |
|
||||
| `Voir_www.example.org` (citation) | `Voir_www.example.org` | `Voir\_www.example.org` |
|
||||
| `&b_http://x.y1`, `b_www.a.b>#` (random fuzz) | as written | `\_` inserted |
|
||||
|
||||
End to end: changesets 1 and 6 of `e2e_in.json`. The raw texts are identical, both reports
|
||||
differ, Rust raises no refusal. `mailto:` did not trigger the difference. Impact and severity
|
||||
are those of finding 1.
|
||||
|
||||
Fix direction: refuse `_` directly before or after `http://`, `https://` or `www.`, and add
|
||||
the probe to the sweep.
|
||||
|
||||
## Reported, non-blocking
|
||||
|
||||
- [LOW] field-segment/nul-mid-segment, `crates/core/src/curation/draft/fields.rs:104`.
|
||||
`c.is_control() && c != '\0'` exempts NUL everywhere, while line 110 refuses it only as the
|
||||
first character of a segment. The ops `set name.f\0r` then `set name {fr,en}` give a valid
|
||||
final document and leave a NUL in the record subject and in both reports' code spans. This
|
||||
was verified end to end: the TypeScript writes the same bytes and Prettier keeps them, so
|
||||
there is no divergence. It contradicts the comment at line 101 and the RUST.md §11.14 row at
|
||||
line 3182 ("a field segment holding a control character … refused"). Fix: refuse `\0`
|
||||
anywhere in a segment.
|
||||
- [LOW] markdown-guard/trailing-space-latent, `crates/core/src/curation/markdown.rs:169`. The
|
||||
guard never refuses a space that ends a line, which Prettier strips. Only a question reaches a
|
||||
line end (`change_line`, review_report.rs:79). Today it is refused by accident: section 10
|
||||
(`professional_decisions`, :319) reprints every question mid-line, where the trailing space
|
||||
becomes a double space (end-to-end changeset 0). A layout change of section 10 would reopen
|
||||
it. Fix: refuse trailing whitespace at the end of a guarded line.
|
||||
- [LOW] anti-gaming/test-allow (context). The new
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` in
|
||||
`crates/core/tests/curation_markdown.rs:6` and `crates/core/tests/curation_review.rs:6` is a
|
||||
test-only allow for reading oracle files, the same pattern as the pre-existing
|
||||
`crates/core/tests/curation_draft.rs:7`. It is not a security suppression. No `nosemgrep`,
|
||||
`nosec`, `eslint-disable`, `ts-ignore` or `noqa` was added.
|
||||
|
||||
## Checklist (9 checks)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Hardcoded secret, token, key, auth URL | none in added lines (audit texts only name the rulesets) |
|
||||
| SQL built by concatenation | no SQL change in the diff |
|
||||
| Unsanitized render of authored text | Markdown: findings 1, 2, LOW trailing space; SVG: clean (below) |
|
||||
| Sensitive endpoint without authz | none (offline dev tool) |
|
||||
| Stack trace, internal path, DB error to a user | none (CLI messages to the operator) |
|
||||
| Secret or PII in a log | none (stdout and stderr carry file names, counts, refusals) |
|
||||
| Tracked `.env` or credential file | only `.env.example` tracked, not in the diff |
|
||||
| Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, a migration outside the diff, dropped), 1 MANUAL |
|
||||
| New suppressions (anti-gaming) | LOW context only |
|
||||
|
||||
## Verified clean
|
||||
|
||||
- Ids at the op (`df82424`): `add` checks the item's key and every nested `children` /
|
||||
`categories` id, `rename` checks the new id (a region code for a region). Record targets
|
||||
resolve to existing, validated ids, change ids match `CHG-\d{3,}`, and reference ids come from
|
||||
the strictly parsed registry or a validated add. I found no remaining path for a line break or
|
||||
a backtick into a code span.
|
||||
- Code spans: 739 probes (NBSP, U+2000, U+3000, U+FEFF, zero-width characters, controls and NUL
|
||||
at the edges and inside, `\`, `|`, `\|` in table code), 0 divergence.
|
||||
- Display width: about 110 pictograph, combining, format and wide characters in cells. Each one
|
||||
is either equal to Prettier or refused (`UnsupportedWidth`).
|
||||
- List items: `assert_single_line` covers the title, rationale, question, citation, URL, DOI and
|
||||
vet values. CHANGELOG cells turn `\n` into a space and escape `|`, and the guard refuses CR
|
||||
and TAB anywhere.
|
||||
- Drawings: the serialiser escapes `& < > "` and `\t \n \r` in attributes and `& < > \r` in
|
||||
text, identical to xmldom 0.9.12 on 11 probe values. `first_unwritable` refuses non-XML code
|
||||
points and CR. The overlay prints `Infinity`, `-Infinity` and `NaN` as `String(number)`.
|
||||
- `compare_drawing`: `parse_svg` (prescan: depth 64, 20,000 elements) runs before any roxmltree
|
||||
parse. The committed file must rebuild to itself byte for byte, the rule must change the tree,
|
||||
and the ruled text must equal the produced one byte for byte. Only `review/app-*` may use the
|
||||
rule; the overlays compare as bytes only. I found no path that accepts a difference other than
|
||||
the declared trigger-name rule. A trigger whose own English name equals the view's French name
|
||||
is refused (a false refusal, and a loud one).
|
||||
- Panics and resources: slicing only at ASCII `**` offsets, `get` elsewhere. The two indexings
|
||||
(`style()[0]`, `split_field(..)?[0]`) cannot be empty. The guard is linear (splits into
|
||||
pieces, never suffixes) and the renderers are indexed. Record memory stays as declared in
|
||||
§11.13.
|
||||
- `import-equine --check`: `legacy-import: 6 files, 0 differences`; `draft: declared:
|
||||
review/app-main.svg (byte-identical once the R3 trigger-name rule is applied)`; `draft: 12
|
||||
files, 0 differences`. `--out` (read): `create_dir` / `create_new` only, `Normal` components
|
||||
only, file names built from validated view ids, seed identity checks as declared.
|
||||
|
||||
## Reproduction
|
||||
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/probe.x3u0rG/`:
|
||||
`rs/` (probe crate, binaries `gateprobe` and `e2e`), `js/fmt.mjs` (Prettier),
|
||||
`js/ts_e2e.mts` (TypeScript pipeline), `js/gen*.py` (probe generators), `e2e_in.json`,
|
||||
`e2e_out.json`, `ts_out.json`, `cmp1.txt`, `cmp2.txt`, `cmp4.txt` (divergence lists).
|
||||
@@ -0,0 +1,166 @@
|
||||
# Security gate R5.4, round 4 (fresh): feature/rust-r5-pipeline-core @ 6d4fdd6
|
||||
|
||||
Scope: `git diff c6e0d85..HEAD` (36 existing files), focus on 124d954 (`curation/report_guard.rs`,
|
||||
`curation/markdown.rs`, `draft/{pipeline,fields,ops}.rs`). Product files read whole: report_guard.rs,
|
||||
markdown.rs, changelog.rs, review_report.rs, overlay.rs, drawing.rs, app_review.rs, svg_compare.rs,
|
||||
labels.rs, stats.rs, bin/import_equine.rs, plus the R5.4 hunks of fields.rs, ops.rs, pipeline.rs, mod.rs.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: BLOCK(3)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 36 files
|
||||
BLOCKING:
|
||||
1. [HIGH] markdown-guard/character-reference — crates/core/src/curation/report_guard.rs:43 — a character reference without its `;`, or with a name over 32 characters, passes the guard and Prettier rewrites it: `a * b` → `a \* b`, `*` → `\*`, `_` → `\_`, `a_b` → `a_b`, `�…042;` (35-character name) → `\*`; a non-terminated reference followed later by a bare `&` before `,` or `"` makes Prettier overwrite that `&`: `< x &` → `< x <`, `A x &` → `A x A` — hint: outside code spans refuse `&#` whatever follows, and `&` followed by a legacy (non-terminated) entity name with or without `;`; keep `&` before other text admitted (the equine REVIEW-REPORT holds `&pid=S1019` in a URL)
|
||||
2. [HIGH] markdown-guard/image-reference — crates/core/src/curation/report_guard.rs:33 — `![…]` is not among the refused constructs; Prettier parses it as an image reference and prints the alt from its decoded, unescaped value: `![a\"b]` → `![a"b]`, `![a\|b]` → `![a|b]`, `![x|y]` in a CHANGELOG cell, `![&]` → `![&]`, `![<]` → `![<]`, `![*]` → `![*]`; the `![` and its `]` can come from two insertions on one list line (title, rationale, question) — hint: refuse `![` outside code spans (the equine reports hold none)
|
||||
3. [HIGH] markdown-guard/wiki-link — crates/core/src/curation/report_guard.rs:34 — the padded-wiki check splits on `[[`, so the text between a `[[` and the first `]]` is never examined when another `[[` sits in between; Prettier's tokenizer (`/^\[\[(?<linkContents>.+?)\]\]/su`, contents trimmed) matches across it: `[[ a [[b]]` → `[[a [[b]]`, also `[[ é [[中]]`, `[[ 1 [[2]]`, `[[a]] [[ b [[c]]`, `[[ x [ [[y]]` — hint: for every `[[` occurrence (overlapping `[[[` included) take the text up to the first following `]]` and refuse it when it starts or ends with whitespace, or refuse `[[` followed by whitespace and whitespace followed by `]]`
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] declaration-wrong (guarantee overstated) — docs/development/RUST.md:3164, 3169, 3184, 3214
|
||||
- [LOW] resource/quadratic-underscore-check — crates/core/src/curation/report_guard.rs:88
|
||||
- [LOW] lint-allow in test files (convention, not a scanner suppression) — crates/core/tests/curation_review.rs:6, crates/core/tests/curation_markdown.rs:6
|
||||
PROOF: semgrep 128 rules on 36 files → 0 findings; checklist 13 checks → 6 findings
|
||||
```
|
||||
|
||||
## Severity basis
|
||||
|
||||
The three blocking findings break the owner requirement this guard exists for: a report is byte for
|
||||
byte what the TypeScript+Prettier pipeline wrote, or it is refused loudly. Each one is a silent
|
||||
difference reachable through the normal authoring channel (a title, rationale or question in
|
||||
`changes.json`, a citation in `references.json`, a label set by an op), with no refusal and a clean
|
||||
exit. There is no confidentiality or availability impact and no actor beyond the author of the
|
||||
curation files; the classification rests on the explicit integrity requirement and on the fact that
|
||||
RUST.md states the opposite (finding M1). In two families the TypeScript output is the one that alters
|
||||
the author's text (`< x &` became `< x <`), so the fix is a refusal, never an imitation of
|
||||
Prettier.
|
||||
|
||||
## Method and evidence (differential, all in the scratchpad)
|
||||
|
||||
- TypeScript side: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/ts/sweep.mts`,
|
||||
the sweep generator's exact `changesFile(probe)` on the mini sheet, run through `packages/anatomy/src`
|
||||
(`applyChangeset`, `renderChangelog`, `renderReviewReport`) and `formatMarkdown` (Prettier 3.8.3,
|
||||
options of `packages/anatomy/src/format.ts`).
|
||||
- Rust side: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/rs/src/main.rs`,
|
||||
a throwaway binary linking `faunosteo-core` (`curation`), same changeset through `apply_changeset`,
|
||||
`render_changelog`, `render_review_report`, `format_report`. Raw texts matched the TypeScript in every
|
||||
case (no RAWDIFF), so every difference is the guard admitting what Prettier rewrites.
|
||||
- Batches: 79 hand-picked probes (20 silent), 2,500 seeded random strings over the admitted alphabet
|
||||
plus `&#`, `&`, `<`, `![`, `[[` atoms (151 silent, every one holding `&#`, a legacy entity or
|
||||
`![`), 3,000 seeded random strings without `&`, `![` or `[[` (0 silent), 55 targeted edges (7 silent),
|
||||
13 entity variants (2 silent). Total 5,647 probes, 180 silent differences, all in the three families.
|
||||
Probe lists and results: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/out/`.
|
||||
- Prettier internals confirmed in `prettier@3.8.3/plugins/markdown.js`: wiki tokenizer
|
||||
`/^\[\[(?<linkContents>.+?)\]\]/su` with `.trim()`, placed before `link`; the unescape restore keeps the
|
||||
source of a one-character text node except `*` and `_` (`Tf=/^\\?.$/su`), which is why `*` and
|
||||
`_` come out as `\*` and `\_`; `imageReference` prints `node.alt`, the decoded value.
|
||||
|
||||
Sample (review list item, probe in the title):
|
||||
|
||||
```
|
||||
rust: - `CHG-001` Titre a * b (niveau C, …
|
||||
ts: - `CHG-001` Titre a \* b (niveau C, …
|
||||
rust: - `CHG-001` Titre ![a\"b] (niveau C, …
|
||||
ts: - `CHG-001` Titre ![a"b] (niveau C, …
|
||||
rust: - `CHG-001` Titre [[ a [[b]] (niveau C, …
|
||||
ts: - `CHG-001` Titre [[a [[b]] (niveau C, …
|
||||
rust: … `probe-ref` Citation < x &, cité par 1 nœud(s).
|
||||
ts: … `probe-ref` Citation < x <, cité par 1 nœud(s).
|
||||
```
|
||||
|
||||
In the CHANGELOG the same probes also shift the table alignment (Prettier pads on the rewritten text).
|
||||
|
||||
## Findings in detail
|
||||
|
||||
### H1 [HIGH] character references not terminated by `;` (report_guard.rs:43-53)
|
||||
|
||||
`reference` requires `after.contains(';')` and `name.len() <= 32`. The parser Prettier uses decodes
|
||||
non-terminated numeric references (`*`, `*`, `*`) and the legacy named ones (`<`, `&`,
|
||||
`©`…), and numeric references take any number of leading zeros, so the 32 cap lets
|
||||
`*` through. A reference decoding to `*` or `_` is escaped
|
||||
(`\*`, `\_`) or merged (`a_b` → `a_b`); a non-terminated reference followed later by a bare `&` (seen
|
||||
before `,` and `"`, i.e. at the end of a citation or a JSON value) gets that `&` replaced by the decoded
|
||||
character. Fix direction: refuse `&#` outside code spans unconditionally, and `&` followed by a legacy
|
||||
entity name (the non-terminated set) regardless of `;`; keep the existing `&name;` rule; keep `&` before
|
||||
anything else admitted, since the equine REVIEW-REPORT carries `&pid=S1019` inside a URL. Add the
|
||||
reproducers above to `r5.markdown-sweep-oracles.test.ts`.
|
||||
|
||||
### H2 [HIGH] image references (report_guard.rs:33-55)
|
||||
|
||||
`rewritten_construct` refuses `](` and `][` but not a shortcut image reference `![…]`. Its alt is printed
|
||||
from the decoded, unescaped value, so any `\"`, `\|` (including the `\|` that `cell()` adds in the
|
||||
CHANGELOG) or reference inside it is rewritten. `![a]` with plain text stays identical, which is why the
|
||||
sweep's `` (refused through `](`) never showed it. Fix direction: refuse `![` outside code
|
||||
spans.
|
||||
|
||||
### H3 [HIGH] wiki links whose contents hold a `[[` (report_guard.rs:34-41)
|
||||
|
||||
`plain.split("[[")` yields the segments between occurrences, not the text the regex matches: for
|
||||
`[[ a [[b]]` the first segment ` a ` holds no `]]` and is skipped, the second `b]]` is unpadded, so the
|
||||
line is admitted while Prettier matches ` a [[b` and trims it. Fix direction: emulate the lazy match
|
||||
from each `[[` position (a `]]` at offset 1 or more), or refuse `[[` followed by whitespace and
|
||||
whitespace followed by `]]`.
|
||||
|
||||
### M1 [MEDIUM] declarations overstated (docs/development/RUST.md)
|
||||
|
||||
- Line 3164: "a `[[ … ]]` padded inside" is refused: false for H3.
|
||||
- Line 3169: "so an unknown form is refused by default": false; the allowlist is per character, and
|
||||
constructs built from admitted characters (`&`, `#`, `!`, `[`, `]`) are admitted.
|
||||
- Line 3184: "the constructs refused whole (links, HTML tags, autolinks, character references)": false for
|
||||
non-terminated or long references (H1), and image references are not refused at all (H2).
|
||||
- Line 3214: the residual risk says an uncovered combination "could still differ" and that the last rounds
|
||||
"found none": three families with 180 reproducers were found in one round. The declaration must be
|
||||
rewritten with the fixes, and the sweep extended with these families.
|
||||
|
||||
### L1 [LOW] quadratic underscore check (report_guard.rs:88)
|
||||
|
||||
For each `_`, `chars[i + 1..].iter().collect::<String>()` copies the rest of the text to test the URL
|
||||
starts: O(n²) per authored text. Measured with the probe binary: 10 KB of `a_` 14 ms, 40 KB 221 ms, 80 KB
|
||||
0.91 s (x4 per doubling), so about 2.5 min per occurrence at 1 MB and hours near the 16 MiB file cap; a
|
||||
title or rationale is checked in up to four places. Owner-authored input of a dev tool, hence LOW. Fix
|
||||
direction: compare the slice in place (`chars[i + 1..].starts_with(&['h','t','t','p',…])` or byte offsets
|
||||
on the `&str`).
|
||||
|
||||
### L2 [LOW] lint allows in the new test files (context)
|
||||
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` heads
|
||||
`curation_review.rs` and `curation_markdown.rs`, the same header as the pre-existing `curation_draft.rs`:
|
||||
test-only file access and unwraps, not a security-scanner suppression. No `nosemgrep`, `nosec`,
|
||||
`eslint-disable`, `ts-ignore` or new allow in library code.
|
||||
|
||||
## Checklist (13 checks)
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Hardcoded secret, token, key, auth URL (added lines, incl. docs and oracle JSON) | none |
|
||||
| 2 | SQL built by concatenation | none in the diff (the seed SQL writer is R5.3, unchanged) |
|
||||
| 3 | Unsanitized render (innerHTML, eval, raw) | none; SVG text goes through the R3 serializer (`& < > "` and tab/newline escaped in attributes) after `first_unwritable` (XML 1.0 code points, CR refused) |
|
||||
| 4 | Sensitive endpoint without authz | not applicable (offline dev tool, no endpoint) |
|
||||
| 5 | Stack trace or internal path to a user | not applicable (CLI stderr for the developer) |
|
||||
| 6 | Secret or PII in a log | none (stdout and stderr carry file names, counts, refusals) |
|
||||
| 7 | Tracked `.env` or credential file | only `.env.example`; nothing in the diff |
|
||||
| 8 | Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`) outside the diff, filtered |
|
||||
| 9 | New suppressions (anti-gaming) | none in library code; L2 |
|
||||
| 10 | Markdown guard differential (5,647 probes) | H1, H2, H3, M1 |
|
||||
| 11 | Strict SVG comparison | holds (below) |
|
||||
| 12 | Resource use of the new code | L1; renderers indexed, document bound unchanged |
|
||||
| 13 | `--check` / `--out` behaviour | holds (below) |
|
||||
|
||||
## What holds
|
||||
|
||||
- Channels: both draft reports pass every raw line through `format_report` (pipeline.rs:273-274); no
|
||||
authored text reaches a heading; list-item fields with a control character are refused
|
||||
(`assert_list_items_single_line`); ids that reach code spans are stable ids or region codes, checked at
|
||||
the op (ops.rs `assert_stable_id`), field segments refuse controls (NUL included) and backticks; a line
|
||||
with an odd backtick count is refused, so no code span can hide a line break. The R5.2 IMPORT-REPORT
|
||||
stays on `format_markdown` with legacy sources only. View ids in file names are stable ids, and
|
||||
`write_exclusive` refuses any non-normal path component.
|
||||
- Strict SVG comparison (`svg_compare.rs`), 11 crafted cases against the committed `app-main.svg`: the
|
||||
legitimate trigger-name difference is the only acceptance; an extra comment, trailing text, a prolog
|
||||
change, another English name, a renamed attribute, a DOCTYPE, an empty or 100,000-level produced file,
|
||||
a 100,000-level committed file and a stylesheet PI in the committed file are all refused, none panics.
|
||||
- `--check` (binary built in the scratchpad, read-only): `legacy-import: 6 files, 0 differences`,
|
||||
`draft: declared: review/app-main.svg`, `draft: 12 files, 0 differences`, exit 0.
|
||||
- `--out` into a fresh `mktemp -d` under the scratchpad: 18 files written; a second run refused
|
||||
`not empty, nothing written` (exit 2); legacy identical to the seed, draft identical except
|
||||
`app-main.svg`. Working tree clean afterwards.
|
||||
- Panics: none found in the new code (slices after an ASCII `**` match, `chars[i + 1..]` with `i < len`,
|
||||
`compact_value` cut bounded by the length test, overlay NaN paths printed as `String(number)`).
|
||||
- semgrep: 128 rules (p/security-audit, p/secrets, p/owasp-top-ten) on 36 files, 0 findings, 0 errors.
|
||||
@@ -0,0 +1,192 @@
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 37 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] report-guard/wiki-link-across-code-spans (silent family F2) — crates/core/src/curation/report_guard.rs:55-59 (rule applied per plain part from :177-183); crates/core/src/curation/draft/review_report.rs:90-100
|
||||
- [MEDIUM] report-guard/opener-closed-inside-code-span (silent family F1) — crates/core/src/curation/report_guard.rs:178-179; crates/core/src/curation/draft/fields.rs:104-112,123-125; crates/core/src/curation/draft/review_report.rs:91-98
|
||||
- [LOW] declaration/code-spans-called-generator-only — docs/development/RUST.md:3166-3173,3176-3178,3221-3230; crates/core/src/curation/report_guard.rs:14-17
|
||||
- [LOW] report-guard/quadratic-cost-per-line — crates/core/src/curation/report_guard.rs:116-122,126-141,55-59
|
||||
- [LOW] anti-gaming/test-only-clippy-allow (context, not a security suppression) — crates/core/tests/curation_review.rs:6; crates/core/tests/curation_markdown.rs:6
|
||||
PROOF: semgrep 162 rules on 37 files → 0 findings; checklist 9 checks → 1 finding (LOW); guard differential (Rust format_report vs Prettier 3.8.3 with the repository options) 147,789 report documents, then the Rust pipeline vs the TypeScript pipeline end to end on 13 changesets → 2 silent families reproduced end to end (MEDIUM), 2 LOW
|
||||
|
||||
# Security gate 5, R5.4 review files (2026-10-23)
|
||||
|
||||
Scope: `git diff c6e0d85..1b12d90` (HEAD of `feature/rust-r5-pipeline-core`), 37 files, read from a
|
||||
pristine `git archive` copy of 1b12d90 in the session scratchpad. Product files read whole:
|
||||
`curation/report_guard.rs`, `curation/markdown.rs`, `curation/mod.rs`, `draft/changelog.rs`,
|
||||
`draft/review_report.rs`, `draft/svg_compare.rs`, `draft/drawing.rs`, `draft/overlay.rs`,
|
||||
`draft/app_review.rs`, `draft/fields.rs`, `bin/import_equine.rs`, the diffs of `pipeline.rs`, `ops.rs`,
|
||||
`mod.rs` and `Cargo.toml`, RUST.md §11.14 and §11.15, and the TypeScript sweep generator.
|
||||
|
||||
## Grading
|
||||
|
||||
No HIGH or CRITICAL: the inputs are the owner's own `changes.json` and `references.json`, the tool is
|
||||
never deployed and exposes nothing, and no finding crosses a security boundary (no write outside the
|
||||
claimed folder, no seed modification, no code execution). The two MEDIUMs breach the owner's
|
||||
stated contract for this tool ("a report never differs silently from what the TypeScript+Prettier
|
||||
pipeline wrote: byte-identical or loudly refused"). The gate does not block on them, but by that
|
||||
contract they are defects to fix before R5.4 closes. Both were reproduced end to end: the raw
|
||||
texts of the two pipelines are identical, the Rust `CHANGELOG.md` is identical, and the Rust
|
||||
`REVIEW-REPORT.md` differs from the TypeScript one with no refusal.
|
||||
|
||||
Both new families have one root cause. The guard decides inline constructs per stretch of text
|
||||
between two backticks. Prettier's parser builds wiki links, link references and liquid tags across
|
||||
the generator's code spans, and its bracket counting does not see code spans at all.
|
||||
|
||||
## MEDIUM 1: wiki link split across two authored fields (F2)
|
||||
|
||||
`rewritten_construct` (report_guard.rs:53-80) runs on each plain part separately (`guard_text`,
|
||||
:177-183). Its padded wiki rule (:55-59) needs the `[[` and the `]]` in the same plain part. In
|
||||
the review report, `change_line` (review_report.rs:90-100) writes the title and the rationale
|
||||
before the generated `Objets : \`…\`` and `Références : …` code spans and the question after them,
|
||||
all on one line. A `[[` in the title or rationale and a `]]` in the question form one wiki link
|
||||
that spans the code spans. Prettier's wiki-link plugin trims its content, so a space next to either
|
||||
bracket is dropped. The guard never sees the pair.
|
||||
|
||||
Reproduced end to end on the sweep's mini sheet (Rust `apply_changeset`, `render_review_report`,
|
||||
`format_report` against the TypeScript `applyChangeset`, `renderReviewReport`, `formatMarkdown`).
|
||||
Changeset: one level C change, op `set finding:hip/hip-dysfunctions/mobility name.en = "Mobility"`,
|
||||
title `Mobilité`, rationale `Libellé repris de [[`, question `Faut-il garder ]] ?`:
|
||||
|
||||
```
|
||||
rust: … Libellé repris de [[ Objets : `finding:…name.en`, … Question : Faut-il garder ]] ?
|
||||
ts : … Libellé repris de [[Objets : `finding:…name.en`, … Question : Faut-il garder]] ?
|
||||
```
|
||||
|
||||
The same happens with title `Renommer [[mobilité` and question `Garder le nom ]] ?`, with rationale
|
||||
`Voir [[ la fiche` and question `Confirmer]] ?`, and with title `Mobilité [[` (the generator's
|
||||
following space pads it). Three random seeds of a mixed sweep without any field trick (48,000
|
||||
documents, 26,244 admitted) gave 55 silent differences, all from this family.
|
||||
|
||||
How contrived: the changeset is fully valid and the document is untouched. It only needs wiki-link
|
||||
brackets split between the title or rationale and the question of one change, with a space next to
|
||||
one bracket. Wiki-link syntax is rare in French clinical prose, so accidental occurrence is
|
||||
unlikely, but no malformed data is involved.
|
||||
|
||||
Fix direction: evaluate the construct rules on the whole line, code spans included, the way the
|
||||
parser does (at least: a `[[` with a later `]]` anywhere on the line, a space next to either);
|
||||
or refuse `[[` and `]]` in the list-item fields unless they form a closed, unpadded pair inside the
|
||||
same field. Add cross-field probes to the sweep. Today it puts the same probe in every field, so
|
||||
an opener in one field and its closer in another lies outside it by construction.
|
||||
|
||||
## MEDIUM 2: an authored opener closed inside a generated code span (F1)
|
||||
|
||||
Inside a code part, the guard admits anything but a carriage return or a tab (:178-179). The module
|
||||
doc and RUST.md call code spans "the generator's alone (ids, targets, field names)", but field
|
||||
names are authored. `is_settable` checks only the first segment of a `set` field
|
||||
(fields.rs:123-125), and `split_field` refuses only empty segments, control characters, backticks,
|
||||
NUL starts and `__proto__` (:91-121). Any other sub-segment reaches the review report's `Objets`
|
||||
code spans through `record_subject`. If the title or the rationale holds an unclosed `[`, `[[` or
|
||||
`{%`, the parser closes it at a `]`, `]]` or `%}` inside that code span, re-pairs every backtick
|
||||
after it, and re-prints the rest of the span as Markdown.
|
||||
|
||||
Reproduced end to end. Change title `Mobilité [voir note`. Ops:
|
||||
`set finding:hip/hip-dysfunctions/mobility name.<130 × "p">.a]_ = "x"`, then
|
||||
`set … name = {"fr":"Mobilité","en":"Mobility"}`. The second op makes the final document valid
|
||||
again, and the padding pushes the stray key past the CHANGELOG's 120-unit value cut, where the guard
|
||||
would otherwise refuse it:
|
||||
|
||||
```
|
||||
rust: … Objets : `finding:…name.ppp….a]_`, `finding:…name`, …
|
||||
ts : … Objets : `finding:…name.ppp….a]\_`, `finding:…name`, …
|
||||
```
|
||||
|
||||
Other reproduced variants: title `Titre [[x` with segment `a]] *b*` (TS prints `_b_`), `Titre [x`
|
||||
with `a][b` (TS prints `a]b`), `Titre {%x` with `a%} *b*`, and the opener in the rationale. The
|
||||
same breakage also turns the tail of a code span into live Markdown. With segment
|
||||
`a] [lien](https://example.org) <b>gras</b> x` and title `voir [x`, the parsed report holds a link
|
||||
and two inline HTML nodes, and `, ` becomes a code span. Both pipelines print that identically, so
|
||||
it is no silent difference, but it gets past the guard's "no link, no HTML tag" rule. The author
|
||||
already controls the file, so nothing is gained in privilege.
|
||||
|
||||
How contrived: very. It needs a field path with a stray sub-segment (a typo-class authoring error)
|
||||
that a later op overwrites or removes, because every sheet object is strict and the final document
|
||||
would be refused otherwise. The stray key must also stay out of the first 120 UTF-16 units of every
|
||||
CHANGELOG value, or the guard refuses it there and the whole run stops. And the change's title or
|
||||
rationale needs an unclosed opener.
|
||||
|
||||
Fix direction: refuse at `split_field` any segment outside the id alphabet (`[A-Za-z0-9_-]`, which
|
||||
every settable path uses: `name.fr`, `view_box.min_x`, `count_range.min`). That is a declared
|
||||
refusal where the TypeScript accepted. Then make the code-span rule an allowlist of what the
|
||||
generator really writes (ids, `CHG-ddd`, `kind:id/…`, dotted field names).
|
||||
|
||||
## LOW: the residual-risk declaration is accurate in kind, wrong in two specifics
|
||||
|
||||
§11.14 and §11.15 correctly call the guarantee empirical and say an unprobed combination could
|
||||
still differ. Two statements are false as written, though. "Code spans are the generator's alone
|
||||
(ids, targets, field names); inside them, anything but a carriage return or a tab" (RUST.md
|
||||
:3170-3173, report_guard.rs:14-17) hides that field sub-segments are authored, and that this channel
|
||||
has the weakest rule of the guard. "Nested padded wiki links" and the other constructs are listed
|
||||
as ruled (:3166-3168), but they are ruled only within one stretch between backticks. The sweep's
|
||||
blind spot also deserves a line: one probe copied into every field never puts an opener and its
|
||||
closer in two different fields. Fix: correct the module doc and §11.14 and §11.15 when F1 and F2 are
|
||||
fixed.
|
||||
|
||||
## LOW: quadratic guard cost per line
|
||||
|
||||
`in_url_token` (:116-122) scans back to the token start for every `&` that is followed by a
|
||||
lowercase letter (:129). The padded-wiki rule (:55-59) runs `contains` over the rest of the part
|
||||
for every `[[` or `]]`. Measured on the release build: a 160 kB token
|
||||
`https://x.org/?a=1&b=1&b=1…` takes 4.1 s and a 640 kB one 28.4 s. 640 kB of ` ]]` takes 5.6 s. Memory
|
||||
stays flat (8 MB). Authored texts have no length cap below the 16 MiB file bound
|
||||
(`MAX_FILE_BYTES`), and each text is guarded in two or three lines, so a multi-megabyte rationale
|
||||
stalls the run for minutes to hours. It is owner-authored and can be interrupted. Fix: one forward
|
||||
pass (track the current token's start and whether it holds a URL literal; precompute the first `[[`
|
||||
and the last `]]` of the line), or a length cap on authored texts.
|
||||
|
||||
## LOW: new lint allows (anti-gaming, context)
|
||||
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` heads the two new
|
||||
integration test files (curation_review.rs:6, curation_markdown.rs:6), the same as the
|
||||
pre-existing curation_draft.rs:7. These are test-only lint allows (oracle file reads, unwraps) in
|
||||
separate test crates. They do not weaken the library's no-I/O rule and do not silence a security
|
||||
scanner. The diff adds no `nosemgrep`, `nosec`, `eslint-disable`, `ts-ignore` or `noqa`. Not
|
||||
blocking.
|
||||
|
||||
## Checked and clean
|
||||
|
||||
- semgrep, pinned packs, `--metrics=off` (EIO_BACKEND=posix), 37 files (rs, ts, md, json, toml):
|
||||
0 results, 0 errors.
|
||||
- Checklist: no secret, token, key or credential URL in the added lines (pattern grep plus
|
||||
p/secrets). No SQL built in the diff (the seed SQL renderer is unchanged R5.3 code). No
|
||||
innerHTML, `eval` or HTML sink. No endpoint, so no authz surface. No secret in a log. No tracked
|
||||
`.env` or credential file (the tracked `.env.example` is the placeholder convention, outside the
|
||||
scope; reading it was refused by a permission deny rule, so it was not opened).
|
||||
- Guardrails scanner (`--domain api,supabase`, on the pristine copy): 0 FAIL, 1 WARN, 1 MANUAL. The
|
||||
WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`) is in
|
||||
a file outside the diff and is excluded.
|
||||
- No authored text reaches a report outside the guard: both reports go through `format_report`
|
||||
(pipeline.rs). Every authored field the reports print passes `assert_authored` (changelog.rs:70-71,
|
||||
91-92; review_report.rs:361-389). Cell pipes are escaped and an authored `\|` is refused. Line
|
||||
breaks become spaces in cells and are refused in list items. A row with another cell count is
|
||||
refused. No authored text ever starts a line, so no heading, section, list or table can be forged.
|
||||
Code spans are the exception, see F1.
|
||||
- Drawings: every label, id, path and number goes through the core serializer's escaping, and
|
||||
`first_unwritable` refuses XML-invalid text and CR. View ids used in file names are stable ids,
|
||||
and `write_exclusive` accepts only plain components.
|
||||
- Strict SVG comparison (`svg_compare.rs`): the committed file must rebuild byte for byte from its
|
||||
tree, the rule must change that tree, and the rebuilt text after the rule must equal the produced
|
||||
file byte for byte. The rule touches only `data-anno-name-en` and `FR / FR` titles of elements
|
||||
that open a view, so it cannot hide another difference.
|
||||
- `import-equine --check`, built in the scratch copy (read-only): `legacy-import: 6 files, 0
|
||||
differences`, `draft: declared: review/app-main.svg …`, `draft: 12 files, 0 differences`, exit 0.
|
||||
Its `--out` path checks (no `..`, parent canonicalized, seed refused by path and by inode,
|
||||
symbolic link refused, non-empty refused, `create_dir` and `create_new` only) were read, not run.
|
||||
- Panics: none found by reading the guard's slicing. A fuzz of 20,000 random multi-line documents
|
||||
(tabs, CR, combining marks, U+2028, emoji, ZWJ, backticks, brackets) produced no panic.
|
||||
|
||||
## Method
|
||||
|
||||
Throwaway probes in the scratchpad only: `examples/guard_probe.rs` (calls `format_report`) and
|
||||
`examples/pipeline_probe.rs` (the sweep's pipeline on a changes file), linked against the pristine
|
||||
copy with `--features curation`; Node scripts running the repository's Prettier 3.8.3 with the
|
||||
options of `packages/anatomy/src/format.ts`, and the TypeScript pipeline of `packages/anatomy/src`
|
||||
through the tsx loader. Differential harness: raw report lines built like the reports (list items,
|
||||
ordered items, table rows with `cell()` escaping), admitted by Rust then compared with Prettier.
|
||||
Code-span channel: 1,476 documents, 0 silent without an opener. Random authored text: 66,000
|
||||
documents, 48,737 admitted, 0 silent. Openers against code-span contents: 8,312 documents (F1).
|
||||
Mixed fields: 72,000 documents (F1 and F2 only, every silent case classified). Fuzz: 20,000.
|
||||
End to end: 13 changesets. No `--out` was run, nothing was deleted, and no file of the repository
|
||||
was modified except this report.
|
||||
@@ -0,0 +1,153 @@
|
||||
# Security gate R5.4, round 6 (fresh, blind)
|
||||
|
||||
Branch `feature/rust-r5-pipeline-core`, HEAD `aeb5050`. Scope:
|
||||
`git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`
|
||||
(25 files). Mode: audit (this file is the only write). The brief carried a list of earlier
|
||||
findings; per the auditor contract it was not used as a checklist. The scan below is complete
|
||||
on its own and covers those areas anyway.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 25 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] resource/changelog-table-amplification — crates/core/src/curation/draft/changelog.rs:100 (via crates/core/src/curation/markdown.rs:133)
|
||||
- [LOW] resource/per-view-drawing-amplification — crates/core/src/curation/draft/pipeline.rs:204
|
||||
- [LOW] resource/guard-underscore-pairing-quadratic-in-isolation — crates/core/src/curation/report_guard.rs:216
|
||||
- [LOW] markdown/invisible-hangul-fillers-admitted — crates/core/src/curation/report_guard.rs:159
|
||||
- [LOW] markdown/unpadded-wiki-link-in-table-cell (context, by design) — crates/core/src/curation/report_guard.rs:58
|
||||
- [LOW] anti-gaming/clippy-allow-in-new-test-file (context, convention) — crates/core/tests/curation_markdown.rs:6
|
||||
- [LOW] anti-gaming/clippy-allow-in-new-test-file (context, convention) — crates/core/tests/curation_review.rs:6
|
||||
PROOF: semgrep 162 rules on 25 files → 0 findings; checklist 13 checks → 7 findings
|
||||
```
|
||||
|
||||
## What was run
|
||||
|
||||
- semgrep 1.168.0, the three pinned packs, `--metrics=off`, on the 25 existing files of the
|
||||
diff: 162 rules, 25 files scanned, 0 results, 0 errors.
|
||||
- Checklist greps on the added lines: secrets, SQL concatenation, unsafe render, logging,
|
||||
suppressions, tracked `.env` / credential files (only `.env.example` is tracked). Nothing
|
||||
found. No endpoint in scope (authz check N/A).
|
||||
- Guardrails scanner (`--domain api,supabase`): 0 FAIL, 1 WARN, 1 MANUAL. The WARN
|
||||
(`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`) is
|
||||
outside the diff, not counted.
|
||||
- `cargo test -p faunosteo-core --features curation`: every suite green. `cargo clippy
|
||||
--all-targets -D warnings` (curation): clean. No `std::fs`, `std::env`, clock or process use
|
||||
in the new core modules (only the binary, which already carried its documented allow).
|
||||
- `import-equine --check`: legacy 6 files 0 differences; draft 12 files 0 differences, one
|
||||
`declared:` line for `review/app-main.svg`.
|
||||
- Differential fuzz of `report_guard::format_report` against Prettier 3.8.3 (the TypeScript
|
||||
formatter, same options as `packages/anatomy/src/format.ts`), through a scratch binary on the
|
||||
core library. 10,201 generated reports (seeded random strings over the admitted alphabet plus
|
||||
Markdown tokens: URL starts, entities, `**`, `[[`, `](`, `<a`, `{%`, `[^1]`, `:warning:`,
|
||||
line-start markers), placed in the list-item fields (with the pipeline's own
|
||||
`assert_single_line` / `assert_authored` pre-filters) and in CHANGELOG table cells (with the
|
||||
`cell()` escape). 2,226 admitted, 7,975 refused. 0 admitted report differs from Prettier's
|
||||
output. AST census of every admitted report (Prettier's own Markdown parser): heading, list,
|
||||
item, table, row and code-span counts always equal the template's; no link other than a bare
|
||||
URL autolink literal, no image, HTML, emphasis, footnote, definition, break, liquid node.
|
||||
One `wikiLink` node, in a table cell (see LOW 5).
|
||||
- Table width parity: 28 unusual letters (combining marks with the Alphabetic property, Indic
|
||||
vowel signs, CJK, fullwidth, halfwidth, Hangul jamo and fillers, non-BMP letters) in a table
|
||||
cell: 26 admitted and identical to Prettier, 2 refused. No silent alignment difference.
|
||||
- Guard timing on 12 pathological 0.4 to 1 MB lines (debug build): 32 to 224 ms each, linear.
|
||||
- `--out` / `--check` file safety, 18 probes in the scratchpad: fresh folder (18 files, `diff
|
||||
-r` identical but the declared `app-main.svg`); non-empty folder, symbolic link (plain,
|
||||
trailing `/`, trailing `/.`), dangling link (target not created), path inside the seed,
|
||||
inside the seed through a linked parent, link to `draft/`, `..`, empty path, existing file,
|
||||
missing parent, `/`, extra arguments: every one refused, nothing written; non-absolute
|
||||
`INIT_CWD` falls back to the current folder. The seed and the git tree are unchanged.
|
||||
- Resource probe on the real equine pipeline (scratch binary, release build, `ulimit -v`).
|
||||
|
||||
## Findings
|
||||
|
||||
### MEDIUM 1: the CHANGELOG grows as rows × the widest cell of each column
|
||||
|
||||
`changelog.rs:100` `level_section` builds one table per level, and `format_report` aligns it
|
||||
(`markdown.rs:133` `align`, `:163` `padded`): every row is padded to the widest cell of its
|
||||
column. Titles and rationales are authored and uncut (only `Avant` / `Après` are cut to 120
|
||||
units). One change with a long rationale plus another change producing many records makes every
|
||||
row as wide as that rationale. Measured on the equine pipeline, all level A:
|
||||
|
||||
| added to changes.json | records | CHANGELOG | max RSS |
|
||||
|---|---|---|---|
|
||||
| +25 KB (200 subtree sets) | 5,226 | 7.8 MB | 45 MB |
|
||||
| +45 KB (same, one 20 KB rationale) | 5,226 | 98 MB | 233 MB |
|
||||
| +105 KB (same, one 80 KB rationale) | 5,226 | 381 MB | 788 MB |
|
||||
| +155 KB (600 sets, one 80 KB rationale) | 14,026 | 1.09 GB | 2.2 GB |
|
||||
|
||||
A rationale cloned into many records (the R5.3 shape) aborted at 4.4 GB under a 6 GB
|
||||
`ulimit` from +175 KB. Within the 16 MiB read cap the demand reaches terabytes: certain OOM of
|
||||
the tool, and `--out` would try to write a file of that size (into a fresh folder, never
|
||||
overwriting). Prettier aligns tables the same way, so the TypeScript had the same output. This
|
||||
is not the declared R5.3 residual (records cost ops × items × value size): here a single wide
|
||||
cell multiplies every row, with the records themselves staying small. No integrity or
|
||||
confidentiality impact; a local developer tool fed by an authored file that goes through review.
|
||||
Hint: compute the aligned size (rows × the sum of column widths) before building the strings and
|
||||
refuse past a stated cap with a loud `PipelineError`, or cap authored title / rationale length.
|
||||
Declare the bound in RUST.md next to the 200,000-value one.
|
||||
|
||||
### LOW 2: two drawings per view, no view count bound
|
||||
|
||||
`pipeline.rs:204` `review_drawings` writes an overlay and an application drawing per view, each
|
||||
a clone of the whole cleaned drawing (`overlay.rs:350`, `app_review.rs:27`), about 137 KB each,
|
||||
all held in memory before any write or comparison. A detail view needs about 25 document values
|
||||
(view, view box, trigger annotation), so the 200,000-value bound allows about 7,700 views:
|
||||
around 2.1 GB of output. Estimated, not measured. Same output as the TypeScript. Hint: the same
|
||||
output-size cap as MEDIUM 1 covers it.
|
||||
|
||||
### LOW 3: `underscore_pairs_later` is O(parts × length) on its own
|
||||
|
||||
`report_guard.rs:216`: for each plain part holding `_` it rescans every later code span. 40,000
|
||||
`a_b \`cN\`` parts (509 KB) took 213 s (debug). The generator never writes such a line: at most
|
||||
two authored parts per line (title + rationale, question) and the separators between code spans
|
||||
hold no `_`; the realistic worst case, one underscore part followed by 100,000 code spans (1 MB),
|
||||
takes 224 ms. So the "linear guard" holds for the generator's line shapes only. Hint: one
|
||||
backward pass that records whether a later code span can close.
|
||||
|
||||
### LOW 4: invisible Hangul fillers pass the allowlist
|
||||
|
||||
`report_guard.rs:159` admits `char::is_alphanumeric`, which includes U+3164, U+115F, U+1160 and
|
||||
U+FFA0 (Hangul fillers, category Lo, rendered blank). An authored title or rationale can carry
|
||||
invisible runs in REVIEW-REPORT.md. No structure is forged and Prettier keeps them (identical
|
||||
output). Hint: refuse Default_Ignorable_Code_Point letters with the other refused classes.
|
||||
|
||||
### LOW 5 (context, by design): unpadded `[[x]]` admitted in a table cell
|
||||
|
||||
Only padded wiki links are refused in cells (`report_guard.rs:58`); list items refuse `[[`
|
||||
and `]]` whole (`review_report.rs:364`). Prettier parses `[[x]]` in a CHANGELOG cell as a
|
||||
`wikiLink` and prints it back byte for byte, so the output matches the TypeScript. Recorded so
|
||||
the per-cell rule stays a conscious choice.
|
||||
|
||||
### LOW 6 (context): clippy allow attributes in the two new test files
|
||||
|
||||
`tests/curation_markdown.rs:6` and `tests/curation_review.rs:6` add
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. They are not
|
||||
security-scanner suppressions (no `nosemgrep`, `nosec`, `eslint-disable`): they are the
|
||||
convention `clippy.toml` documents ("Tests and the worker binary allow these lints locally where
|
||||
they own I/O"), present in the same form in all 34 existing integration test files. Not
|
||||
blocking.
|
||||
|
||||
## Areas checked and found sound
|
||||
|
||||
- Report structure: no authored text reaches a line start (control characters refused in
|
||||
list-item fields, `\n` → space and `\r` refused in cells); code spans come from the generator
|
||||
only (authored backticks refused, content limited to `[A-Za-z0-9_:/. -]`, field segments to
|
||||
`[A-Za-z0-9_-]`, change ids `CHG-\d{3,}`, reference ids declared and stable, ids written by an
|
||||
op checked at the op); `](`, `][`, `![`, `<` before a letter / `/` / `!` / `?`, character
|
||||
references, `{%`, `%}` refused; `*` only as `**word**`; `_` only intraword plus the
|
||||
code-span closer rule; backslash only before `"` or `|`; table cell count checked.
|
||||
- SVG: elements and attribute names are fixed or come from the frozen drawing read by the
|
||||
core's reader (refused if anything is removed); values escaped (`& < > "`, `\t \n \r` as
|
||||
references in attributes); XML-invalid code points and CR refused (`drawing.rs:41`); file
|
||||
names built from stable view ids.
|
||||
- `--check`: the tree fallback only for `review/app-*`, only with the declared rule, and only
|
||||
when the committed file rebuilds from its tree byte for byte, the rule changes it and the
|
||||
result equals the produced bytes; the committed file goes through the core reader's prescan
|
||||
first.
|
||||
- `--out`: `..` refused, final symbolic link refused (trailing slash handled), parent
|
||||
canonicalised, seed refused by path and by (device, inode) of every seed folder, existing
|
||||
destination identity-checked and must be empty, folders `create_dir`, files `create_new`,
|
||||
components all `Normal`. The swap window between check and use stays as declared (§11.13).
|
||||
@@ -0,0 +1,88 @@
|
||||
# Security gate R5.4 (round 7): review files in Rust
|
||||
|
||||
Branch `feature/rust-r5-pipeline-core`, HEAD `777c800`. Scope: `git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md` (26 files). Blind, complete scan. No prior verdict was read.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: BLOCK(2)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 26 files
|
||||
BLOCKING:
|
||||
1. [HIGH] resource-bound/output-amplification — crates/core/src/curation/draft/changelog.rs:68 (record_row), :116-119 (table built whole); crates/core/src/curation/markdown.rs:52-63 (table), :139-159 (bound checked after the raw rows exist); crates/core/src/curation/draft/pipeline.rs:247-252; docs/development/RUST.md:3206 — the CHANGELOG raw table (rows × authored rationale/title) is fully built, joined and split before MAX_TABLE_COLUMNS_TOTAL is checked, so the declared case "80 KB rationale over 14,000 records" still exhausts memory: a 1.9 MB changes.json aborts at a 3 GB address-space cap; RUST.md's "refused before any row is built" is false — hint: bound before render_changelog builds a row (project rows × widest authored cell per level from the records, or cap the raw length while table() accumulates), and pin it with a pipeline-level test (build + render of the declared case under a memory ceiling), not format_markdown on a hand-built table
|
||||
2. [HIGH] resource-bound/record-amplification — crates/core/src/curation/draft/ops.rs:197 (rename), :384 (remove); crates/core/src/curation/draft/records.rs:18-28 (out of diff, same pipeline) — every rename/remove record deep-copies the whole item (before and after) and every record clones the ChangeSummary (title, rationale, question); MAX_DOCUMENT_VALUES (200,000) bounds the document after an op, never the sum of the records: an 810 KB changes.json (one 5,000-child add, 400 renames back and forth) reaches 4.3 GB RSS at build, linear at ~10.7 MB per rename; ~1,400 renames exceed the 15 GB dev machine; R5.4's CHANGELOG then serialises each whole item again (compact_value) — hint: bound the records (total values or bytes recorded, checked as each record is pushed, refused with Guard) or record ids/paths and share the ChangeSummary (index or Rc) instead of cloning
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] new lint relaxation `#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6 (integration tests only, same line as every existing test file; not a security-rule suppression)
|
||||
- [LOW] pre-existing suppression `#![allow(clippy::disallowed_methods)]` — crates/core/src/bin/import_equine.rs:17 (the tool's declared I/O file, context)
|
||||
- [LOW] error echo size — crates/core/src/curation/report_guard.rs:29-31 (UnsupportedMarkdown carries the whole refused line, multi-MB on stderr; authored content, no secret)
|
||||
- [LOW] subfolder race — crates/core/src/bin/import_equine.rs:353-389 (write_exclusive re-checks the claimed root only, not the subfolders this run created; same-user swap, declared out of model in RUST.md §11.13; unchanged by this diff)
|
||||
PROOF: semgrep 128 rules on 26 files → 0 findings; checklist 12 checks → 6 findings (2 HIGH, 4 LOW)
|
||||
```
|
||||
|
||||
## Checklist detail
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Hardcoded secret, token, key, credentialed URL | none in the added lines |
|
||||
| SQL by concatenation | none (no SQL in the diff) |
|
||||
| Unsanitized render (innerHTML, eval, raw) | none |
|
||||
| Sensitive endpoint without authz | not applicable (offline CLI and pure core) |
|
||||
| Stack trace or internal path to a user | none (CLI messages to the operator) |
|
||||
| Secret or PII in a log | none; `println!` lines print file names and counts |
|
||||
| Tracked `.env` or credential file | only `.env.example` tracked |
|
||||
| guardrails `scan.py --domain api,supabase` | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff: dropped) |
|
||||
| Anti-gaming (new suppressions) | two test-file lint allows, convention of all test files, LOW |
|
||||
| Report structure forgery (sections, list items, code spans, links, HTML) | held, see below |
|
||||
| SVG well-formedness and injection | held: serializer escapes `& < > "` and CR/TAB/LF in attributes, `first_unwritable` refuses non-XML-1.0 code points and CR, element and attribute names are fixed, no href or event attribute written |
|
||||
| Resource bounds and file safety | 2 HIGH (above); quadratic scans and `--out`/`--check` held, see below |
|
||||
|
||||
## Finding 1: CHANGELOG raw table amplified before its bound
|
||||
|
||||
`render_changelog` builds one row per record, each carrying the change's title and rationale (`changelog.rs:89`, `:94`). `table()` joins every row, wraps it, and joins again. `format_report` then guards each line, and `align` splits every row into owned strings. Only after all of that does it compare `row_width × rows` with `MAX_TABLE_COLUMNS_TOTAL`. So the bound caps the padded copy only. The raw table, which already holds N × R bytes, gets built four to five times first.
|
||||
|
||||
Measured with the full pipeline (`build_equine_draft` + `render_draft_artifacts` on the real legacy sources and the committed changeset, plus one appended change of `set` ops):
|
||||
|
||||
| changes.json | rationale × set ops | Result |
|
||||
|---|---|---|
|
||||
| 545 KB | 20 KB × 3,000 | Guard refusal, peak 373 MB |
|
||||
| 506 KB | 100 KB × 2,000 | Guard refusal, peak 1.19 GB |
|
||||
| 1.04 MB | 400 KB × 4,000 | `memory allocation failed`, signal 6, at the 6 GB cap |
|
||||
| 1.92 MB | 80 KB × 14,000 (the case RUST.md declares held) | `memory allocation failed`, signal 6, at the 3 GB cap |
|
||||
|
||||
The 16 MiB read cap allows inputs of hundreds of GB of raw table. Without a ulimit the process grows until the OOM killer acts. The failure is fail-closed: `folders()` finishes before `claim_destination`, so nothing gets written. The impact is availability of the developer machine, and that includes `--check`. The test `an_aligned_table_past_the_bound_is_refused_before_it_is_built` calls `format_markdown` on a table it built by hand, so it cannot see the pipeline's allocation.
|
||||
|
||||
## Finding 2: records deep-copy whole items
|
||||
|
||||
`ChangeRecord` keeps a `ChangeSummary` clone plus deep copies of `before` and `after`, and for a rename or remove those copies are the whole item (`records.rs:18`). An `add` of a 5,000-child category followed by N renames that swap its id back and forth gives:
|
||||
|
||||
| renames | build-only RSS |
|
||||
|---|---|
|
||||
| 0 | 56 MB |
|
||||
| 100 | 1.13 GB |
|
||||
| 200 | 2.20 GB |
|
||||
| 400 | 4.34 GB (810 KB changes.json) |
|
||||
|
||||
`MAX_DOCUMENT_VALUES` is checked on the document after each op, not on what the records accumulate. The cloning lines are R5.3 code, but they sit in a file of this diff (`ops.rs`, whose `apply_rename` the diff modifies), and the R5.4 renderers consume every record. The TypeScript had the same amplification but V8's heap limit stopped it with a clean error. The Rust process has no such ceiling.
|
||||
|
||||
## What held
|
||||
|
||||
Markdown guard. I ran a differential of `report_guard::format_report` (Rust) against Prettier 3.8.3 `format(…, parser: 'markdown')` with the repository options, on raw reports shaped like the generator's output: list items with title, rationale and question, CHANGELOG rows with `cell()` applied, bibliography and decision lines. Authored texts followed the same preconditions as the code (no backtick; no control character or wiki/liquid bracket in list items). Results:
|
||||
- 201,000 random probes over the admitted alphabet plus adversarial tokens (`&`, `&#`, URL starts, `**`, `_`, `\`, brackets, wide and zero-width characters). 20,969 were admitted and 0 diverged.
|
||||
- 960 targeted probes: reference-style links with a space, `[a]: b` mid-line, `1.`, `#`, `+`, `>` at a cell start, bare URLs with `_`, `)`, `&`, `x@y`, `**é**`, `a\|b` outside tables, `<3`, `<-`. 689 were admitted and 0 diverged.
|
||||
|
||||
None of these forged a section, list item, code span, link or HTML. A newline in a list-item field is refused, and table cells turn newlines into spaces. The guard's scans stayed linear: 18 lines of 4 MB with 20,000 code spans each ran in 1.19 s.
|
||||
|
||||
`--check` and `--out`:
|
||||
- `compare_file` falls back on trees only for `review/app-*` with the declared rule. `compare_drawing` returns an error with no rule, requires the committed file to rebuild byte for byte from its tree, and requires the rule to change it.
|
||||
- `inside_by_identity` now fails closed with an explicit error.
|
||||
- Reads stay canonical-inside-seed, regular-file-only and capped at 16 MiB.
|
||||
- Written names are `Component::Normal` only, and view ids are stable ids.
|
||||
- Writes use `create_new`, and the existing destination is identity-checked against the seed.
|
||||
|
||||
SVG: `drawing_text` refuses unwritable text, and the overlay and the app layer only emit fixed element and attribute names. Depth is bounded by the core reader's prescan before `first_difference` or `TriggerNames::apply` recurse.
|
||||
|
||||
## Probes (scratchpad only, no tracked change, nothing deleted)
|
||||
|
||||
- `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/probe/src/main.rs` (guard probe), `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/probe/src/bin/amp.rs` (pipeline memory probe; `BUILD_ONLY=1` stops after the build)
|
||||
- `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/scripts/fuzz.py`, `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/scripts/targeted.py`, `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/scripts/prettier.mjs`
|
||||
- `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/scripts/mkchg.py` (finding 1 inputs), `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/scripts/mkren.py` (finding 2 inputs)
|
||||
|
||||
Memory probes ran under `ulimit -v` (3 to 6 GB). Cargo built into a scratchpad `CARGO_TARGET_DIR`. `git status` stayed clean at `777c800`.
|
||||
@@ -0,0 +1,148 @@
|
||||
# Security gate R5.4, round 8 (fresh): 2026-10-23
|
||||
|
||||
Scope: `git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`,
|
||||
HEAD 5880dfa, branch `feature/rust-r5-pipeline-core`. Mode: audit (REPORT path given; this file
|
||||
is the only write). Blind run: no prior verdict or finding list read.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: BLOCK(1)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 28 files
|
||||
BLOCKING:
|
||||
1. [HIGH] resource-bound/records-checked-after-op — crates/core/src/curation/draft/apply.rs:140,145 (sites: apply.rs:66-96 mark_to_confirm, ops.rs:126-168 set_subtree, ops.rs:417-452 apply_cite, records.rs:61-76 Recorder::record) — the document-values and records-weight bounds only run after a whole op (and after the B/C marking); inside one op every record clones the change texts and the op value, once per item, so about 1 MB of changeset allocates GBs before any bound runs (measured: 711 KB → 1.37 GB RSS; 1.1 MB → abort at a 3 GB cap; 646 KB → 2.0 GB; 494 KB → 2.1 GB). RUST.md:3244-3245 says "tens of MB at most", which is false — hint: count the weight as each record is pushed (Recorder::record and mark_to_confirm), refuse before cloning; keep a per-op values budget for subtree writes (set/cite); or share ChangeSummary/Op (Arc) instead of cloning them; include CiteOp in record_weight; correct the RUST.md residual
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] resource-bound/review-drawings-per-view (declared residual, now measured) — crates/core/src/curation/draft/pipeline.rs:204-226
|
||||
- [LOW] record_weight ignores the op cloned into each record for cite/rename/remove/move — crates/core/src/curation/draft/records.rs:125-128
|
||||
- [LOW] TriggerNames::apply linear view search per trigger element (views × triggers, --check only) — crates/core/src/curation/draft/svg_compare.rs:36-50
|
||||
- [LOW] new test-file lint allows `#![allow(clippy::disallowed_methods, …)]`, test crates only, same convention as 20 earlier test files, not a security-scanner suppression — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
- [LOW] pre-existing (context) `#![allow(clippy::disallowed_methods)]` in the CLI — crates/core/src/bin/import_equine.rs:17
|
||||
PROOF: semgrep 162 rules on 28 files → 0 findings; checklist 15 checks → 5 findings (1 HIGH, 1 MEDIUM, 3 LOW) + 1 pre-existing LOW
|
||||
```
|
||||
|
||||
## Blocking finding 1: the records and values bounds run after the op
|
||||
|
||||
What the code claims: `apply.rs` bounds the records at 16 Mi weight units, "checked after every
|
||||
op, before the next one records anything". RUST.md §11.15 states the residual as "one op still
|
||||
records up to about three copies of an item under the 200,000-value bound before the records
|
||||
bound is checked (tens of MB at most)".
|
||||
|
||||
What happens. `Recorder::record` (records.rs:61-76) clones `self.change` (title, rationale,
|
||||
question, reference ids) and `self.op` (including a set's or an add's value) into every record.
|
||||
`assert_depth` and `assert_recorded` (apply.rs:139-140) run only once `apply_op` returns, and
|
||||
`mark_to_confirm` (apply.rs:66-96) is checked only after it has pushed one record per touched
|
||||
item (apply.rs:145). One op that reaches N items therefore allocates about N × (author text or
|
||||
value) before any bound runs. N is bounded only by the 200,000-value document cap; the size of
|
||||
the text or value is bounded only by the 16 MiB read cap.
|
||||
|
||||
Three paths, measured with a scratch probe built on `faunosteo-core` (`--features curation`,
|
||||
the real legacy import, the committed changes and references, with one or two hostile changes
|
||||
appended). Each run had `ulimit -v 3000000`:
|
||||
|
||||
| Path | changes.json | Peak RSS | Outcome |
|
||||
|---|---|---|---|
|
||||
| level B `add` of a node with 10,000 children, rationale 128 KiB → `mark_to_confirm` pushes 10,001 records, each cloning the rationale | 711 KB | 1,372,972 KB | refused afterwards ("records weigh more") |
|
||||
| same, rationale 512 KiB (about 5 GB needed) | 1.1 MB | 2,996,672 KB | `memory allocation of 524288 bytes failed`, SIGABRT |
|
||||
| level A add of 10,000 children, then `set` `description` (64 KiB value) with `scope: subtree` → 10,000 writes plus 10,000 records holding the value twice | 646 KB | 2,027,252 KB | refused afterwards |
|
||||
| add of 2,000 children, then `cite` 4,000 declared references with `scope: subtree` (references.json 150 KB) → 8 M values written before `assert_depth` | 344 KB | 2,112,620 KB | refused afterwards ("more than 200000 values") |
|
||||
|
||||
The cost grows linearly in N × size, so a changeset of a few MB asks for tens to hundreds of
|
||||
GB. The documented "tens of MB at most" is wrong, and the `mark_to_confirm` path is declared
|
||||
nowhere. Impact is availability only: a hostile or careless `changes.json` run through
|
||||
`pnpm r5:import-equine --check|--out`, or through the test suites, exhausts the memory of the
|
||||
developer machine. The OOM killer may then take down other processes, such as the Docker
|
||||
Supabase stack. No integrity or confidentiality impact. It is blocking because it breaks a
|
||||
bound that both the code and the threat model of this gate require, and because the
|
||||
documented residual misstates it.
|
||||
|
||||
Fix direction:
|
||||
- Enforce the budget at push time: route `mark_to_confirm` through the same accounting, and
|
||||
make `Recorder::record` refuse once the running weight would pass the bound.
|
||||
- Bound the per-op writes of subtree `set` and `cite` (items × value size, or a running values
|
||||
budget) before writing.
|
||||
- Alternatively, stop cloning: share `ChangeSummary` and `Op` across records (`Arc`), so a
|
||||
record weighs only its own before and after.
|
||||
- Count the cite op's `reference_ids` in `record_weight`.
|
||||
- Correct RUST.md §11.15 and add a test per path (marking, subtree set, subtree cite) that
|
||||
fails before any GB-scale allocation.
|
||||
|
||||
Probe source: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/sec8/probe/src/main.rs`
|
||||
(modes `add <rationale bytes> <children>`, `value2 <value bytes> <children>`,
|
||||
`cite <references> <children>`).
|
||||
|
||||
## Reported findings
|
||||
|
||||
MEDIUM, review drawings per view (declared in RUST.md §11.15, now measured). Each view of the
|
||||
curated document yields `review/annotations-<view>.svg` and `review/app-<view>.svg`. Each file
|
||||
holds a full copy of the cleaned drawing, and all files sit in memory before the CLI writes
|
||||
them. Level A adds of a detail view plus its trigger: 500 views → 1,011 files, 138.8 MB,
|
||||
176 MB RSS; 2,000 views → 4,011 files, 547 MB, 638 MB RSS. That is about 273 KB per view, so
|
||||
about 1.3 GB at the records bound (about 4,800 views), consistent with the declared "about
|
||||
2 GB" estimate. The multiplier is the drawing's size. The frozen `horse.svg` is 130 KB, but
|
||||
the core reader admits up to 2 MiB, so a replaced drawing would scale the same views to about
|
||||
19 GB. Not blocking because it is declared and the estimate holds. A view-count or output-bytes
|
||||
bound would close it.
|
||||
|
||||
LOW findings:
|
||||
- `record_weight` counts the op only for `set` and `add` (records.rs:125-128), but every record
|
||||
clones the op. A cite's `reference_ids` list is copied uncounted into each record, so the
|
||||
weight under-counts by about a third on that path.
|
||||
- `TriggerNames::apply` searches the view list linearly for every element carrying
|
||||
`data-anno-opens-view` (views × triggers). This runs in `--check` only, on committed files,
|
||||
in seconds at the bounds.
|
||||
- The two new test files carry the crate's usual test allow (`disallowed_methods`, `unwrap`,
|
||||
`expect`) for fixture reads. They are test crates only and do not suppress a security scanner.
|
||||
- Context, outside the diff: `clippy.toml`'s `disallowed-methods` bans `fs::read`,
|
||||
`fs::write`, `File::open|create` and `env::var`. It does not ban `OpenOptions::open`,
|
||||
`fs::create_dir`, `fs::read_dir`, `fs::remove_*` or `fs::rename`, so "the core has no I/O"
|
||||
rests on review for those. A grep of `crates/core/src/curation` finds no `std::fs`,
|
||||
`std::env`, `std::process` or clock use today.
|
||||
|
||||
## What was checked and held
|
||||
|
||||
- Markdown guard, differential against Prettier 3.8.3: `report_guard::format_report` against
|
||||
the repository's Prettier (`parser: markdown`, the repository options), on raw reports in the
|
||||
generator's three line forms: the list item of `change_line`, the 8-column CHANGELOG row via
|
||||
`cell()`, and the bibliography line. 16,395 cases (hand-picked cell-start and line constructs,
|
||||
CJK, Hangul, Arabic with harakat, Devanagari and Thai marks, URLs with `_` and `&name=`,
|
||||
`{{ }}`, wiki and strong forms, plus three seeded random sets). 4,696 were admitted. The only
|
||||
3 differences were wiki links opened in one list-item field and closed in another across code
|
||||
spans. `format_report` was called directly there; in the pipeline,
|
||||
`review_report::assert_single_line` refuses `[[`/`]]` in every list-item field first. Table
|
||||
cells parse on their own and showed no difference. No silent difference is reachable in
|
||||
this sample.
|
||||
- Structure forging: line breaks never reach a line start. List-item fields refuse control
|
||||
characters, and `cell()` maps `\n` to a space. `\\|` (an authored `\|` after escaping) is
|
||||
refused. Code spans are limited to `[A-Za-z0-9_:/. -]`, authored backticks are refused, and
|
||||
field segments are limited to `[A-Za-z0-9_-]`. Ids written by add and rename must be stable
|
||||
ids at the op.
|
||||
- SVG: hostile annotation and view names (`<a href>`, `&`, `A`, `]]>`, `<!-- -->`,
|
||||
`<?pi?>`, quotes, U+0085, U+2028, NBSP, tab, newline) and a path `d` holding
|
||||
`" onload="x` produce four well-formed drawings (xml.dom.minidom). The values stay data,
|
||||
with no `onload` attribute and no `a` or `script` element. Non-XML characters and CR are
|
||||
refused (`drawing_text`). The cleaned drawing goes through the core reader first: 2 MiB,
|
||||
20,000 elements, depth 64, DOCTYPE and ENTITY refused. `compare_drawing` parses the committed
|
||||
file with the capped reader before roxmltree and requires a byte-exact rebuild.
|
||||
- CLI file safety: `--check` passes at HEAD (legacy-import 6/0, draft 12/0, app-main declared).
|
||||
`--out` refusals were run in a scratch folder: empty path, symlink, symlink with a trailing
|
||||
`/`, non-empty folder, `..`, a destination through a symlink into the seed or into `draft/`,
|
||||
a missing parent. A fresh folder gets 18 files, and a second run refuses "not empty". Writes
|
||||
are `create_new`, folders are `create_dir` and components must be `Normal`. Reads are
|
||||
canonical, confined to the seed, regular files only, capped at 16 MiB. Same-user TOCTOU is
|
||||
declared. The working tree stayed clean.
|
||||
- Quadratic scans: the guard is linear (the `&`, `<`, `[[` and URL scans work per segment).
|
||||
`underscore_pairs_later` rescans only from the few authored plain parts. The renderers are
|
||||
hashed. No input under the caps reached a measurable guard time (max 0 ms per case).
|
||||
- Checklist: no hardcoded secret, token or auth URL in the diff or the oracles. No SQL built in
|
||||
scope; `seed.rs` is unchanged since R5.3. No HTML render, endpoint, authz surface or logging
|
||||
in scope. The CLI's error messages go to the local operator only. The only tracked env file
|
||||
is `.env.example`; reading it was refused by the permission rules (rule: Bash `cat .env.example`
|
||||
denied), it was not retried another way, and it is outside the diff. Guardrails scanner
|
||||
(`--domain api,supabase`): 0 FAIL, 1 WARN, `SB-POLICY-NOT-OWNER`
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff and dropped
|
||||
per the diff-scope rule, plus 1 MANUAL.
|
||||
- Anti-gaming: no `nosemgrep`, `nosec`, `eslint-disable` or `unsafe` added. The only new allows
|
||||
are the two test-file lint allows listed above.
|
||||
|
||||
Semgrep note: the three packs carry few Rust rules (162 rules loaded on 28 files, 0 findings),
|
||||
so semgrep alone says little about this diff. The verdict rests on the checklist and the
|
||||
probes.
|
||||
@@ -0,0 +1,88 @@
|
||||
SECURITY — VERDICT: BLOCK(1)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 29 files (`git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`, HEAD 8bdb079)
|
||||
BLOCKING:
|
||||
1. [HIGH] resource-bound / records weight undercount — crates/core/src/curation/draft/records.rs:176-177 (`parts_weight`) — `change.reference_ids` costs 1 unit per byte, but every record clones the whole `ChangeSummary`, and each 1-char id clone takes about 56 bytes (24-byte String header plus the smallest malloc chunk). The 16 Mi-unit bound therefore lets roughly 0.9 GB of cloned ids through before it trips. — hint: charge each id at 64 + len, as `op_weight` already does for `Cite`, or stop cloning the summary into every record (share it, or index the change)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] resource-bound / parse amplification (pre-existing R5.3 code, outside the diff) — crates/core/src/curation/draft/schema.rs:405-412 (`parse_changeset`: Value tree, `value.clone()`, typed value, `assert_same_shape` serialising it back), plus the Change clones in pipeline.rs:147-159
|
||||
- [MEDIUM] guardrails SB-POLICY-NOT-OWNER (outside the diff) — supabase/migrations/20261008110000_anatomical_sheets.sql:1177
|
||||
- [LOW] new lint allowance (clippy, not a security suppression; same convention as the 20 existing integration test files) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
PROOF: semgrep 162 rules on 29 files → 0 findings; checklist 8 checks → 4 findings (1 HIGH, 2 MEDIUM, 1 LOW)
|
||||
|
||||
## Detail
|
||||
|
||||
### 1. [HIGH] The records bound does not bound memory for change reference ids
|
||||
|
||||
The bound comes from R5.4 (`records::charge`, `MAX_RECORDED_WEIGHT` = 16 Mi units). It is
|
||||
documented (records.rs:36-41, RUST.md §11.14) as keeping the records, and the reports built
|
||||
from them, small. `parts_weight` adds `change.reference_ids` as the sum of their byte lengths.
|
||||
Each record clones the change's summary, so it clones the whole id vector too. Duplicate ids
|
||||
are accepted (`non_empty_strings` does no dedup), and one declared 1-char reference id is enough.
|
||||
|
||||
Measured with a scratch harness that runs `build_equine_draft` + `render_draft_artifacts` on
|
||||
the seed's legacy sources and frozen drawing (read-only), a crafted `changes.json`, and
|
||||
`references.json` plus one reference `r`. One level A change with `reference_ids` = `["r"] × N`
|
||||
and M `set` ops on one finding (`/usr/bin/time -f %M`):
|
||||
|
||||
| changes.json | N | M | outcome | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| equine (committed) | — | — | 11 files written | 17.8 MB |
|
||||
| 1.0 MB | 200,000 | 100 | refused at op 83 by the records bound | 948 MB |
|
||||
| 2.0 MB | 400,000 | 60 | refused at op 41 by the records bound | 971 MB |
|
||||
| 8.0 MB | 2,000,000 | 8 | draft built, refused at the table bound | 1.30 GB |
|
||||
| 16.4 MB (under the 16 MiB cap) | 4,100,000 | 4 | draft built, refused at the table bound | 1.67 GB |
|
||||
|
||||
The bound trips only after about 0.9 GB is already allocated: a 1 MB input reaches 948 MB.
|
||||
That breaks the brief's criterion (peak memory well below the GB range for any input under
|
||||
the 16 MiB caps), and it breaks the R5.4 claim that the records bound keeps the reports'
|
||||
inputs small. Other record parts are charged closely enough: JSON values cost 64 units for
|
||||
roughly 32 to 110 bytes, string bytes cost 1 unit for 1 byte.
|
||||
|
||||
Fix direction: charge 64 + len per change reference id, or keep a single copy of each
|
||||
change's summary and stop cloning it into every record. Add a test with a duplicated
|
||||
reference-id vector next to `the_records_bound_holds_inside_one_op_and_the_marking`.
|
||||
|
||||
### 2. [MEDIUM] Parse-phase amplification (outside the diff)
|
||||
|
||||
`parse_changeset` on the 16.4 MB / 4.1 M-id file peaks at 1.075 GB by itself (scratch
|
||||
`parseonly` binary). That is about 65 times the input: the parsed Value tree, a clone of it
|
||||
passed to `from_value`, the typed value, and `assert_same_shape` serialising the typed value
|
||||
back. After parsing, `build_equine_draft` clones the changes again, `apply_change` builds a
|
||||
summary, and `DraftResult.changes` holds more summaries, each about 230 MB for that input.
|
||||
This code is R5.3 and does not appear in c6e0d85..HEAD, so it is reported, not blocking.
|
||||
It falls under the same "any input under the 16 MiB caps" criterion and is worth handling
|
||||
in the same fix (avoid the clone before `from_value`; compare shapes without re-serialising
|
||||
everything).
|
||||
|
||||
### Checked with no finding
|
||||
|
||||
- `--out` / `--check` file safety (import_equine.rs), by reading: refuses an empty path or
|
||||
`..`; the parent is canonicalised; a destination inside the seed is refused by prefix and by
|
||||
(dev, inode) of every seed folder; a symlinked or non-empty destination is refused; writes
|
||||
use `create_dir` + `create_new`, never overwrite, never delete; generated names must be
|
||||
plain `Normal` components (an absolute path or `..` is refused); reads are canonicalised
|
||||
into the seed, regular files only, 16 MiB cap; `--check` never lists through a link. The
|
||||
TOCTOU on a single-user machine is the declared residual (RUST.md §11.13).
|
||||
- Markdown structure and parity: the scratch differential fuzz of `format_report` against
|
||||
Prettier 3.8.3 (repo options) ran 33,000 random reports built from admitted and
|
||||
construct-prone tokens in list items, bibliography lines and table cells. 8,146 were
|
||||
admitted, 0 differed. Hand-written constructs (`[a] (b)`, `{{ }}`, `[[a]]`, `<`, `|`,
|
||||
`\"`, `\|`, setext and list markers mid-line) gave no difference. Every authored field is
|
||||
mid-line behind a generated prefix; line breaks and controls are refused in list items and
|
||||
turned into spaces in cells as the TypeScript did; code spans only carry stable ids,
|
||||
targets and field segments limited to `[A-Za-z0-9_-]`.
|
||||
- Table alignment: every Alphabetic or Numeric code point from U+00A0 to U+3FFFF in a cell
|
||||
(149,044) is aligned as Prettier aligns it, or refused (emoji candidates, `UnsupportedWidth`).
|
||||
- SVG well-formedness: attributes and text are escaped (`& < > "`, `\t \n \r` as references),
|
||||
XML-invalid code points and CR are refused before writing (`drawing::first_unwritable`),
|
||||
element names come from the reader's allowlist or are constants, and ids are stable ids.
|
||||
- Existing suites: `cargo test -p faunosteo-core --features curation --release --test
|
||||
curation_review --test curation_markdown --test curation_draft` → 30 passed.
|
||||
- Secrets, SQL concatenation, eval/innerHTML, sensitive logging, tracked `.env`: none in the
|
||||
diff (the only `println!` added prints a generated file name; `.env.example` is the
|
||||
placeholder file).
|
||||
|
||||
### Not covered by this run
|
||||
|
||||
I did not finish the bound analysis of the review drawings' markers and labels
|
||||
(`overlay.rs`, `app_review.rs` through the R3 layer). Nothing is concluded about them here.
|
||||
@@ -0,0 +1,231 @@
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 25 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] check/declared-tree-rule-too-wide — crates/core/src/curation/draft/svg_compare.rs:104
|
||||
- [LOW] check/unchecked-xml-characters — crates/core/src/curation/draft/app_review.rs:33
|
||||
- [LOW] check/markdown-structure-injection — crates/core/src/curation/draft/review_report.rs:64
|
||||
- [LOW] check/quadratic-renderers — crates/core/src/curation/draft/review_report.rs:223
|
||||
- [LOW] anti-gaming/test-clippy-allow (context) — crates/core/tests/curation_review.rs:6
|
||||
PROOF: semgrep 162 rules on 25 files → 0 findings; checklist 8 checks + anti-gaming + 5 brief probes → 5 findings
|
||||
```
|
||||
|
||||
# R5.4 security gate (fresh), 2026-10-23
|
||||
|
||||
Repository `/home/bchanot/Documents/faunosteo`, branch `feature/rust-r5-pipeline-core`, HEAD
|
||||
`0a79c52`, scope `git diff c6e0d85..HEAD` (25 existing files). Every changed product file was
|
||||
read whole: `crates/core/src/curation/draft/{labels,changelog,review_report,drawing,overlay,
|
||||
app_review,svg_compare,pipeline,stats,ops,mod}.rs`, `crates/core/src/bin/import_equine.rs`,
|
||||
plus `crates/core/src/curation/markdown.rs`, `anatomy/svg/{serialize,tree}.rs` and
|
||||
`anatomy/compose.rs` where the new code relies on them.
|
||||
|
||||
## Method
|
||||
|
||||
- semgrep 1.168.0 (`EIO_BACKEND=posix`), `--config p/security-audit --config p/secrets
|
||||
--config p/owasp-top-ten --metrics=off --quiet --json` on the 25 files: 162 rules, 0
|
||||
results, 0 errors. These packs cover Rust thinly, so the manual review carries the weight.
|
||||
- Checklist (CLAUDE.md defaults): secrets, SQL concatenation, unsanitized render, authz,
|
||||
error exposure, secrets in logs, tracked `.env`, guardrails scanner
|
||||
(`python3 -B ~/.claude/skills/guardrails/scripts/scan.py <root> --domain api,supabase`:
|
||||
0 FAIL, 1 WARN, 1 MANUAL; the WARN `SB-POLICY-NOT-OWNER` is in
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the delta, so it
|
||||
is not reported here).
|
||||
- Anti-gaming: the diff was scanned for new suppressions.
|
||||
- Probes, all in the scratchpad: a throwaway crate linking `faunosteo-core` (feature
|
||||
`curation`) at
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/probe.DaWdIB`,
|
||||
and `import-equine` built with `CARGO_TARGET_DIR` in the scratchpad. `--check` ran
|
||||
read-only on the repository. `--out` ran only inside the fresh
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/out.6SamF5`.
|
||||
Nothing was deleted, and `git status` stayed clean.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. [MEDIUM] The tree fallback hides more than the declared trigger-name rule
|
||||
|
||||
`crates/core/src/curation/draft/svg_compare.rs:104-123` (`compare_drawing`), used by
|
||||
`crates/core/src/bin/import_equine.rs:196-215` for `review/app-*.svg`.
|
||||
|
||||
When the bytes differ, both files are parsed with `parse_svg`. That function builds the tree
|
||||
of the root element only, and it keeps only elements and text
|
||||
(`anatomy/svg/tree.rs:155, 182-193`). Everything the tree does not carry is therefore never
|
||||
compared:
|
||||
|
||||
- the prolog: XML declaration and its `encoding`, processing instructions, comments;
|
||||
- comments and processing instructions inside the root and after it;
|
||||
- entity and character-reference forms, CDATA against escaped text, attribute quoting,
|
||||
`<g/>` against `<g></g>`;
|
||||
- attribute-value whitespace that the parser normalises.
|
||||
|
||||
Measured on the real committed `draft/review/app-main.svg` against the produced one, with
|
||||
the rule applied:
|
||||
|
||||
| Committed file changed by | `compare_drawing` |
|
||||
|---|---|
|
||||
| nothing | `Ok(TreeAfterRule)` |
|
||||
| `<?xml-stylesheet type="text/css" href="https://attacker.invalid/hide.css"?>` and a comment in the prolog, a comment and a PI right after `<svg …>`, a trailing comment | `Ok(TreeAfterRule)` |
|
||||
| also `encoding="UTF-8"` replaced by `encoding="ISO-8859-1"` | `Ok(TreeAfterRule)` |
|
||||
|
||||
In all three cases `--check` prints the same `declared: review/app-main.svg (trees equal
|
||||
under the R3 trigger-name rule)` line it prints today and exits 0. The same holds for
|
||||
`app-head.svg`, which is byte-identical today: a comment-only change would turn it into a
|
||||
"declared" file. The only visible sign would be the extra line.
|
||||
|
||||
Impact: the owner requirement "`--check` fails on any unauthorised difference" is not met
|
||||
for the two application drawings. A stylesheet processing instruction is honoured by
|
||||
browsers that open a standalone SVG: it could restyle or hide markers in the review drawing
|
||||
and fetch an external URL, and `--check` would still pass. Exploiting it needs write access
|
||||
to the repository. The files are dev review artifacts: the app, the bundle and the uploads
|
||||
never read them (grep of `apps/app`, `vite`, `packages/supabase`). Hence MEDIUM, not
|
||||
blocking.
|
||||
|
||||
Three declarations are materially wrong:
|
||||
- `svg_compare.rs:7` says "Any other difference is reported".
|
||||
- `import_equine.rs:185-186` and the printed "declared" line suggest the rule alone explains
|
||||
the difference.
|
||||
- RUST.md §11.15 says the comparison "accepts exactly the trigger-name rule and nothing
|
||||
else". §11.13's R5.4 condition did mention "declaration, prefixes and whitespace
|
||||
declared", but §11.15 does not carry that through, and it never covered processing
|
||||
instructions or comments.
|
||||
|
||||
Fix direction: once the rule has been applied to the committed tree, require the committed
|
||||
prolog to be byte-equal to the produced one. Then require
|
||||
`prolog + serialize(committed tree after rule)` to equal the produced text byte for byte.
|
||||
The core serializer writes the produced file, and `app-head.svg` already matches it byte for
|
||||
byte. At minimum: compare the prologs and the text after the root, refuse any comment or PI
|
||||
node in the committed document, and print "declared" only when the rule actually changed
|
||||
the committed tree. Then correct the three declarations.
|
||||
|
||||
### 2. [LOW] XML-invalid characters from authored strings reach the review drawings unchecked
|
||||
|
||||
`crates/core/src/curation/draft/app_review.rs:33-36` calls `render_layer` directly. The R3
|
||||
composition passes the layer through `checked` / `xml_safe` (`anatomy/compose.rs:221-236`,
|
||||
"refuse one XML cannot carry"), and this path skips that step. The same gap exists in
|
||||
`overlay.rs:194-198` (path `d`, ids in titles and labels).
|
||||
|
||||
Measured: a changeset that sets `annotation:hock` `name` to `{"fr":"Jarret\u0001<b>&","en":"Hock\u000b"}`
|
||||
is accepted, and the produced `review/app-main.svg` is not well-formed (roxmltree: "a
|
||||
non-XML character '\u{1}'"). A path geometry `d` holding U+0001 in an overlay gives the same
|
||||
result. This is not markup injection: the serializer escaping holds (see "Verified"). It is
|
||||
a malformed output written by `--out` without any refusal. `--check` then fails, closed.
|
||||
|
||||
Fix direction: run `is_xml_text` over the produced layer and overlay (reuse compose's
|
||||
`checked`) and refuse. This changes nothing on the committed data.
|
||||
|
||||
### 3. [LOW] Markdown structure injection from authored text
|
||||
|
||||
- `review_report.rs:64-85` (`change_line`: `title.fr`, `rationale`, `question` written raw
|
||||
in list items);
|
||||
- `review_report.rs:295-320` (questions);
|
||||
- `review_report.rs:245-278` (`citation`, `url`, `doi`);
|
||||
- `changelog.rs:80, 85` (table cells through `markdown.rs:43-49` `cell()`). `cell()` escapes
|
||||
`|` and `\n`, but not `\r`, and not a backslash in front of a pipe.
|
||||
|
||||
Measured with an added change:
|
||||
- a rationale `"Ligne un.\n\n## 10. Décisions réservées au professionnel\n\nAucune question ouverte. <img …>"`
|
||||
forges a second section 10 inside section 6 of `REVIEW-REPORT.md` (lines 92 and 343) and
|
||||
passes raw HTML through;
|
||||
- a `\r` in a title reaches `CHANGELOG.md` raw. GFM treats a CR as a line ending, so the
|
||||
table row breaks;
|
||||
- an authored `\|` becomes `\\|`. `format_markdown` and micromark both read that as an
|
||||
escaped backslash followed by a real delimiter, so the row gains a ninth column under an
|
||||
eight-column header.
|
||||
|
||||
Context: the text comes from the repository owner, and the code matches the TypeScript
|
||||
(`markdown.ts`). The current data holds none of this: no LF, CR, pipe or backtick in any
|
||||
title, rationale or question. The one `<0697:toih>` in a DOI and its URL renders as text,
|
||||
because a scheme cannot start with a digit. Gitea strips raw HTML. The code spans hold only
|
||||
constrained strings: `CHG-\d{3,}`, stable ids, resolved targets and allowlisted fields.
|
||||
§11.11's "Markdown risks" are about Prettier differences, so this risk is not declared
|
||||
anywhere.
|
||||
|
||||
Fix direction: refuse CR, LF and control characters in single-line authored fields at
|
||||
`parse_changeset` and `parse_references` (title, question, citation, doi). Either refuse
|
||||
them in the rationale too, or render newlines as spaces in list items. Escape `\` in front of
|
||||
`|` in `cell()`. Optionally escape `<`.
|
||||
|
||||
### 4. [LOW] Quadratic renderers on authored volume
|
||||
|
||||
- `review_report.rs:223-234` `citation_count` rebuilds `all_findings(document)`, one
|
||||
allocated `FindingEntry` per finding, for every reference: O(R × F).
|
||||
- `overlay.rs:87-143`: linear `find` over `previous` and `kept.contains`, O(A²) per view.
|
||||
- `review_report.rs:47-54` `subjects`: changes × records.
|
||||
|
||||
Measured with `render_review_report` on synthetic documents:
|
||||
|
||||
| References | Leaves | Time |
|
||||
|---|---|---|
|
||||
| 1,000 | 1,000 | 0.35 s |
|
||||
| 4,000 | 4,000 | 5.9 s |
|
||||
| 8,000 | 8,000 | 22.6 s |
|
||||
|
||||
Extrapolated, not measured: about 3 to 4 minutes for a balanced document at the 200,000-value
|
||||
bound. The equine data (48 references, about 600 findings) renders in milliseconds:
|
||||
`--check` takes 0.13 s wall time and 18 MB RSS. This is the same class as the declared
|
||||
"minutes for a 16 MiB changeset" risk (§11.13), but these renderers are not declared.
|
||||
|
||||
Fix direction: count citations in one pass into a `HashMap<&str, usize>`, index `previous`
|
||||
and `kept` with a `HashMap` / `HashSet`, and group records by change id once.
|
||||
|
||||
### 5. [LOW, context] New clippy allow in a test crate
|
||||
|
||||
`crates/core/tests/curation_review.rs:6`
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. This is an
|
||||
integration test, outside the library, with the same header as all 33 files of
|
||||
`crates/core/tests`. It is a lint allow, not a security-scanner suppression. No new
|
||||
`nosemgrep`, `nosec`, `eslint-disable`, `ts-ignore` or `noqa` appears in the diff. Not
|
||||
blocking.
|
||||
|
||||
## Verified, no finding
|
||||
|
||||
- SVG markup injection: element and attribute names are literals. Values go through the
|
||||
core serializer, which escapes `& < > "` and `\t \n \r` in attributes and `& < > \r` in
|
||||
text. Probe: an annotation name `J"><script>alert(1)</script>&`, an English name
|
||||
`]]><!--x-->`, a view name `T</title><script>y</script>` and an overlay path
|
||||
`d="M0 0…\"/><script>x</script>"`. The output is well-formed (apart from finding 2's
|
||||
control character), with 0 `script` elements and every payload escaped.
|
||||
- File names built from view ids: views are `stable_id` (`[a-z0-9][a-z0-9_-]{0,63}`),
|
||||
enforced by the typed parse in `pipeline.rs:206` before any write. `write_exclusive`
|
||||
also refuses any component that is not `Normal`. The `--check` reads stay canonical
|
||||
inside the seed.
|
||||
- Panics on authored input: `compact_value` slices only past 120 UTF-16 units and decodes
|
||||
lossily. The prolog slice ends at roxmltree's root start (`<`, a char boundary).
|
||||
`style()[0]` is a static slice. Polygon centroid: at least 3 points (typed), NaN otherwise,
|
||||
never a panic. No `unwrap`, `expect` or unchecked index on data in the product files.
|
||||
Recursion in `first_difference` and `TriggerNames::apply` is bounded by the reader's
|
||||
depth cap (64).
|
||||
- `--check` is read-only: no file in `legacy-import/` or `draft/` changed and `git status`
|
||||
stayed clean. Output: `legacy-import: 6 files, 0 differences`, `draft: declared:
|
||||
review/app-main.svg …`, `draft: 12 files, 0 differences`. A file that is missing, not
|
||||
produced or not UTF-8 fails. A generated folder must canonicalise to the seed's child.
|
||||
Files other than `review/app-*` have no rule and must be byte-equal. `horse.svg` is
|
||||
compared with itself, but its sha256 drives `assets.json`, `document.json` and the seed
|
||||
SQL, so a change to it fails the check.
|
||||
- `--out`, probed in the mktemp folder:
|
||||
- refused, nothing written: a non-empty folder, a symbolic link (with and without a
|
||||
trailing `/`), a file, `x/../fresh`, a missing parent, an empty path;
|
||||
- written: a fresh relative path through `INIT_CWD` (18 files), and a fresh folder under a
|
||||
parent link inside the scratchpad (the parent is resolved as documented);
|
||||
- refused: a second run on the same destination ("not empty");
|
||||
- `diff -r` against the seed: `legacy-import` identical, `draft` identical except
|
||||
`review/app-main.svg`, the declared trigger names.
|
||||
- R5.3 carry-overs: a seed walk that fails is now an error, and the claim refuses ("the
|
||||
seed folder cannot be read"). An existing destination folder is identity-checked
|
||||
(device, inode) against every folder of the seed. Checked by reading only: a second
|
||||
mount path needs a bind mount (root), and a symlink to the repository was not used,
|
||||
because the brief forbids `--out` into the repository.
|
||||
- `ops.rs` cite: the `HashSet` keeps `before.includes(id)` semantics (string equality only).
|
||||
- No secret, token, auth URL or private key in the diff (the `.claude` audit texts only cite
|
||||
the semgrep command). No SQL is built in the diff. stdout and stderr carry paths, counts
|
||||
and refusals only. The only tracked env file is `.env.example`.
|
||||
|
||||
## Residual risks already declared (RUST.md §11.13, §11.15)
|
||||
|
||||
Checked and accurate, apart from the §11.15 sentence corrected by finding 1:
|
||||
- the emoji refusal of the alignment;
|
||||
- the single-user TOCTOU of `--out`, mitigated by `create_new`;
|
||||
- records memory and the bound-walk time;
|
||||
- the frozen `horse.svg`.
|
||||
@@ -0,0 +1,101 @@
|
||||
# Verification: Rust R5.4 review files (2026-10-23)
|
||||
|
||||
Contract `.claude/tasks/contracts/2026-10-23-rust-r5-review.md`. Fresh verifier, report-only (persisted by the orchestrator).
|
||||
|
||||
## Round 1 on 0a79c52: verifier ECARTS(2), security PASS (1 MEDIUM, 4 LOW)
|
||||
|
||||
- Verifier: C1, C4 to C9 MET (30 tests, `--check` 0/12 + 1 declared, `--out` ×2 identical, anatomy 143, floor clean; 140 targeted + 10,500 random changesets). C2 NOT MET: `format_markdown` reproduces only Prettier's table alignment; authored Markdown syntax or whitespace (`*`, `_`, double spaces, tabs, line breaks in list items, stray backticks, `\|`) is rewritten by Prettier and not by the port (931 accepted / 909 differing in formatted Markdown on a special-character alphabet, 0 on a safe alphabet); a row with an extra cell lost a cell silently. C3 NOT MET: a CR in a label gives an undeclared tree difference (xmldom raw CR vs core ` `). Notes: emoji refusal set wider than emoji; tree fallback ignores prolog/comments/PIs; stale test comment; quadratic renderers.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security.md`): MEDIUM tree fallback too wide (stylesheet PI, comments, encoding change accepted as "declared"); LOW XML-invalid characters reach the drawings unchecked; LOW Markdown structure injection (rationale forging a section, CR, `\|`); LOW quadratic renderers; LOW test allow (context).
|
||||
- Fixed: Markdown guard (`UnsupportedMarkdown`) built from a TypeScript sweep `oracles/r5/markdown-sweep.json` (220 probes: 187 identical, 33 refused, 0 silent; equine-safe probes never refused; `**strong**` spans kept as the import report needs), single-line check of list-item fields, table cell-count check; strict drawing comparison (committed file must rebuild from its tree, rule must change it, produced equals rebuilt-after-rule byte for byte; attack test); drawings refuse XML-invalid characters and CR; renderers indexed; stale comment; docs §11.14-11.15 (guard rules, declared table, emoji breadth, residual risks).
|
||||
|
||||
## Round 2
|
||||
|
||||
Fresh verifier and security on the fixed commit: below.
|
||||
|
||||
## Round 2 on 68dec25: verifier ECARTS(2), security PASS (3 MEDIUM, 2 LOW)
|
||||
|
||||
- Verifier: C1, C3 to C6, C8, C9 MET. C2: silent Prettier rewrites the guard missed (inline link destinations and titles, autolinks with an entity, `[[ a ]]`, character references decoding to `*`/`_`), shown end to end on the equine pipeline with one appended change; §11.15 sentence on links wrong. C7: `markdown-sweep.json` not excepted by the contract. Differential: 2,430 targeted + 4,500 random cases, 0 raw-text or drawing differences.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-2.md`): MEDIUM ids with line breaks (added then removed, renamed, nested, field segments) forging report sections through code spans, raw CR in code spans; MEDIUM uncapped roxmltree parse of the committed drawing before the prescan (stack overflow at depth 100,000); MEDIUM `中_中` / `[a]( x )` silent and §11.14-11.15 overstated; LOW Infinity printed `null`; LOW test allows (context).
|
||||
- Fixed: construct rules (`](`, `][`, padded `[[ ]]`, autolinks, character references) from new sweep probes; underscore rule ASCII-only (the sweep showed `é_é` rewritten too); CR/TAB refused anywhere in a line; ids validated at the op (add, nested, rename) and field segments without control characters or backticks; core reader before roxmltree in the comparison; `Infinity`; contract clarification for the second oracle; docs (counts 247/187/60, declared rows, §11.15). Tests: forged ids, field segments, 100,000-level drawing.
|
||||
|
||||
## Round 3
|
||||
|
||||
Fresh verifier and security on the fixed commit: below.
|
||||
|
||||
## Round 3 on 077d91b: verifier ECARTS(2), security BLOCK(2)
|
||||
|
||||
- Verifier: C1, C3 to C8 MET; C2: 57 synthetic + 5 equine end-to-end silent cases (37,510 + 20 probed): touching strong pairs `**a****b**`, bare URLs next to strong spans; C9: §11.14/11.15 claims contradicted, line and test counts, NUL exemption.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-3.md`) BLOCK(2): HIGH strong-span model (whitespace-only, backslash before a delimiter, touching pairs, underscore next to a delimiter), HIGH underscore glued to a bare URL; LOW NUL mid-segment, LOW trailing space latent, LOW test allows.
|
||||
- Root change: the denylist leaked every round → allowlist guard `curation/report_guard.rs` (`format_report`, the draft's two reports only; the import report keeps `format_markdown`): letters, digits, single spaces, a fixed punctuation set, ASCII intraword `_` not glued to a URL, `\"` and `\|`, `**word**` between spaces, no construct; everything else refused. Sweep regrown: 762 probes incl. the new families, a label alone (cell start) and 400 seeded random strings, digests only: 304 identical, 458 refused, 0 silent. NUL refused anywhere in a field segment; misplaced comment fixed; docs rewritten as an empirical guarantee.
|
||||
|
||||
## Round 4
|
||||
|
||||
Fresh verifier and security on the fixed commit: below.
|
||||
|
||||
## Round 4 on 6d4fdd6: security BLOCK(3) (verifier pending at fix time)
|
||||
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-4.md`): HIGH non-terminated / long character references (`* b` → `\*`, `< x &` → `< x <`), HIGH image references `![…]` (alt reprinted decoded), HIGH nested padded wiki links (`[[ a [[b]]`); MEDIUM §11.14/§11.15 overstated ("unknown form refused by default", "refused whole", "found none"); LOW quadratic underscore check. Control: 3,000 random strings without `&`, `![`, `[[`: 0 silent.
|
||||
- Fixed: `&` admitted only before a space or as a URL query separator `&name=` (name not a legacy-reference prefix, the 106 HTML legacy names); `![` refused; padded `[[`/`]]` refused when the matching half exists; URL checks without allocation; sweep 782 probes (new families, random alphabet with `&#`, `&`, `<`, `![`, `[[`, `]]`): 279 identical, 503 refused, 0 silent; docs restated exactly.
|
||||
- Verifier round 4 (on 6d4fdd6, arrived after the security fixes): C2 cross-code-span families: a plain `_` paired by remark with a closing `_` in a later code span (`to_do … \`ref_\``, `a_b \`c_\``; reachable without authored backticks through a reference id `ref_`), a link bracket and a bare URL running into an authored code span; C9 docs (code spans "generated", "refused by default"); floor `.skip(1)` hit in a test. 555,646 Prettier context docs: all pairs and triples of admitted characters 0 silent; 1,080 silent with authored code spans.
|
||||
- Fixed: authored backticks refused in every authored field (`report_guard::assert_authored`, CHANGELOG cells and review list items); a plain `_` followed by a closing `_` in a later code span refused (the equine `{"min_x":…,"min_y":…}` stays admitted); sweep: reference `probe_ref`, three probes citing `probe_ref_`, five cross-span probes: 790 probes, 279 identical, 511 refused, 0 silent; floor hit rewritten (`match_indices`); CLAUDE.md folder line; docs and module header restated.
|
||||
|
||||
## Round 5
|
||||
|
||||
Fresh verifier and security on the fixed commit: below.
|
||||
|
||||
## Round 5 on 1b12d90: verifier ECARTS(2), security PASS (2 MEDIUM, 3 LOW)
|
||||
|
||||
- Verifier: C1, C3 to C8 MET. C2: 59 silent of 306,192 probes, all one class: a wiki link opened in one list-item field and closed in another of the same line (code spans between). C9: draft line count (4,376, gain 1,468), report_guard.rs line count, wiki rule wording.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-5.md`): MEDIUM F2 the same wiki link across fields; MEDIUM F1 an authored field segment (`name.x]`) inside a generated code span closing an author's `[`; LOW docs "code spans are the generator's alone" wrong (segments are authored); LOW quadratic `in_url_token` and padded-wiki scans; LOW test allows (context).
|
||||
- Fixed: field segments limited to `[A-Za-z0-9_-]` at the op (declared refusal; TypeScript accepted any); code spans admit only the generator alphabet `[A-Za-z0-9_:/. -]`; list-item fields refuse `[[`, `]]`, `{%`, `%}` whole; plain parts refuse `{%`/`%}`; URL-token membership in one pass, padded wiki through one find/rfind; sweep 798 probes (eight cross-field construct pairs): 268 identical, 530 refused, 0 silent; segment refusal test with its reason; docs counts (draft 4,385, gain 1,477, report_guard.rs 258) and rules restated.
|
||||
|
||||
## Round 6 on aeb5050: verifier C1 to C8 MET (C9 unverifiable: gates run in parallel; 8 FLOOR lines = keywords quoted in audit prose), security PASS (1 MEDIUM, 6 LOW); `pnpm check:all` green 14:02:00
|
||||
|
||||
- Verifier: 0 silent over 100,000 guard probes × 3 contexts, 45,000 cross-field pairs (28 guard-level, all stopped first by `assert_single_line`), 32,000 end-to-end changesets TS vs Rust, 148,921 letter/digit code points in a cell. Observations: stale `draft/mod.rs` header; RUST.md: `format_markdown` gained the cell-count refusal (import report too), "combining marks" wording (alphabetic marks pass, aligned alike); declared emoji case skipped its review report and drawings; TODO R5.4a cites `format_markdown`.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-6.md`): MEDIUM CHANGELOG table amplification (rows × widest cell: +155 KB changeset → 1.09 GB, 2.2 GB RSS; TypeScript alike); LOW per-view drawings bounded only by the value bound (≈2 GB estimate); LOW `underscore_pairs_later` O(parts × length) in isolation, unreachable (≤2 authored parts per line); LOW invisible Hangul fillers admitted (Prettier keeps them); LOW unpadded `[[x]]` in a cell by design; LOW test allows (convention).
|
||||
- Fixed: aligned table bounded at 32 Mi display columns before any row is built (`Guard`, test `an_aligned_table_past_the_bound_is_refused_before_it_is_built`; equine widest ≈0.5 MB), declared in §11.14; residual risks (drawings bound, Hangul fillers) in §11.15; `draft/mod.rs` header; RUST.md wording (format_markdown shared refusals, alphabetic marks); declared emoji case now holds its review report (same text or declared refusal) and drawings; TODO R5.4a.
|
||||
|
||||
## Round 7 on 777c800: security BLOCK(2) (verifier pending at fix time); `pnpm check:all` green 14:43:14
|
||||
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-7.md`): HIGH output amplification: the CHANGELOG raw table (rows × authored rationale) built whole before the round-6 bound (1.9 MB changes.json aborts at 3 GB), RUST.md "refused before any row is built" false; HIGH record amplification: rename/remove records deep-copy the whole item and clone the change texts, no total bound (400 renames of a 5,000-child item → 4.3 GB). LOW test allows, bin allow (context), error echo size, subfolder race (declared out of model).
|
||||
- Fixed: records bounded in total (`apply::MAX_RECORDED_WEIGHT` 16 Mi units, `records::record_weight`, checked after every op; equine 1,084,422), test `the_records_cannot_outgrow_their_bound`; CHANGELOG tables projected from their cells before the raw table is joined (`markdown::assert_table_bound`), pipeline-level test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`; RUST.md rows rewritten exactly, residual risk (one op's copies before the check).
|
||||
|
||||
## Round 8 on 5880dfa: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:02:04
|
||||
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-8.md`): HIGH records bound checked only after a whole op and after the B/C marking: one op records once per item, each record cloning the change texts and the op value (B add of 10,000 children + 128 KiB rationale → 1.37 GB; subtree set 64 KiB → 2.0 GB; subtree cite 4,000 refs → 2.1 GB); RUST.md "tens of MB at most" false. MEDIUM review drawings per view (≈273 KB each view, 2,000 views → 547 MB). LOW `record_weight` ignored the op for cite/rename/remove/move; LOW `TriggerNames::apply` linear view search; LOW test allows, bin allow (context).
|
||||
- Fixed: the records weight charged at each record before the change and the op are cloned (`records::charge`, `RecordParts`; `Recorder::record` and `mark_to_confirm` return the refusal; op weight for every kind; equine 1,195,000), test `the_records_bound_holds_inside_one_op_and_the_marking`; document bound test from a padded baseline (the records bound now stops a growing changeset first); review drawings projected and bounded at 64 MiB before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`), test `review_drawings_past_their_bound_are_refused_before_rendering`; RUST.md rows and residual risks rewritten exactly.
|
||||
|
||||
## Round 9 on 8bdb079: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:24:59
|
||||
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-9.md`): HIGH records weight undercount: the change's reference ids at one unit per byte while each cloned id costs ≈56 bytes (200,000 one-character ids × 100 set ops: 948 MB before the bound; 16.4 MB file: 1.67 GB). MEDIUM `parse_changeset` ≈65× the input (1.075 GB at 16.4 MB; R5.3, outside the diff); MEDIUM guardrails WARN in a migration outside the diff; LOW test allows. Not covered: markers and labels bound of the drawings.
|
||||
- Fixed: every cloned string (texts, ids, op strings, object keys) weighs 64 + its bytes (`records::text_weight`; equine 1,965,944, bound 8.5× above); ids case added to `the_records_bound_holds_inside_one_op_and_the_marking`; parse amplification declared in §11.15 (owner decision on a lower input cap); RUST.md row updated.
|
||||
|
||||
## Round 10 on 29eb499: verifier C1 to C8 MET, C9 NOT-MET (stale counts) + 10 FLOOR (audit prose), security BLOCK(1); `pnpm check:all` green 15:38:01
|
||||
|
||||
- Verifier: 47,470 TS-vs-Rust Markdown cases 0 silent (the unguarded aligner gave 34 on the same inputs), 117 SVG label cases bytes or declared rule. C9: RUST.md counts stale (review tests 9 → 10, suites, draft lines 4,385 → 4,532) and the `report_guard.rs` header ("anything but CR or tab" inside code spans). Info: "`<` before a letter" means an ASCII letter; number cells 24 bytes where a JS number prints 25.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-10.md`): HIGH the drawings bound counted the cleaned drawing only: an unnamed trigger copies the opened view's names four times (485 KB changeset → 131 MB `app-main.svg`; 1.2 MB → abort at 3 GB). LOW tables bound per table (three level tables ≈97 MB together); LOW stale `report_guard.rs` header; LOW add/move position anchor not charged; LOW test allows.
|
||||
- Fixed: drawings projected as an upper bound per annotation (`pipeline::review_drawings_bytes`: 1,024 + 24 × (JSON text + longest view name) per annotation of the view, overlay's two documents and the app drawing; equine 1,390,428), test `unnamed_triggers_repeating_a_long_view_name_are_refused_before_rendering`; CHANGELOG tables bound summed over the three level tables (`assert_table_bound` with a running total), number cells 25; position anchor charged (equine records 1,966,163); `report_guard.rs` header; RUST.md counts (draft 4,600 lines, tests 11/2/18/5/3/7), ASCII letter wording, rows rewritten.
|
||||
|
||||
## Round 11 on 411e105: verifier C1 to C8 MET, C9 NOT-MET (2 gaps) + 10 FLOOR (audit prose), security PASS (2 LOW); `pnpm check:all` green 16:02:45
|
||||
|
||||
- Verifier: own differential 64,000 changesets TS + Prettier vs Rust, 0 silent, 0 raw mismatch; bounds far below their limits on equine and synthetic oracles; adversarial projections held except one. C9: (a) draft line count 4,600 → 4,601; (b) `review_drawings_bytes` 477 bytes under what is written for a change removing every annotation (fixed parts of each drawing, layer group, overlay group and legend, never charged). Obs: test comment cites §11.12 for the emoji refusal (§11.10/§11.14).
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-11.md`): PASS. 146,000 random cases vs Prettier, 44,330 admitted, 0 differ; drawings wrote 0.12 to 0.41 of the projection on hostile documents. LOW test allows (convention); LOW bare URLs autolinked in any authored field (visible text = target, TypeScript alike: owner choice).
|
||||
- Fixed: 4,096 bytes per drawing for its fixed parts (`pipeline::DRAWING_FIXED_BYTES`; equine projection 1,406,812 for 564,239 written), test `the_drawings_projection_bounds_what_is_written` (equine and no annotations); RUST.md counts (4,606 lines, review 12 tests); test comment sections.
|
||||
|
||||
## Round 12 on c4e717d: verifier C1 to C8 MET, C9 NOT-MET (1 doc gap) + 12 FLOOR (audit prose, reported apart); `pnpm check:all` green 16:41:34
|
||||
|
||||
- Verifier: own differential 37,264 cases (32,000 random biased to the admitted alphabet, 5,264 handcrafted placements), 0 silent, 0 raw difference; drawings projection ≥ written on 8 adversarial drafts (tightest margin 15,477 B, no annotations); every figure of §11.14/§11.15 checked exact. C9: the emoji refusal is named at the aligner (`UnsupportedWidth`) in RUST.md (synthetic case line, pictograph row) and the TS generator comment, while the report guard refuses first (`UnsupportedMarkdown`); `UnsupportedWidth` applies in the draft reports only to alphanumeric candidates (Ⓜ, 🅰) and in IMPORT-REPORT.
|
||||
- Fixed: the two RUST.md passages and the generator comment (oracle unchanged, generator green in check mode).
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-12.md`): PASS. 7,200 random changesets through both real pipelines (898 admitted, 0 raw and 0 formatted differences), 73,000 template cases (38,378 admitted, 0 differ), display width on every code point U+0020 to U+10FFFF, guard linear on stress lines. LOW `underscore_pairs_later` quadratic, unreachable (≤2 authored plain parts per line); LOW `review_drawings_bytes` views × annotations (≈2 s at the value bound); LOW test allows (convention); LOW §11.15 parse residual names `changes.json` only (`references.json` same parser).
|
||||
- Declared: the four LOWs in §11.15 (no code change: a fix would restart the gates; owner decision).
|
||||
|
||||
## Round 13 on 88e4274: verifier C1 to C8 MET, C9 NOT-MET (doc wording) + 13 FLOOR (audit prose, product paths clean)
|
||||
|
||||
- Verifier: 2,290 new probes TS + Prettier vs Rust, 0 silent; widths of 149,106 alphanumeric code points equal Prettier's (123 refused); drawings bound held on escape probes; every figure exact. C9: §11.14 says the synthetic oracle covers "references with URL and DOI" and "a level C question per priority" (no URL, no medium priority there; both covered by the equine report); `&name=` admitted only when the name starts with an ASCII lowercase letter. Notes: `markdown.rs` comment "the one glyph wider than its bytes" inaccurate; `scripts/import-equine.sh` header "(R5.2, R5.3)".
|
||||
- Fixed: the three wordings, the comment, the script header.
|
||||
|
||||
## Round 14 on eb667ad: verifier C1 to C9 MET, ECARTS(13) = FLOOR only (audit prose quoting suppression keywords; product paths clean on c6e0d85 and develop)
|
||||
|
||||
- Verifier: own differential 17,634 changesets TS + Prettier vs Rust, 12,212 report pairs identical, 22,928 refused, 0 silent, 0 raw difference (planted mutation detected); SVG differential 65 changesets, 236 drawings byte-identical, 4 under the R3 rule, 0 undeclared; every figure exact; `git diff c4e717d..HEAD` changes no executable line. Notes: emoji test accepts either refusal kind (code gives UnsupportedMarkdown); CLAUDE.md "Next: R4, R5" stale (pre-existing); RUST.md §11 heading lists up to R5.3.
|
||||
- Closing the 13 FLOOR lines needs a contract Clarification (owner); audit records stay append-only.
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
# Security audit: R5.5 upload tool, fix commits 56593af and fad85c0
|
||||
|
||||
- date: 2026-10-23 | branch `feature/rust-r5-pipeline-core` | scope `git diff a481af5..fad85c0`
|
||||
- context read: `crates/core/src/bin/upload_sheet_assets/{main,storage}.rs`, `crates/core/src/curation/upload.rs`,
|
||||
`scripts/{upload-sheet-assets,sheet-assets-e2e,floor-clarification-check}.sh`, `crates/core/tests/upload_tool.rs`,
|
||||
contract `.claude/tasks/contracts/2026-10-23-rust-r5-upload.md`
|
||||
- mode: audit (fresh, blind; no prior verdict used). Nothing was edited but this file.
|
||||
- not run, by rule: `upload-sheet-assets` against any real stack, `scripts/sheet-assets-e2e.sh`, any service key,
|
||||
any transfer tool, any recursive delete.
|
||||
|
||||
## What was measured
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| semgrep 1.168.0, p/security-audit + p/secrets + p/owasp-top-ten, 75 rules, 11 files (the 10 diff files + `curation/upload.rs`) | 0 findings, 0 errors |
|
||||
| guardrails `scan.py --domain api,supabase` | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`), outside the diff, dropped |
|
||||
| `cargo test --locked -p faunosteo-core --features upload --test upload_tool` (loopback stub, made-up key, built without `SUPABASE_SERVICE_ROLE_KEY` in the env) | 12 passed, 0 failed |
|
||||
| `floor-guard.sh develop`, then the script's three regexes applied to its output | 22 findings: 21 requalified (SUPPRESS in `.claude/audits/*.md`), 1 declared (`STUB .claude/memory/evals.md:205`), 0 remaining |
|
||||
| `/proc/sys/net/ipv{4,6}/ip_nonlocal_bind` | 0 and 0 |
|
||||
| UDP bind probe (no packet sent) on this host (`enp89s0` 192.168.1.100/24, brd 192.168.1.255) | 192.168.1.100 OK; 192.168.1.255 OK; 224.0.0.1 OK; 0.0.0.0 OK; 192.168.1.0, 192.168.1.101, 10.255.255.254 refused |
|
||||
| TCP connect to 192.168.1.255:54321 | refused by the kernel, ENETUNREACH (no SYN leaves) |
|
||||
| new suppressions in added lines (`nosemgrep`, `nosec`, `eslint-disable`, `@ts-ignore`, `noqa`, `#[allow`, `shellcheck disable`, `floor-guard: allow`) | none in code; the only hits are prose in `.claude/audits/2026-10-23-rust-r5-upload-security.md` |
|
||||
| tracked env files | `.env.example` only |
|
||||
|
||||
## Targeted review of the fixes
|
||||
|
||||
**Error classification and retry** (`storage.rs:17`, `:73-87`, `:189-214`). `ATTEMPTS = 4` gives one try and three
|
||||
retries, sleeps 1.0 s, 1.5 s, 2.0 s, so contract C6 "retried three times" holds. Retried: `Io`, `Timeout`,
|
||||
`HostNotFound`, `ConnectionFailed`, `Protocol`, and a 5xx status. Every other `ureq::Error` is final.
|
||||
`BodyExceedsLimit` maps to a final `State` refusal: the ureq 3.4.2 `LimitReader` raises it wrapped in an
|
||||
`io::Error`, and `From<io::Error>` unwraps it back (`ureq-3.4.2/src/error.rs:216-227`), so the guard does fire. The
|
||||
test `upload_tool.rs:319` proves it, with exactly one object GET and no write. A retried POST stays safe, because
|
||||
`x-upsert: false` turns a replay into a 409 handled as a duplicate and the object is then re-hashed. A retried
|
||||
confirm is server-idempotent. Secret exposure: the final branch prints only a fixed text or the numeric limit. The
|
||||
retried branch prints `error.to_string()`. For `Io`, `Timeout` and `Protocol` that text carries no header value:
|
||||
ureq-proto's `BadHeader` holds the `http::Error` text, not the value, and `BadAuthorizationHeader` has no payload
|
||||
(`ureq-proto-0.6.4/src/error.rs:45-53`). Request analysis also runs before the prelude is written
|
||||
(`client/sendreq.rs:42`), so a refused key never reaches the wire. The fix is a strict improvement: before it,
|
||||
`Http`, `BadUri`, `TooManyRedirects` and the rest were echoed and retried. One residual inaccuracy is listed as L1.
|
||||
|
||||
**DEV-address ownership** (`main.rs:112-124`). `UdpSocket::bind((ip, 0))` sends nothing and is closed on drop. It
|
||||
runs before the key is read, and `check_destination` still requires a private IPv4 equal to the DEV host
|
||||
(`upload.rs:190-200`). Unspecified and multicast addresses bind but are not private, so they stay refused. The
|
||||
directed broadcast of an attached private subnet passes, but the kernel refuses the TCP connection. The test
|
||||
`upload_tool.rs:342` discriminates: without the filter the run would reach the network (exit 3), not a
|
||||
`Destination` refusal (exit 1). Residuals: L2.
|
||||
|
||||
**Keys off every argv in the e2e script** (`sheet-assets-e2e.sh:18-32`). The key reaches the binary only through a
|
||||
prefix assignment inside the `$(...)` subshell (env, not argv). The leak check feeds the patterns to
|
||||
`grep -f <(printf ...)`: `printf` is a builtin, and grep's argv holds only `/dev/fd/N`. The keys come from the
|
||||
status JSON on stdin; the `python3 -c` argv carries only the field name. `env -u SUPABASE_SERVICE_ROLE_KEY cargo
|
||||
build` puts the variable name on argv, never its value. Dropping `env -i` lets the binary inherit the caller's env.
|
||||
This is no exposure: the tool reads only `SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, `FAUNOSTEO_DEV_HOST` and
|
||||
`INIT_CWD` (the last two are unset there), it uses `proxy(None)`, and the crate has no logger. An empty key or
|
||||
status field would put an empty pattern into `-f`, which matches every output. That fails the case: fail-safe.
|
||||
|
||||
**Launcher** (`upload-sheet-assets.sh:15-16`). Under `set -euo pipefail`, a failed build stops the script before
|
||||
the `exec`. Cargo, build scripts and proc macros no longer see the key. Only the `exec`'d binary receives it, in its
|
||||
environment, with the folder and `--check` as its only argv. Residual: L3.
|
||||
|
||||
**FLOOR declared line** (`floor-clarification-check.sh:16`, `:23-27`). `DECLARED_RE` is anchored and covers one
|
||||
kind, one file and one line, with the dots escaped and a trailing space, so `:2050` cannot match. On the branch it
|
||||
matches exactly the one line the R5.4 contract names (`2026-10-23-rust-r5-review.md:69`). Nothing else is
|
||||
requalified, `requalified` does not count it, and it is printed under `DECLARED`. The negative proof still fails
|
||||
(real SUPPRESS in `src/*.ts`, `src/*.rs` and an active contract). Residual: L4.
|
||||
|
||||
## Findings
|
||||
|
||||
No CRITICAL, no HIGH, no MEDIUM, no new ungated suppression.
|
||||
|
||||
- **L1** `storage.rs:196` (with `:200-202`): `Protocol(_)` counts as transient. That bucket also holds deterministic
|
||||
local failures: `BadAuthorizationHeader` for a key with non-ASCII bytes (the `http` crate accepts bytes 0x80 and up
|
||||
in a value, ureq-proto's `to_str` refuses them), `OutputOverflow` for a huge key, and `HttpParseFail` for a
|
||||
malformed answer. Each is retried three times and ends as `Network`, exit 3. The `final_error` doc comment says
|
||||
"a key a header refuses, a malformed answer" are final, which is only true for control characters (`Http`). This
|
||||
costs about 4.5 s and a wrong exit code, and nothing is sent or printed. Hint: make `Protocol` final, or retry only
|
||||
its response-side variants, and correct the comment.
|
||||
- **L2** `main.rs:112-115`: the bind probe also accepts the subnet broadcast (measured: 192.168.1.255). With
|
||||
`FAUNOSTEO_DEV_HOST=192.168.1.255` and `SUPABASE_URL=http://192.168.1.255:54321`, the destination check passes,
|
||||
the kernel refuses the TCP connection (ENETUNREACH, measured), the key is never sent, and the run ends `Network`
|
||||
instead of `Destination`. The probe also degrades to "any private IPv4" on a host with `ip_nonlocal_bind=1`
|
||||
(0 here). Hint: after the bind, `connect((ip, port))` the UDP socket (sends nothing; a broadcast gives EACCES,
|
||||
measured) and require `local_addr().ip() == ip`, and document the sysctl dependency.
|
||||
- **L3** `upload-sheet-assets.sh:15-16` and `sheet-assets-e2e.sh:15,18`: the binary path `target/release/...` is
|
||||
hard-coded, but cargo honours `CARGO_TARGET_DIR`, `CARGO_BUILD_TARGET` and a user `build.target-dir`. If any is
|
||||
set, the build lands elsewhere and the script `exec`s a stale binary, possibly pre-fix, with the service key in its
|
||||
environment (or a missing one). None is set on this machine (measured; no `~/.cargo/config.toml`, and the workspace
|
||||
`.cargo/config.toml` has no target dir). Hint: `env -u CARGO_TARGET_DIR -u CARGO_BUILD_TARGET` on the build, or
|
||||
take the path from `cargo build --message-format=json`.
|
||||
- **L4** `floor-clarification-check.sh:16`: the exemption is pinned by line number, not content. If a
|
||||
`/prune-memory` rewrite puts another STUB on `evals.md:205`, that STUB is silently exempt, while the original
|
||||
finding fails loudly. The negative proof has no `evals.md`, so the regex's narrowness is proven only by reading.
|
||||
Since this change, the script prints OK while the finding waits for the owner's decision. That is a quality-gate
|
||||
question for the verifier and has no security effect. Hint: also match the snippet prefix (`Anomalies: LOW
|
||||
report.rs`), and add a proof case with a STUB on another `evals.md` line.
|
||||
- **L5** `upload_tool.rs:346-347`: the "not this machine" address 10.255.255.254 is the one WSL2 assigns to `lo`.
|
||||
On such a host the test would connect locally with the made-up key and fail with exit 3. This is a portability
|
||||
nit (fail-loud) with no exposure. Hint: use a TEST-NET address (192.0.2.1 is not private, so pick a private
|
||||
address checked to be non-local at run time).
|
||||
|
||||
Context only, outside the diff, not counted: `supabase/config.toml` sets no `jwt_secret` (only `jwt_expiry` at
|
||||
`:89`). The local stack therefore likely signs with the CLI's public default secret, and port 54321 is open to the
|
||||
LAN by design (CLAUDE.md "Environments and LAN access"). Anyone on the LAN who knows the public default
|
||||
service_role JWT would then hold service_role on the local stack, which makes the tool's key secrecy moot there. I
|
||||
did not verify this against the running stack. The diff's RUST.md text names the same point.
|
||||
|
||||
## Verdict block
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 11 files (10 in a481af5..fad85c0 + crates/core/src/curation/upload.rs read as context)
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] retry-classification (Protocol treated as transient, doc comment overstates finality) — crates/core/src/bin/upload_sheet_assets/storage.rs:196
|
||||
- [LOW] dev-address-probe (subnet broadcast binds; relies on ip_nonlocal_bind=0) — crates/core/src/bin/upload_sheet_assets/main.rs:112
|
||||
- [LOW] stale-binary-exec (hard-coded target/release vs CARGO_TARGET_DIR) — scripts/upload-sheet-assets.sh:16, scripts/sheet-assets-e2e.sh:15
|
||||
- [LOW] floor-declared-line (pinned by line number; no negative proof for DECLARED_RE) — scripts/floor-clarification-check.sh:16
|
||||
- [LOW] test-portability (10.255.255.254 is lo on WSL2) — crates/core/tests/upload_tool.rs:346
|
||||
PROOF: semgrep 75 rules on 11 files → 0 findings; checklist 9 checks (secrets, SQL concat, unsafe render, authz, error exposure, secret in log, tracked .env, guardrails, new suppressions) + 5 targeted fix reviews → 5 findings (all LOW); cargo test upload_tool 12/12 green; floor classification 22 → 21 requalified, 1 declared, 0 remaining
|
||||
```
|
||||
@@ -0,0 +1,185 @@
|
||||
# Security audit: Rust R5.5 upload tool (2026-10-23)
|
||||
|
||||
Mode: audit. Scope: `git diff 637ea63..HEAD` (commits 2c44506, a481af5), 25 files, branch
|
||||
`feature/rust-r5-pipeline-core`. Contract: `.claude/tasks/contracts/2026-10-23-rust-r5-upload.md`.
|
||||
Fresh, blind scan. No run against a real stack, no transfer tool, no service key used.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 25 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] secret-on-argv — scripts/sheet-assets-e2e.sh:29, :31
|
||||
- [LOW] secret-in-build-env — scripts/upload-sheet-assets.sh:14
|
||||
- [LOW] destination-trust (cleartext key, DEV address not proven local) — crates/core/src/curation/upload.rs:194-198
|
||||
- [LOW] error-classification (non-transient errors retried as Network) — crates/core/src/bin/upload_sheet_assets/storage.rs:76-81
|
||||
- [LOW] doc-exit-code — docs/development/SUPABASE.md:152 vs crates/core/src/bin/upload_sheet_assets/main.rs:167-175
|
||||
- [LOW] toctou-file-read (documented, hash-mitigated) — crates/core/src/bin/upload_sheet_assets/main.rs:49-67
|
||||
- [LOW] confirm-not-hash-bound (pre-existing RPC) — crates/core/src/bin/upload_sheet_assets/storage.rs:173-181
|
||||
- [LOW] lint-allow (architecture lint, not security) — crates/core/src/bin/upload_sheet_assets/main.rs:21 + 4 test files
|
||||
- [LOW] floor-guard-waiver (fixture lines, not security) — scripts/floor-clarification-check.sh:42-45
|
||||
- [LOW] serde-tolerance (pre-existing, characterised only) — crates/core/tests/characterisation_serde_tolerance.rs
|
||||
PROOF: semgrep 45 rules on 25 files → 0 findings; checklist 21 checks → 10 findings
|
||||
```
|
||||
|
||||
## Method
|
||||
|
||||
- semgrep 1.168.0, the three pinned packs, `--metrics=off`, on the 25 existing files of the diff:
|
||||
45 rules applied (Rust, bash, JSON, YAML, Markdown), 0 findings, 0 errors, 0 skipped rules.
|
||||
Semgrep's Rust coverage is thin, so the review below is mostly manual reading.
|
||||
- Checklist (8): hardcoded secret, SQL by concatenation, unsanitised render, sensitive endpoint
|
||||
authz, internals surfaced, secret in a log, tracked `.env` / credential file, guardrails scanner
|
||||
(`scan.py --domain api,supabase`: 0 FAIL, 1 WARN outside the diff, see below).
|
||||
- Anti-gaming (1): every added suppression or waiver line in the diff.
|
||||
- Brief focus (12): destination allowlist, redirects, proxy, key exposure (logs, errors, argv, env,
|
||||
stubs, e2e), path traversal and symlinks, TOCTOU, size and memory bounds, overwrite / delete /
|
||||
visibility, partial failure and resume, error classification, retries on non-idempotent calls,
|
||||
dependency additions.
|
||||
- Dynamic, hermetic only: `cargo test -p faunosteo-core --features upload --locked` on
|
||||
`upload_tool`, `curation_upload`, `curation_caps`, `characterisation_serde_tolerance`: 25/25
|
||||
pass (loopback stub, made-up key, `SUPABASE_*` unset). A scratch probe (scratchpad crate, path
|
||||
dependency, `--offline`) fed 40 crafted URLs to `check_destination`.
|
||||
|
||||
## What holds (verified, not taken from the docs)
|
||||
|
||||
- Destination allowlist. 40 probes: decimal, hex, octal and short IPv4 forms (`2130706433`,
|
||||
`0x7f000001`, `0177.0.0.1`, `127.1`) all normalise to a loopback and are accepted as such;
|
||||
IPv4-mapped IPv6 (`[::ffff:127.0.0.1]`, `[::ffff:7f00:1]`), `0.0.0.0`, `0`, `localhost.`,
|
||||
`localhost.localdomain`, `127.0.0.1.nip.io`, public addresses, names, `ws:`, `file:`, `https:`,
|
||||
zone ids are refused. Userinfo and backslash tricks (`evil.com\@127.0.0.1`, `127.0.0.1:54321@evil.com`,
|
||||
`evil.com#@127.0.0.1`) are refused. Accepted oddities (`http://@127.0.0.1:p`, percent-encoded or
|
||||
full-width `localhost`, `http:127.0.0.1:p`, trailing `/.`) all reduce to an allowed host. The
|
||||
decisive property: the origin used for every request is rebuilt from the parsed `Host` and port
|
||||
(`upload.rs:224-233`), never the raw string, so no check/use parser differential exists.
|
||||
- The key is read only after the destination check (`main.rs:110-115`) and after every local file
|
||||
is verified (`main.rs:177-178`): a refused destination or a bad file sends nothing.
|
||||
- No redirect: `max_redirects(0)` in ureq 3.4.2 returns the 3xx as a response, never follows it and
|
||||
never errors (`ureq-3.4.2/src/run.rs:224-236`, `config.rs:257-258`); the 3xx then fails
|
||||
`expect_status`. No proxy: `.proxy(None)` overrides the default `Proxy::try_from_env()`
|
||||
(`ALL_PROXY` / `HTTP_PROXY`). No `gzip` / `brotli` features (workspace `Cargo.toml:43`), so no
|
||||
decompression bomb; every body is capped (1 MiB, objects 16 MiB + 1).
|
||||
- Key never in a URL, a message or a log: two headers only (`storage.rs:88-89`), `Secret` has a
|
||||
redacted `Debug` and no `Display`, `Storage` has no `Debug`; `required_env` names the variable,
|
||||
never the value; ureq error texts (`error.rs:229-276`) carry no header value; server bodies are
|
||||
never echoed (`expect_status`). Test key is a made-up literal (`upload_tool.rs:14`).
|
||||
- No overwrite, delete or visibility change: only GET and POST are issued (`storage.rs:104, 123,
|
||||
142, 161, 177`), uploads carry `x-upsert: false`, the bucket must answer `public: false`, an
|
||||
existing object is hashed and other bytes refuse the run.
|
||||
- Order and resume: whole plan computed read-only before the first write (`main.rs:180`); the
|
||||
object is read back and hashed before the row is confirmed (`main.rs:147-163`). The POST retry is
|
||||
safe: a retried create that already landed answers Duplicate, which leads to the read-back check;
|
||||
the confirm RPC is idempotent (`20261009100000_sheet_asset_storage.sql:41-50`).
|
||||
- Paths: folder and files canonicalised and confined with component-wise `starts_with`; manifest
|
||||
file names are `[A-Za-z0-9._-]`, no leading dot, `.svg`; ids are lowercase hex / uuid, so the
|
||||
object key and the PostgREST filter carry no injection.
|
||||
- Memory: drawings are read under 16 MiB, then refused above 2 MiB by the SVG reader
|
||||
(`prescan.rs:33`), so at most 64 x 2 MiB is held; curation files now capped at 4 MiB with
|
||||
`take(cap + 1)` (`import_equine.rs:93-99`) and in the core (`schema.rs:386-392`).
|
||||
- C1: `curation/upload.rs` has no `std::fs`, `std::env`, socket or clock. `ureq` is optional under
|
||||
`upload` only; `crates/wasm` enables `pdf-read, pdf-write` only. Cargo.lock adds one edge
|
||||
(`faunosteo-core -> ureq`); ureq 3.4.2 was already locked for the worker, no new package.
|
||||
|
||||
## Findings
|
||||
|
||||
### M1 [MEDIUM] service and anon keys on process argv in the e2e script
|
||||
|
||||
`scripts/sheet-assets-e2e.sh:29` runs `env -i ... SUPABASE_SERVICE_ROLE_KEY="$key" "$BIN"`:
|
||||
`env` is `/usr/bin/env`, an external process, so the key sits in its argv, readable by every local
|
||||
user through `/proc/<pid>/cmdline` and `ps`, and recorded by any execve audit rule. Line 31,
|
||||
`grep -qF -e "$key" -e "$SERVICE" -e "$ANON"`, puts the service and anon keys on the argv of an
|
||||
external `grep`. This contradicts the script's own header ("held in variables, never printed") and
|
||||
the C7 intent. Impact is bounded: the key is the local stack's, the machine is single-user, the
|
||||
window is short, the owner runs the script. Hint: pass the key as a bash prefix assignment in a
|
||||
subshell that first unsets the variables you want cleared (`( unset FAUNOSTEO_DEV_HOST INIT_CWD;
|
||||
SUPABASE_URL="$url" SUPABASE_SERVICE_ROLE_KEY="$key" exec "$BIN" "$@" )`), and match with
|
||||
`grep -qF -f <(printf '%s\n' "$key" "$SERVICE" "$ANON")` (`printf` is a builtin).
|
||||
|
||||
### L1 [LOW] service key in the environment of the compile step
|
||||
|
||||
`scripts/upload-sheet-assets.sh:14` runs `cargo run --release`: on a first run, or after any source
|
||||
or lockfile change, cargo builds with the operator's exported `SUPABASE_SERVICE_ROLE_KEY` visible
|
||||
to every build script and proc-macro of the dependency tree. The tree is locked and cargo-deny'd,
|
||||
so this is a supply-chain exposure only. Hint: `env -u SUPABASE_SERVICE_ROLE_KEY cargo build
|
||||
--release --locked ...`, then `exec target/release/upload-sheet-assets "$@"`.
|
||||
|
||||
### L2 [LOW] destination trust: cleartext key, DEV address not proven local
|
||||
|
||||
The key always travels over plain `http` (https is refused), with no server authentication. Any
|
||||
process listening on the allowed local port receives it: declared and accepted in RUST.md §11.16.
|
||||
The DEV branch (`upload.rs:194-198`) accepts any private IPv4 equal to `FAUNOSTEO_DEV_HOST`
|
||||
without proving it is an address of this machine; if both variables name another LAN host, the key
|
||||
crosses the LAN in clear. C2 allows this and `scripts/dev-env.sh:15` derives the address from
|
||||
`hostname -I` by default. Hint: in the binary, `UdpSocket::bind((ip, 0))` succeeds only for a
|
||||
local interface address; refuse otherwise.
|
||||
|
||||
### L3 [LOW] non-transient client-side errors retried and reported as Network
|
||||
|
||||
`storage.rs:76-81` maps every `ureq::Error` to `Network`, retries three times, exit 3. That
|
||||
includes `BodyExceedsLimit` when the destination holds an object above 16 MiB + 1 under the key
|
||||
(a State condition: downloaded three times, then "network error") and `Error::Http` from a key
|
||||
holding characters invalid in a header (a usage error). No write follows either case, so no
|
||||
security consequence, but C6 says refusals are final and C8 that errors are classified. Hint:
|
||||
match `BodyExceedsLimit` to `State` and `Http` / `BadUri` to a non-retried kind.
|
||||
|
||||
### L4 [LOW] exit code 2 documented as "usage"
|
||||
|
||||
`docs/development/SUPABASE.md:152` lists "2 usage"; a wrong argument count exits 1 with kind
|
||||
`Content` (`main.rs:167-175`, asserted by `upload_tool.rs:314`). The code matches C8 ("2 usage
|
||||
(argument not UTF-8)"); the runbook overstates. Hint: say "2 an argument is not UTF-8".
|
||||
|
||||
### L5 [LOW] check/open race on file reads (documented, mitigated)
|
||||
|
||||
`main.rs:49-67`: canonicalise, then `metadata`, then `open` the canonical path; a component swap
|
||||
between them is not defended, as the comment states. Mitigation is real: the bytes read must hash to
|
||||
the manifest's sha256 and pass the SVG reader, so a swap cannot exfiltrate unknown content. Single
|
||||
user working copy.
|
||||
|
||||
### L6 [LOW] confirmation not bound to the hash (pre-existing RPC)
|
||||
|
||||
`storage.rs:173-181` confirms by asset id; the row hash is checked at plan time and the object is
|
||||
verified just before, but `confirm_sheet_asset_upload(asset_id)`
|
||||
(`supabase/migrations/20261009100000_sheet_asset_storage.sql:34-52`, outside the diff) does not
|
||||
re-check `sha256`. A row hash change between plan and confirm would confirm an unverified pairing.
|
||||
No API role writes asset rows (`authenticated` holds SELECT only), so the window needs a
|
||||
service-role actor. Note for a future migration (an expected-hash parameter), not this diff.
|
||||
|
||||
### L7 [LOW] new lint allows (context, not a security suppression)
|
||||
|
||||
`main.rs:21` `#![allow(clippy::disallowed_methods)]` and the same inner attribute (plus
|
||||
`unwrap_used`, `expect_used`) at the head of the four new test files. `clippy.toml` lists clock,
|
||||
file and environment methods: an architecture lint for the core's I/O purity (C1), the same pattern
|
||||
as `import_equine.rs` and `report_compare.rs`. Not a security checker, not blocking. CLAUDE.md
|
||||
still says "the two files allowed std::fs"; it is three now.
|
||||
|
||||
### L8 [LOW] floor-guard waivers on fixture lines (context)
|
||||
|
||||
`scripts/floor-clarification-check.sh:42-45` carry `# floor-guard: allow negative-proof fixture`
|
||||
on lines that write `@ts-ignore`, `nosemgrep` and `nosec` into a `mktemp -d` throwaway repo. The
|
||||
tokens are bash string literals; they silence no semgrep or security check on this repository
|
||||
(no finding on those lines, semgrep ignores `nosec`). The waiver silences floor-guard (a test-floor
|
||||
guard, printed WAIVED) and carries no `[gated <date>]` marker. Not blocking under the security
|
||||
rule; reported so the orchestrator sees it.
|
||||
|
||||
### L9 [LOW] serde tolerance of the gated readers (pre-existing, characterised only)
|
||||
|
||||
`crates/core/tests/characterisation_serde_tolerance.rs` pins that the R2 contracts accept a struct
|
||||
as a positional array and a unit enum as a one-key map, where Zod refused. This reaches the
|
||||
untrusted PDF import path: an integrity risk for crafted files (fields assigned in the wrong order
|
||||
without error), not a memory or execution one. No reader changed in this diff; the owner decision
|
||||
is pending before R5.6 (RUST.md §11.16).
|
||||
|
||||
## Outside the diff (not counted)
|
||||
|
||||
- guardrails WARN `SB-POLICY-NOT-OWNER`
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177` (`species_select_active` not
|
||||
tied to `auth.uid()`): file not in the diff.
|
||||
- Blind spot for the owner: `supabase/config.toml` sets no `jwt_secret` and no signing keys, so
|
||||
the local stack most likely signs with the Supabase CLI default secret, making the local
|
||||
`service_role` key a publicly documented value. ENVIRONMENTS.md exposes the API port 54321 to
|
||||
the LAN. If that holds, anyone on the private network can already act as `service_role` on the
|
||||
DEV stack, and the care this tool takes with the key protects little there. I did not run
|
||||
`supabase status` to confirm the key value.
|
||||
- Not verified here: `scripts/sheet-assets-e2e.sh` against the real stack (owner-run by design;
|
||||
not run, per the brief). The anon-key and wrong-key cases rely on the real Storage answer.
|
||||
@@ -0,0 +1,21 @@
|
||||
# R5.5 verification — round 1 (fresh verifier, Opus; pinned model out of credits) on 2c44506 + a481af5
|
||||
|
||||
Contract `.claude/tasks/contracts/2026-10-23-rust-r5-upload.md`. Persisted by the orchestrator (the verifier is read-only).
|
||||
|
||||
VERIFY — VERDICT: ECARTS(4)
|
||||
|
||||
- C1 MET, C2 MET (`http://127.0.0.1:80` refused, stricter, pinned), C3 MET, C4 MET
|
||||
- C5 NOT-MET: an object over MAX_ASSET_BYTES+1 hits ureq BodyExceedsLimit, mapped to Network and retried; exit 3 instead of a final State (no write happens, safety holds)
|
||||
- C6 NOT-MET: ATTEMPTS = 3 is three attempts (two retries), not "retried three times"
|
||||
- C7 MET (observation: e2e script puts the key on `env` argv, security M1)
|
||||
- C8 MET, C9 MET, C10 MET
|
||||
- C11 NOT-MET: SUPABASE.md exit codes say "2 usage", code exits 1 on usage; exit-1 list misses manifest and handshake. Nit: serde table "zone as array" row unpinned and contradictory wording
|
||||
- C12 MET (no frozen path in the diff, documentSha256 27c7383c…2660, import --check 0 differences 6+12)
|
||||
- C13 MET with caveat: `floor-clarification-check.sh` cannot print OK while the declared evals.md:205 STUB stands
|
||||
- C14 UNVERIFIABLE by the verifier (check:all ran elsewhere: green 21:21:38, log in the session scratchpad)
|
||||
- FLOOR diff-scoped: clean, 4 waived (fixture lines authorised by the R5.4 clarification)
|
||||
- Runs: cargo test upload 251/29 suites, curation 241/27, workspace 277 (1 ignored), cli 259; fmt, clippy (5 configs), deny ok; `--check` RSS 18696 KB
|
||||
|
||||
## Fixes after round 1 (orchestrator)
|
||||
|
||||
C5: `final_error` maps BodyExceedsLimit to a final State, test `an_oversized_object_is_a_final_state_refusal_not_retried`. C6: ATTEMPTS = 4 (one try, three retries). C11: SUPABASE.md exit codes and retry wording; RUST.md table row reworded. C13: the script lists the declared line apart (still not requalified) and prints OK. Security M1, L1, L2, L3 fixed (RUST.md §11.16, gate round 1).
|
||||
@@ -0,0 +1,208 @@
|
||||
# Security audit: R5.5 closing corrections (diff 13db553..257a29b)
|
||||
|
||||
- date: 2026-10-24 | branch `feature/rust-r5-pipeline-core` | mode: audit (targeted diff), fresh run
|
||||
- contract: `.claude/tasks/contracts/2026-10-24-rust-r5-5-closing.md` (K1-K8)
|
||||
- scope: `git diff 13db553..257a29b`, 15 files (8 code/doc + 7 `.claude/` registry, task, plan files)
|
||||
- nothing run against a real stack, no service key used, no transfer tool, no delete, no code edited
|
||||
|
||||
## Focus 1: `check_v3_shape` and the untrusted v3 entries
|
||||
|
||||
Entries traced (wasm `crates/wasm/src/lib.rs` → `crates/core/src/facade.rs` → `ops.rs` / `session.rs`):
|
||||
|
||||
| Entry | Path | Shape check |
|
||||
|---|---|---|
|
||||
| `SheetHandle.import_pdf` / `import_pdf_bytes` | `facade::import_pdf` → `pdf/import.rs:29` `import_archive` → `read_archive_document` → `read_report_document` | yes (`report/read.rs:80`) |
|
||||
| `archive.import` op | `ops.rs:345` → `session::import_archive` → same | yes |
|
||||
| `draft.restore` op, draft text | `ops.rs:316` → `session.rs:85` `read_report_document` | yes |
|
||||
| `draft.restore` op, archive text | `session.rs:103-106` `parse_envelope` | yes (`archive/read.rs:124-125`) |
|
||||
| `restore_copy_bytes` | `facade.rs:86-101` → `restore_draft(.., None)` | yes |
|
||||
| `report-compare` CLI | `pdf/compare/mod.rs:133` → `read_archive_document` | yes |
|
||||
|
||||
These are the only paths by which an external v3 report enters. The remaining typed reads of
|
||||
`Report` take the store's report, which is always the core's canonical output: the reducers
|
||||
(`ops.rs:64-87`), `archive.snapshot` (`ops.rs:301-303`), the export session (`pdf/write/input.rs:26`),
|
||||
and `export_from_envelope` (`pdf/write/export.rs:117-130`, a public library fn with test-only
|
||||
callers, not exported by wasm or a binary).
|
||||
|
||||
Panic review: `check_v3_shape` (`report/read.rs:55-71`) uses only `Value::get`, `as_array`,
|
||||
`is_object`, `is_string`, `enumerate` and `format!`. No indexing, no unwrap, no recursion. A
|
||||
non-object or `null` input returns `Ok` and the typed read refuses it. No panic path.
|
||||
|
||||
Authentic shapes: the core's writer emits `sheet` and `practitioner` as objects and `severity` as
|
||||
a string or omits it (`skip_serializing_if`). Zod refused arrays, one-key maps and `null` for every
|
||||
earlier v3 writer. `null` severity was already refused before this diff: `js::present` calls
|
||||
`Severity::deserialize(null)`, which fails. No authentic shape is newly refused. Hermetic suites
|
||||
were rerun here and are green: `archive_rules` 38, `report_rules` 17, `characterisation` 4,
|
||||
`characterisation_serde_tolerance` 5, `pdf_import` 3.
|
||||
|
||||
Bypass found (M1 below). The check reads keys on the raw value, so any shape where the parent
|
||||
container is itself positional skips it, because `Value::get(&str)` on an array returns `None`.
|
||||
|
||||
## Focus 2: upload tool, `Protocol` and `own_address`
|
||||
|
||||
`Protocol` is final: `storage.rs:189-197` retries `Io`, `Timeout`, `HostNotFound` and
|
||||
`ConnectionFailed` only, plus 5xx (`storage.rs:79-80`). `final_error` (`storage.rs:203-215`)
|
||||
carries no error text, so neither the key nor the header shows. The test
|
||||
`a_malformed_answer_is_final_and_not_retried` is green: one request, exit 1.
|
||||
|
||||
`own_address` (`main.rs:114-122`), measured on this host with the same syscalls (UDP bind then
|
||||
connect, nothing sent):
|
||||
|
||||
| DEV host | bind | connect | local_addr == ip | own_address |
|
||||
|---|---|---|---|---|
|
||||
| 255.255.255.255 (and the host's `brd` addresses, unit test) | ok | EACCES (13) | n/a | false |
|
||||
| 0.0.0.0 | ok | ok | 127.0.0.1, no | false |
|
||||
| 224.0.0.251, 239.255.255.250 (multicast) | ok | ok | yes | **true** (L3) |
|
||||
| another host's private address | EADDRNOTAVAIL (`ip_nonlocal_bind` = 0) | n/a | n/a | false |
|
||||
|
||||
Even with `ip_nonlocal_bind=1`, the connect route lookup refuses a non-local source address
|
||||
(no `IP_FREEBIND` / `IP_TRANSPARENT` is set). So a non-local unicast address does not pass.
|
||||
|
||||
Defense in depth holds. `host_is_allowed` (`curation/upload.rs:190-200`) only accepts the DEV
|
||||
host when the URL host is loopback or RFC 1918 and parses equal to it. Multicast and public
|
||||
addresses are neither. The kernel also refuses a TCP connect to multicast.
|
||||
|
||||
Packet: `own_address` sends nothing. A UDP `connect` only records the peer, there is no
|
||||
`send`/`send_to`, and the socket is dropped on return. It joins no multicast group.
|
||||
|
||||
## Focus 3: secrets and SQL claims
|
||||
|
||||
- No secret value in the diff. The added lines contain no JWT (`eyJ…`), no CLI default secret
|
||||
string, no `sb_secret_` / `sb_publishable_` key and no password literal. `iss: supabase-demo`
|
||||
is a claim name, not a key.
|
||||
- Repo-wide (tracked, at 257a29b) search for the CLI default JWT secret, `eyJhbGciOi`,
|
||||
`eyJpc3Mi` and `sb_secret_`/`sb_publishable_` keys. Only `crates/worker/src/config.rs:324,326`
|
||||
matches: a fabricated test token (payload `{"role":"service_role","exp":…}`, signature
|
||||
literally `sig`) and the placeholder `sb_secret_abc`. These are not credentials. The only
|
||||
tracked env file is `.env.example`.
|
||||
- `supabase/config.toml` `[auth]` (lines 73-96) sets no `jwt_secret` and no signing keys, which
|
||||
matches ENVIRONMENTS.md §3.6. The claim that the service key is the public demo key was not
|
||||
checked independently: that would mean reading the key value.
|
||||
- `confirm_sheet_asset_upload`: `20261009100000_sheet_asset_storage.sql:57-58` runs `revoke all
|
||||
… from public, anon, authenticated` and `grant execute … to service_role`. No later migration
|
||||
re-grants it. The function (lines 34-52) sets `uploaded_at` with no `storage.objects` existence
|
||||
or byte check. pgTAP `006_sheet_asset_storage.sql:73-80` covers editor 42501, service-role
|
||||
confirm of A and B, and idempotence. It has no `anon` assertion, which matches the doc.
|
||||
- The bucket has no INSERT, UPDATE or DELETE policy. The only `storage.objects` policies in
|
||||
all migrations are `anatomical_sheet_assets_read` (select), `avatars_read` (select) and
|
||||
`avatars_upload` (insert, pinned to `bucket_id = 'avatars'`, `20261013100000:284-291`).
|
||||
RUST.md §11.18 is accurate.
|
||||
- No file under `supabase/` changed in the diff (K5).
|
||||
|
||||
## Findings
|
||||
|
||||
### M1 [MEDIUM] `check_v3_shape` bypassable through a positional parent
|
||||
`crates/core/src/domain/report/read.rs:61-69`, `crates/core/src/domain/archive/read.rs:124-125`
|
||||
|
||||
- An observation written as a positional array (`[id, zones, findings, freeTextFindings,
|
||||
{"mild": null}, comment]`) is accepted on every entry. `observation.get("severity")` returns
|
||||
`None` on an array, and serde's derived `Observation` visitor reads the sequence, one-key
|
||||
severity map included.
|
||||
- In `parse_envelope`, a `report` written as an array (or the whole envelope as an array) makes
|
||||
`check_v3_shape` see no keys. All three refused shapes (sheet array, practitioner array,
|
||||
severity map) are then accepted.
|
||||
- In the import path a report-level or envelope-level array is still refused, because
|
||||
`version_of` and `envelope_version_of` require an object.
|
||||
- K4 states that the three shapes are refused at both v3 entries. That holds only when every
|
||||
parent is an object.
|
||||
- Impact is low. The field values still pass every field rule and the core re-serialises
|
||||
canonically, so there is no injection. What remains is a parser differential with the N-1 Zod
|
||||
reader. Established by reading: it is the same `visit_seq` mechanism the R5.5
|
||||
characterisation proved for `sheet` and `practitioner`. No probe was executed, because writing
|
||||
a test is outside this audit.
|
||||
- Fix direction (owner scope): require `observations[i]` to be an object, and in
|
||||
`parse_envelope` require both `data` and `report` to be objects before the shape check. A
|
||||
generic guard is an alternative.
|
||||
|
||||
### M2 [MEDIUM, pre-existing, outside the diff, documented by it] DEV API reachable as `service_role` from the LAN
|
||||
`supabase/config.toml:73-96`, `docs/development/ENVIRONMENTS.md` §3.6
|
||||
|
||||
- The stack runs on the CLI's public default JWT secret and 54321 is LAN-open by design.
|
||||
- Any device on the private network can therefore act as `service_role` on DEV data, bypassing
|
||||
RLS. The doc states this accurately and leaves the choice to the owner. Production is not
|
||||
involved.
|
||||
- Not introduced by this diff, so not gated here. Resolve it before any non-test data reaches
|
||||
DEV: own `jwt_secret` / signing keys through `env()`, or narrow the 54321 rule.
|
||||
|
||||
### L1 [LOW] Remaining serde tolerance on the other nested v3 structs
|
||||
`crates/core/src/domain/report/read.rs:52-54`
|
||||
|
||||
- `animal`, `owner`, `zones[]`, `findings[]`, `vertebrae[]` and `nextAppointment` variants as
|
||||
positional arrays, and `side` as a one-key map, are still accepted.
|
||||
- This is within the owner's go ("three shapes only"). But the doc comment ("The shapes Zod
|
||||
refused and serde's derives accept") reads as a complete list, and BDR-069's rationale
|
||||
("crafted file could swap same-typed fields") applies to these shapes too.
|
||||
- Hint: word the comment and §11.18 as "three of the shapes".
|
||||
|
||||
### L2 [LOW] Error text contradicts behaviour
|
||||
`crates/core/src/domain/report/read.rs:66`
|
||||
|
||||
The message says "expected a string or null", but `null` is refused. The test's own comment
|
||||
says so (`characterisation_serde_tolerance.rs`, "null stays refused").
|
||||
|
||||
### L3 [LOW] `own_address` accepts IPv4 multicast
|
||||
`crates/core/src/bin/upload_sheet_assets/main.rs:110-122`
|
||||
|
||||
- Measured: bind and connect succeed and `local_addr` equals the address. The "own unicast"
|
||||
claim in the doc comment and in RUST.md §11.18 is therefore inaccurate.
|
||||
- Not exploitable: `host_is_allowed` requires RFC 1918 or loopback, and TCP to multicast is
|
||||
refused by the kernel. The unit test (`main.rs:255-266`) does not cover multicast.
|
||||
- Hint: add `!ip.is_multicast()`, or rely on and state the private-range filter.
|
||||
|
||||
### L4 [LOW] `own_address` unit test is vacuous without `ip`
|
||||
`crates/core/src/bin/upload_sheet_assets/main.rs:237-244`
|
||||
|
||||
If the `ip` command is missing, `unwrap_or_default` yields empty lists and the host unicast and
|
||||
broadcast loops check nothing. The test still passes. This affects test strength only.
|
||||
|
||||
### L5 [LOW, pre-existing, documented] `confirm_sheet_asset_upload` does not check the object
|
||||
`supabase/migrations/20261009100000_sheet_asset_storage.sql:34-52`
|
||||
|
||||
The guard is EXECUTE (service_role only) and no API role can place bytes, so this is not
|
||||
exploitable today. The debt is recorded in RUST.md §11.18: check the object's existence and size
|
||||
before any public exposure.
|
||||
|
||||
### L6 [LOW] Typed reads of `Report` outside the two checked entries
|
||||
`crates/core/src/ops.rs:64-87, 301-303`, `crates/core/src/pdf/write/input.rs:26`,
|
||||
`crates/core/src/pdf/write/export.rs:117-130`
|
||||
|
||||
These take store or core-canonical reports today, so none of them is an external entry. If
|
||||
`export_from_envelope` ever reads a file-borne envelope, it skips `check_v3_shape`.
|
||||
|
||||
## Anti-gaming
|
||||
|
||||
The diff adds no `nosemgrep`, `nosec` or `eslint-disable` comment and no new `#[allow]`. The
|
||||
`#![allow(clippy::…)]` in `characterisation_serde_tolerance.rs` predates the diff and is a lint
|
||||
allowance, not a security suppression.
|
||||
|
||||
## Notes
|
||||
|
||||
- One command was refused by the permission rules: `git check-ignore -v .env.worker .env; grep
|
||||
-n "SERVICE_ROLE\|SECRET\|KEY" .env.example`. It was not rerun in another form. The
|
||||
tracked-env check used `git ls-files` instead, and the only result is `.env.example`.
|
||||
- Guardrails scanner (`--domain api,supabase`): 0 FAIL, 1 WARN, 1 MANUAL. The WARN
|
||||
(`SB-POLICY-NOT-OWNER`, `20261008110000_anatomical_sheets.sql:1177`) is in a file outside the
|
||||
diff and is filtered out. MANUAL is not reported.
|
||||
- Hermetic runs: `cargo test -p faunosteo-core --features upload --bin upload-sheet-assets
|
||||
--test upload_tool --test characterisation_serde_tolerance --test archive_rules --test
|
||||
report_rules` gave 1 + 13 + 5 + 38 + 17 passed. `--features pdf-read --test pdf_import --test
|
||||
characterisation` gave 3 + 4 passed. `upload_tool` targets an in-process loopback stub with a
|
||||
made-up key.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 15 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] check_v3_shape bypass via positional parent — crates/core/src/domain/report/read.rs:61-69, crates/core/src/domain/archive/read.rs:124-125
|
||||
- [MEDIUM] DEV stack service_role reachable from LAN (pre-existing, documented) — supabase/config.toml:73-96, docs/development/ENVIRONMENTS.md §3.6
|
||||
- [LOW] other nested v3 serde tolerance, doc comment overstates completeness — crates/core/src/domain/report/read.rs:52-54
|
||||
- [LOW] error text "or null" contradicts refusal — crates/core/src/domain/report/read.rs:66
|
||||
- [LOW] own_address accepts IPv4 multicast (blocked downstream by host_is_allowed) — crates/core/src/bin/upload_sheet_assets/main.rs:114-122
|
||||
- [LOW] own_address test vacuous without `ip` — crates/core/src/bin/upload_sheet_assets/main.rs:237-244
|
||||
- [LOW] confirm_sheet_asset_upload no object check (pre-existing, documented) — supabase/migrations/20261009100000_sheet_asset_storage.sql:34-52
|
||||
- [LOW] internal typed Report reads skip the shape check — crates/core/src/pdf/write/export.rs:117-130, crates/core/src/ops.rs:64-87
|
||||
PROOF: semgrep 74 rules on 15 files → 0 findings (0 errors); checklist 8 checks → 2 findings (+ guardrails 0 FAIL / 1 WARN out of diff, filtered); targeted review 3 focus areas → 6 findings
|
||||
```
|
||||
@@ -0,0 +1,168 @@
|
||||
# Security audit: R5.6 preparation (diff 1f8873d..3b9a395 + the unapplied switch patch)
|
||||
|
||||
- date: 2026-10-24 | branch `feature/rust-r5-pipeline-core` | mode: audit (targeted delta), fresh run
|
||||
- scope: `git diff 1f8873d..HEAD` (12 files: `fingerprint.rs`, its test and oracle, the Node differential,
|
||||
`Cargo.toml`, `mod.rs`, `rust-check.sh`, RUST.md §11.19, plan, TODO, journal, the patch file) and
|
||||
`.claude/tasks/plans/r5-6-switch.patch` read as the future change (9 files: `pipeline.rs`, `archive/read.rs`,
|
||||
`report/rebase.rs`, `sheet/loaded.rs`, `facade.rs`, `pdf/compare/lines.rs`, `pdf/write/input.rs`,
|
||||
`.prettierignore`, `tests/fingerprint_consumers.rs`)
|
||||
- owner decisions checked against: convention `sha256-rfc8785-validated-v1`; `TRUSTED_ALIASES` the only alias
|
||||
authority, bound to sheet and content; `sheet-ref.json` block descriptive, a contradiction refused; an archive's
|
||||
alias list never trusted
|
||||
- not done on purpose: no worktree, the patch was not applied (`git apply --check` only: applies cleanly). The
|
||||
auditor contract forbids mutating code, even a scratch copy; every patch claim below comes from reading.
|
||||
No service key, no network, no transfer tool, no delete, no code edited. Tree clean before and after.
|
||||
- run: `cargo test -p faunosteo-core --features fingerprint,pdf-read --locked --offline --test sheet_fingerprint`
|
||||
with `CARGO_TARGET_DIR` in the session scratchpad → 9 passed. Independent Node recomputation of the draft
|
||||
(`node -e`, no file written): sha256 of the bytes = `27c7383c…2660` (the alias), raw JCS = `a56e4a24…88a1` =
|
||||
`oracles/r5/fingerprints.json` `b_rfc8785`.
|
||||
|
||||
## Focus 1: can a forged, unknown or foreign-sheet alias make a different document count as the same?
|
||||
|
||||
No. `equivalence` (`crates/core/src/domain/sheet/fingerprint.rs:62-79`) returns `TrustedAlias` only when the two
|
||||
values are exactly the registered `(canonical, alias)` pair, either way round, under the registered `sheet_id`.
|
||||
There is no transitivity, no prefix or case folding, and `alias_convention` plays no part. Every unknown value is
|
||||
`Different`, which fails safe: rebase (data kept, items flagged), snapshot dropped, export refused.
|
||||
|
||||
At each of the patch's call sites the sheet id is compared before or alongside the equivalence, and the shared id is
|
||||
the one passed in:
|
||||
|
||||
| Site (patch line) | Sheet-id guard | Id passed to `equivalence` |
|
||||
|---|---|---|
|
||||
| `report/rebase.rs` (patch L122-126) | `same_sheet &&` | `sheet.reference.sheet_id` |
|
||||
| `archive/read.rs` `snapshot_matches` (patch L92-100) | `snapshot.sheet_id == report.sheet_id &&` | `report.sheet.sheet_id` |
|
||||
| `pdf/write/input.rs` `assert_exportable` (patch L248-258) | `sheet_id != … \|\|` | `reference.sheet_id` |
|
||||
| `pdf/compare/lines.rs` `sheet_line` (patch L212-230) | `sheet_id == … &&` | `reference.sheet_id` |
|
||||
|
||||
`check_declared` (`fingerprint.rs:90-108`) can only refuse. `LoadedSheet::from_bundle` drops the declared block
|
||||
once checked (patch L170-179) and stores no alias. An archive cannot carry an alias list: `SheetSnapshotRef` is
|
||||
`deny_unknown_fields` (`crates/core/src/domain/archive/snapshot.rs:42`), and nothing reads one.
|
||||
|
||||
What the registered pair asserts is proven. The committed `draft/document.json` hashes to the alias by bytes and to
|
||||
the canonical value through `sheet_fingerprint` (test `the_draft_gives_the_frozen_validated_value`), so both values
|
||||
are the same bytes. The report stamp and the snapshot stamp are both attacker-controlled strings that could already
|
||||
be set to the bundled value before the patch. The pair therefore gives a forger no new capability.
|
||||
|
||||
## Focus 2: does the patch loosen an archive or envelope check, or hide a real content difference?
|
||||
|
||||
`snapshot_matches` and `assert_exportable` are loosened by exactly the registered pair and nothing else. They hide no
|
||||
content difference, because the pair is one content. The loosening has one side effect on data compatibility,
|
||||
finding M1 below. It is not a forging vector.
|
||||
|
||||
## Focus 3: JCS (`canonical_json`) edge cases
|
||||
|
||||
- **Member order:** sorted by `encode_utf16()` vectors (`fingerprint.rs:136-142`), which is RFC 8785's UTF-16
|
||||
code-unit order. Astral vs BMP is correct (`U+FB33` after `U+1F600`), cross-checked against Node's `<`.
|
||||
- **Strings:** `"` and `\` escaped, U+0000-U+001F with the short forms, otherwise lowercase `\u00xx`. DEL, `/`,
|
||||
U+2028 and U+2029 stay verbatim, with no normalisation (`é` ≠ `e`+U+0301). This matches RFC 8785 §3.2.2.2 and
|
||||
`JSON.stringify`. Lone surrogates are refused at parse (`crates/core/src/domain/js.rs:42-47`), so they get no hash
|
||||
and are never altered.
|
||||
- **Numbers:** `js_number` (V8 oracle, 44,322 rows, green). Parsing uses serde_json `float_roundtrip` (workspace
|
||||
`Cargo.toml`), so it is correctly rounded like `JSON.parse`. A big integer goes `u64 → f64` with nearest-even
|
||||
rounding, which equals JS (`9007199254740993 → …992`). `JsInt` is f64-backed (`js.rs:94-125`), so there is no
|
||||
i64 precision divergence. Non-finite values cannot be reached (the parser refuses them).
|
||||
- **Injectivity:** `canonical_json` is injective on `Value` except `-0`/`0`, both printed `0`. That collision is
|
||||
spec-conformant, and the core treats the two the same.
|
||||
- **Duplicate keys:** last wins, as in `JSON.parse` (L2).
|
||||
- **Unknown keys:** the typed reading is strict (`deny_unknown_fields` on every object, `schema.rs:4`), so no
|
||||
collision can come from unknown keys (L1 is a documentation error).
|
||||
- **Defaults:** filling Zod defaults merges only documents with the same meaning.
|
||||
|
||||
Net: two documents collide only when the core itself cannot tell them apart. The same document always gives the same
|
||||
hash in this build and in an RFC 8785 implementation fed the same validated value.
|
||||
|
||||
## Focus 4: can the validation before hashing be bypassed?
|
||||
|
||||
No, not on any stamping path. `sheet_fingerprint` (`fingerprint.rs:207-221`) runs a strict parse, the strict R2 type,
|
||||
`validate_sheet_document` with zero issues (warnings included), then `to_value`, JCS and SHA-256. The patch's only
|
||||
stamping site hashes the exact text written as `document.json` (`pipeline.rs:309` and `:340`, patch L37).
|
||||
`canonical_json` is public and unvalidated, but no stamping caller uses it. The legacy TypeScript writer
|
||||
(`import:equine:legacy` until R5.7) still stamps v0 bytes, and the load check does not notice (M2).
|
||||
|
||||
## Findings
|
||||
|
||||
### M1 [MEDIUM] Envelopes with mixed stamps after the switch lose their snapshot in every pre-switch reader
|
||||
|
||||
- After the switch, an old report keeps its `27c7…` stamp on the alias path. The patch's own test asserts this:
|
||||
`a_report_on_the_historical_value_is_not_rebased_and_keeps_its_stamp`, patch L300; the code is patch L122-126.
|
||||
- The export snapshot is stamped from the loaded sheet:
|
||||
`crates/core/src/domain/archive/snapshot_rules.rs:60` → `12d7…`.
|
||||
- The patched `assert_exportable` accepts the pair and writes it (`crates/core/src/pdf/write/input.rs:204-213`
|
||||
after the patch). Every old report re-exported after the switch therefore yields `report=27c7…`,
|
||||
`sheetSnapshot.ref=12d7…`.
|
||||
- Readers that compare stamps strictly read these envelopes as `snapshot_rejected: sheet_mismatch`:
|
||||
- the frozen N-1 reader (`packages/core-wasm/tests/n-1/domain/archive/envelope.ts:97-102`);
|
||||
- today's reader (`crates/core/src/domain/archive/read.rs:116-119`), which is what a `git revert` rollback
|
||||
restores;
|
||||
- any deployed pre-switch build.
|
||||
- Effect: the report is kept, but the archived labels are lost for those readers. Plan §B.12 anticipates only the
|
||||
rebase of `12d7…` archives, not this.
|
||||
- It also makes the `TRUSTED_ALIASES` entry permanent. Removing it later (R5.7 cleanup or after) would drop these
|
||||
snapshots in the current reader too.
|
||||
- No rehearsal could catch it, because the seed was never switched (L5).
|
||||
- Fix direction, an owner decision: (a) stamp the snapshot with the report's own stamp when `equivalence` says
|
||||
`TrustedAlias`; or (b) re-stamp the report with the canonical value on the alias path, which contradicts "stamp
|
||||
kept"; or (c) accept it, and record "a registry entry is never removed" plus an N-1 test of a re-exported old
|
||||
report.
|
||||
|
||||
### M2 [MEDIUM] The declared-block check is incomplete against "a contradiction must be refused" and plan §B.4/§B.11
|
||||
|
||||
Where: patch L148-149 and L170-179 (`crates/core/src/domain/sheet/loaded.rs`), and `fingerprint.rs:90-108`.
|
||||
|
||||
- **(a) The check is opt-in.** The block is `#[serde(default)] Option<…>`, and `BundledRef` ignores unknown keys. An
|
||||
absent or misspelled block (`fingerprints`) skips every check, but plan §B.4 says a reference without a known
|
||||
convention name is refused.
|
||||
- **(b) A legacy value under v1 passes.** `check_declared` iterates only the listed aliases.
|
||||
`document_sha256: "27c7…"` with `{convention: v1, aliases: []}` passes: a sheet declaring v1 while carrying the
|
||||
v0 value.
|
||||
- **(c) No hex check.** Nothing checks the bundle's `document_sha256` as 64-hex at load. Plan §B.11 lists "a
|
||||
malformed hex value" as a tested refusal, and that refusal is absent.
|
||||
- **(d) Loose block type.** `DeclaredFingerprint` accepts unknown keys.
|
||||
|
||||
No false equivalence follows, because `equivalence` never reads the block. The residual risk is convention drift that
|
||||
goes undetected, for example `import:equine:legacy` rewriting `sheet-ref.json` without the block.
|
||||
|
||||
Fix direction: refuse when `document_sha256` is a registered alias of this sheet under a declared v1; apply
|
||||
`is_sha256_hex` to the ref; put `deny_unknown_fields` on the block type; decide whether absence is refused after the
|
||||
switch.
|
||||
|
||||
### Low findings
|
||||
|
||||
| ID | Location | Finding | Direction |
|
||||
|---|---|---|---|
|
||||
| L1 | `fingerprint.rs:6-7` | The doc says "unknown keys dropped". The reading is strict (`schema.rs:4`): an unknown key gives `FingerprintError::Shape`. The behaviour is safer than documented. | Correct the doc |
|
||||
| L2 | `js.rs:42-47`, oracle case 6 | Duplicate member names are accepted with last wins, where RFC 8785 assumes I-JSON. `{"k":1,"k":2}` and `{"k":2}` share a fingerprint. The core and `JSON.parse` read both the same; a first-wins consumer would differ. | Accepted by design; state it in the v1 definition |
|
||||
| L3 | patch L219-230 (`lines.rs:347-366` after the patch) | The alias branch of `sheet_line` drops the draft/published status that the same-sheet line prints (`lines.rs:351-358`). | Diagnostic only |
|
||||
| L4 | `fingerprint.rs:112` | `canonical_json` is public and has no validation step. No stamping caller uses it today. | Keep `sheet_fingerprint` the only stamping entry |
|
||||
| L5 | TODO.md R5.6 prep line; RUST.md §11.19; patch L268 | The evidence conflicts. TODO says "rehearsed (rust:check + check:all 14:32:05 green)"; RUST.md says four suites fail with the generator switched. The rehearsal never regenerated the seed, so the app, N-1 and report-compare suites ran on `27c7…` with the alias paths dormant. The test header cites a non-existent `r5-6-consumers.patch`. | Rerun the gates on a switched seed at the switch |
|
||||
| L6 | patch `fingerprint_consumers.rs` | No consumer-level test covers a foreign `sheet_id` carrying the registered pair (rebase, archive, export, compare). The four sites are correct by reading; only `equivalence` is unit-tested for it (`sheet_fingerprint.rs`). | Add one case per site |
|
||||
| L7 | patch L37 | `sheet_fingerprint` parses and validates before the drawings' resource bound (RUST.md §11.19). This is an operator tool on repository input. | Move it after the bound, as planned |
|
||||
| L8 | `sheet_fingerprint.rs:6`, patch L272 | New `#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. All 39 existing test crates carry the same allow, it applies to test targets only, and it is not a security suppression. | Context |
|
||||
| L9 | `rebase.rs:40` | Pre-existing, kept by the patch: equal stamps skip the rebase without comparing `sheet_id`. The bundle's `document_sha256` is never recomputed at load (wasm has no `fingerprint` feature), so the build is trusted, as with v0. | Context |
|
||||
|
||||
Checklist and tool notes:
|
||||
|
||||
- Semgrep's packs carry little Rust taint coverage; the manual review above carries this scope.
|
||||
- The guardrails scanner reports 1 WARN (`SB-POLICY-NOT-OWNER`,
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`). It is outside this delta and was filtered.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 21 files (12 diff files incl. the patch file, 7 patch target files at HEAD, js.rs + json.rs as context); semgrep on 18 (journal.md, TODO.md, RUST.md read and grepped only)
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] mixed-stamp envelopes rejected by every pre-switch reader (N-1, rollback, older builds); registry entry becomes permanent — crates/core/src/domain/archive/snapshot_rules.rs:60, patch L122-126 (report/rebase.rs), crates/core/src/pdf/write/input.rs:204-213, packages/core-wasm/tests/n-1/domain/archive/envelope.ts:97-102
|
||||
- [MEDIUM] declared-block check opt-in, legacy value under v1 accepted, no hex check, block not strict — patch L148-149 + L170-179 (domain/sheet/loaded.rs), crates/core/src/domain/sheet/fingerprint.rs:90-108
|
||||
- [LOW] doc says unknown keys dropped, reading is strict — crates/core/src/domain/sheet/fingerprint.rs:6-7
|
||||
- [LOW] duplicate member names accepted (last wins), not I-JSON — crates/core/src/domain/js.rs:42-47
|
||||
- [LOW] report-compare alias line drops draft/published status — patch L219-230 (pdf/compare/lines.rs)
|
||||
- [LOW] canonical_json public without validation (no stamping caller) — crates/core/src/domain/sheet/fingerprint.rs:112
|
||||
- [LOW] rehearsal evidence conflicts, seed never switched, header cites missing r5-6-consumers.patch — .claude/tasks/TODO.md, docs/development/RUST.md §11.19, patch L268
|
||||
- [LOW] no consumer-level foreign-sheet test of the registered pair — patch tests/fingerprint_consumers.rs
|
||||
- [LOW] fingerprint computed before the drawings bound — patch L37 (curation/draft/pipeline.rs)
|
||||
- [LOW] test-crate clippy allow, existing convention, not a security suppression — crates/core/tests/sheet_fingerprint.rs:6, patch L272
|
||||
- [LOW] pre-existing: rebase on equal sha ignores sheet_id; bundle sha never recomputed at load — crates/core/src/domain/report/rebase.rs:40
|
||||
PROOF: semgrep 157 rules on 18 files → 0 findings (0 errors); checklist 8 checks → 0 findings in scope (guardrails 0 FAIL / 1 WARN out of scope, filtered); targeted review 4 focus areas → 2 MEDIUM, 9 LOW; cargo test sheet_fingerprint 9 passed; git apply --check clean (not applied)
|
||||
```
|
||||
@@ -0,0 +1,15 @@
|
||||
# R5.6 preparation — fresh verifier (Opus) on 3b9a395, persisted by the orchestrator (verifier is read-only)
|
||||
|
||||
VERIFY — VERDICT: ECARTS(2) (criteria P1-P7 from the dispatch brief, no contract file on disk)
|
||||
|
||||
- P1 inert: MET (no consumer, seed unchanged, --check 0, 27c7…2660, wasm/app untouched, Cargo.lock unchanged)
|
||||
- P2 RFC 8785: MET (2 RFC vectors, 44,322 V8 rows, 7 Node cases reproduced; 11 of 24 RFC appendix numbers unpinned)
|
||||
- P3 validation before hashing, draft = 12d78872…74e2: MET
|
||||
- P4 registry: MET
|
||||
- P5 census 147 / 124 / 123: MET (test asserts floors only; non-alias stamp = ffff… in security.snapshot-another-sheet.pdf)
|
||||
- P6 switch patch: NOT-MET: report-compare build E0063 (BundledRef.fingerprint missing in bin/report_compare.rs); rest checked (five consumers, session.rs through rebase only, alias line in report-compare)
|
||||
- P7 docs: NOT-MET: "rehearsed green" false with the whole patch (4 curation tests fail, cli build broken); drawings-projection test not fixable by reordering; "four suites" = four tests in two suites
|
||||
- Outside criteria: switched-seed simulation adds 3 failures (fingerprint_consumers precondition ×2, pdf_import O7 oracle on migrated v2 stamps); alias line drops draft status; patch header cited a missing file; absent block accepted at load
|
||||
- FLOOR: clean
|
||||
|
||||
Orchestrator after: patch fixed (report-compare, strict block, NotCanonical + test, header); RUST.md §11.19, plan §E, TODO corrected; EVAL-031.
|
||||
@@ -0,0 +1,167 @@
|
||||
# Security audit: R5.6 fingerprint switch, activated (9e87016)
|
||||
|
||||
Mode audit, fresh and blind. Scope: `git diff 59521dc 9e87016` (31 files, all present at HEAD),
|
||||
branch `feature/rust-r5-pipeline-core`, working tree clean before and after. Contract:
|
||||
`.claude/tasks/contracts/2026-10-25-rust-r5-6-rehearsal.md` (S1-S8 + Clarification 2026-10-25) and the
|
||||
owner's activation brief. No file modified except this report; no network beyond semgrep's rule
|
||||
download; no service key, no Supabase, no transfer tool, no `pnpm format`.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 31 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] rebase-exact-equality-no-sheet-id (pre-existing, R2) — crates/core/src/domain/report/rebase.rs:43
|
||||
- [LOW] registry-and-declaration-check-public — crates/core/src/domain/sheet/fingerprint.rs:44, :103
|
||||
- [LOW] snapshot-assets-outside-fingerprint (pre-existing trust model) — crates/core/src/domain/archive/snapshot_rules.rs:79
|
||||
- [LOW] caller-built-export-input-trusted (pre-existing, documented) — crates/core/src/pdf/write/input.rs:205
|
||||
- [LOW] new-lint-allow-in-test-file (not a security suppression) — crates/core/tests/r5_6_switch.rs:8
|
||||
PROOF: semgrep 129 rules on 31 files → 0 findings; checklist 8 checks → 0 findings in scope (+1 guardrails WARN outside the diff); alias-path review 14 checks → 0 blocking, 5 LOW
|
||||
```
|
||||
|
||||
## Tooling
|
||||
|
||||
- semgrep 1.168.0, pinned packs, `--metrics=off`: 129 rules run on the 31 changed files, 0 findings,
|
||||
0 errors.
|
||||
- Guardrails scanner (`~/.claude/skills/guardrails/scripts/scan.py --domain api,supabase`): 0 FAIL,
|
||||
1 WARN, 1 MANUAL. The WARN (`SB-POLICY-NOT-OWNER`,
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, `species_select_active`) is in a file
|
||||
outside the diff: out of scope, not counted. MANUAL not reported.
|
||||
- Hermetic tests run in the main tree (no file written outside `target/`):
|
||||
- `cargo test -p faunosteo-core --features cli --locked`: `r5_6_switch` 14/14, `sheet_fingerprint` 11/11
|
||||
(incl. `nesting_is_bounded_with_a_controlled_error`), `pdf_import` 4/4, `pdf_compare` 16/16;
|
||||
lib unit `the_registered_pair_holds_one_way_for_its_sheet_only` 1/1.
|
||||
- vitest `packages/core-wasm/tests/r5-6.n-1.test.ts` 2/2 and `apps/app/tests/unit/r5-6.switch.test.ts`
|
||||
2/2 (BUILD_STAMP accepted, module current).
|
||||
- `bash scripts/import-equine.sh --check` (compare-only, read before running): legacy-import 6 files
|
||||
0 differences, draft 12 files 0 differences.
|
||||
|
||||
## Checklist (STEP 3), scoped to the diff
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Hardcoded secret / token / key / auth URL | none (added lines grepped; the hex values are public fingerprints) |
|
||||
| SQL by concatenation | none (no SQL in the diff) |
|
||||
| Unsanitised render (innerHTML, eval, raw) | none |
|
||||
| Sensitive endpoint without authz | N/A: no endpoint, RPC or policy in the diff |
|
||||
| Stack trace / internal path to the user | none new. `facade.rs:56` maps `LoadError::Fingerprint` to `invalid_sheet` with the refusal's Debug text (a recomputed hex or a serde reason about the page's own bundle), no path, no secret |
|
||||
| Secret / PII in a log | none (no log call added) |
|
||||
| Tracked `.env` / credential file | only `.env.example` tracked |
|
||||
| Guardrails scanner | run, nothing in scope (see above) |
|
||||
|
||||
Anti-gaming (STEP 4): no new `nosemgrep`, `nosec`, `eslint-disable`, `@ts-ignore` in the diff. One new
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` heads the new test file
|
||||
`crates/core/tests/r5_6_switch.rs:8`. It is the shared header of 20+ integration test files (the core's
|
||||
no-I/O and no-panic lints, which tests that read fixtures must relax), not a security-rule suppression:
|
||||
reported LOW, not blocking.
|
||||
|
||||
## Alias paths attacked (the brief)
|
||||
|
||||
1. **`equivalence` crate-private.** `fingerprint.rs:72` is `pub(crate)`. Its one non-test caller is
|
||||
`LoadedSheet::recognises_historical_stamp` (`loaded.rs:169`). The wasm crate references no
|
||||
fingerprint item (`crates/wasm/src` grep empty).
|
||||
2. **One decision point.** Every alias decision goes through `recognises_historical_stamp`
|
||||
(`loaded.rs:163-171`): `build_sheet_snapshot` (`snapshot_rules.rs:62-64`, behind `archive.snapshot`
|
||||
`ops.rs:361-363`), `rebase_report_on_sheet` (`rebase.rs:43-46`, reached by `session.rs:89` draft
|
||||
restore and `session.rs:127` import), `sheet_line` (`compare/lines.rs:350-358`). The only other
|
||||
reader of `TRUSTED_ALIASES` in `src` is the generator (`pipeline.rs:185`), which writes the
|
||||
descriptive list into `sheet-ref.json`; that list is never read back for a decision
|
||||
(`declared.aliases` is only passed to `check_declared`, `loaded.rs:199-204`).
|
||||
3. **Forging the private verified state.** `verified_fingerprint` (`loaded.rs:75`) is a private field;
|
||||
`from_bundle` is the only constructor (`loaded.rs:121-151`), `LoadedSheet` derives neither
|
||||
`Deserialize` nor `Default`, struct-update syntax is impossible outside the module, and
|
||||
`unsafe_code = "forbid"` applies to core and wasm (workspace lints, `Cargo.toml:55`, both crates
|
||||
`[lints] workspace = true`). `Clone` copies the state with the document, which the recomputation then
|
||||
re-proves. The wasm `SheetHandle` keeps its `LoadedSheet` private (`crates/wasm/src/lib.rs:27`), so a
|
||||
page cannot mutate it.
|
||||
4. **Recomputation at the decision.** The predicate requires a verified value, `verified ==
|
||||
reference.document_sha256`, the registry pair `(sheet_id, alias → verified)` in one direction, and
|
||||
`document_fingerprint(&self.document) == verified` now. A mutated pub `document` or `reference`, a
|
||||
cloned sheet re-pointed to another sheet id, a v0 sheet, or a stamp the registry does not bind all
|
||||
return false (tests `a_mutated_document_or_reference_never_keeps_the_alias`,
|
||||
`the_historical_value_on_another_sheet_id_is_not_recognised`). A bundle with another sheet id and
|
||||
the genuine document verifies but never matches the registry (entry pinned to the equine id); a
|
||||
bundle with the equine id and other content verifies to another value the registry does not bind.
|
||||
Granting the alias therefore needs the genuine content (SHA-256 preimage otherwise).
|
||||
5. **Injectivity of the fingerprinted form.** The value hashes the typed reading serialised back. Checked
|
||||
for collapses that would let two behaviours share a value: no `#[serde(skip)]` or `flatten` in
|
||||
`sheet/schema.rs`; `skip_serializing_if = "Option::is_none"` (lines 169, 176) round-trips; the
|
||||
untagged enums (`SheetView`, `Geometry`) carry distinguishing fields. The app consumes the same
|
||||
typed document, so text-only differences the reading erases are never behaviour.
|
||||
6. **v0 / v1 and direction.** No block = v0, unverified, no privilege (`check_reference` returns
|
||||
`Ok(false)`, `loaded.rs:196-198`); an unverified v0 sheet carrying the canonical value still
|
||||
rebases an old report and its direct export is refused (`SheetMismatch`, test
|
||||
`the_direct_export_on_an_unverified_v0_sheet_with_the_canonical_value_is_refused`). The registry is
|
||||
consulted only as alias → canonical, so a v0 (rollback) sheet never recognises a `v1` stamp.
|
||||
`check_declared` refuses a declared value that is a registered alias or not 64 lowercase hex
|
||||
(`fingerprint.rs:112-118`).
|
||||
7. **`null` and malformed blocks.** `present_block` (`loaded.rs:46-50`) turns an explicit `null` into a
|
||||
shape error; `DeclaredFingerprint` is `deny_unknown_fields` with required `convention` and
|
||||
`aliases` (`loaded.rs:52-57`); a string block and an unknown key are refused (tests). A misspelt key
|
||||
on the non-strict `BundledRef` only downgrades to v0: fail-closed, no privilege.
|
||||
8. **Aliases never from a report, archive or bundle.** Archives cannot carry one: `SheetSnapshotRef`
|
||||
and the snapshot are `deny_unknown_fields` (`archive/snapshot.rs:42, 59`); the archive reader keeps
|
||||
strict equality between snapshot and report stamps (`archive/read.rs:116-119`, unchanged).
|
||||
9. **Export input strict.** `assert_exportable` (`pdf/write/input.rs:204-210`) is unchanged by the diff:
|
||||
strict sheet id and stamp equality, no registry. The alias only reaches an archive through the
|
||||
core-built snapshot of a verified sheet.
|
||||
10. **`report:compare`.** The alias line prints both raw values truncated to 12 chars through `esc`
|
||||
(`compare/lines.rs:366-376`); `same_sheet` still requires the sheet id; the bin reads the seed's
|
||||
block through the same `present_block` and `from_bundle` verification (`bin/report_compare.rs:40-44,
|
||||
66`).
|
||||
11. **`canonical_json` depth.** Bounded at `MAX_CANONICAL_DEPTH = 128` with `CanonicalError::TooDeep`
|
||||
(`fingerprint.rs:132-154`), above serde_json's parse limit (127 nested containers accepted), so
|
||||
every parsed bundle fits and a value built in code beyond it errors instead of overflowing the stack.
|
||||
Tested at 128 / 129, arrays and objects.
|
||||
12. **Recomputation cost.** In the predicate the recomputation is the last conjunct, after the registry
|
||||
match: a forged or unknown stamp costs a string compare, only the registered historical stamp
|
||||
triggers one recomputation (one per export snapshot, rebase or compare line). The load recomputes
|
||||
once per `SheetHandle` when a block is present (page runtime, `runtime.ts:322`, and each import
|
||||
worker, `handler.ts:48`), O(document), 302,794 B for the equine draft. Not attacker-amplifiable:
|
||||
the bundle is build-time, a report triggers at most one recomputation per call. The 14 switch tests
|
||||
(loads, imports, exports) run in 0.81 s in a debug build.
|
||||
13. **Seed artifacts.** Only `draft/sheet-ref.json` and `draft/report-migration-map.json` change.
|
||||
`document.json` (sha256 27c7383c…), `assets/horse.svg` (0ea5202f…) and `equine_draft.sql`
|
||||
(c5a24312…) hash identically at 59521dc and HEAD. The committed block is consistent: value
|
||||
`12d78872…`, convention `v1`, alias `27c7383c…`, which is the registry entry and the old file-bytes
|
||||
hash. The migration map changes only its `documentSha256`.
|
||||
14. **Oracles, goldens, N-1.** 0 files changed under `crates/core/tests/oracles`, `crates/core/tests/pdf`,
|
||||
`apps/app/tests/fixtures`, `packages/core-wasm/tests/n-1`. The new
|
||||
`crates/core/tests/fixtures/r5-6/pre-switch/{sheet-ref,report-migration-map}.json` are byte-identical
|
||||
to the 59521dc seed files (sha256 2c0716f1…, 4139c51d…), so the redirected historical tests read
|
||||
the real pre-switch inputs and cannot mask an anatomical change. Exact-equality semantics are
|
||||
unchanged and documented as no proof of content (`fingerprint.rs:13-18`).
|
||||
|
||||
## Reported findings (LOW, non-blocking)
|
||||
|
||||
**L1. Rebase exact-equality branch has no sheet-id check** — `crates/core/src/domain/report/rebase.rs:43`.
|
||||
Pre-existing since R2, unchanged in meaning: a report of another sheet id stamped with the current
|
||||
sheet's value skips the rebase. The new alias branch does check the sheet id. Hint: add
|
||||
`report.sheet.sheet_id == sheet.reference.sheet_id` to the equality branch in a later step (behaviour
|
||||
change, owner decision).
|
||||
|
||||
**L2. Registry and declaration check stay public** — `crates/core/src/domain/sheet/fingerprint.rs:44`
|
||||
(`TRUSTED_ALIASES`), `:103` (`check_declared`). Neither grants anything by itself and no consumer misuses
|
||||
them today (integration tests and the generator need them), but a future crate consumer could rebuild an
|
||||
alias decision from the registry without the recomputation. Hint: `#[doc(hidden)]` plus a doc line, or
|
||||
move the tests in-crate and make both `pub(crate)`.
|
||||
|
||||
**L3. Archived drawings are outside the document fingerprint** —
|
||||
`crates/core/src/domain/archive/snapshot_rules.rs:79` (`snapshot_assets(sheet)`). A hand-made bundle with
|
||||
the genuine document but other SVG texts would archive them under the historical (or the canonical)
|
||||
stamp. Pre-existing trust model (the bundle is trusted for what it declares, `fingerprint.rs:15-18`); the
|
||||
alias adds no privilege the canonical stamp did not already have. Hint: bind asset hashes into the
|
||||
document or check `asset_hashes` against the texts at load, if the threat model ever includes
|
||||
non-build bundles.
|
||||
|
||||
**L4. Caller-built export input trusted for its self-consistent report and snapshot** —
|
||||
`crates/core/src/pdf/write/input.rs:205`. A page script can hand the export any matching pair; the
|
||||
registry is never consulted there and every reader decides again on import. Documented residual,
|
||||
unchanged by the diff.
|
||||
|
||||
**L5. New lint allow in a test file** — `crates/core/tests/r5_6_switch.rs:8`. Shared test-file
|
||||
convention for the core's I/O and panic lints, not a security suppression.
|
||||
|
||||
SECURITY — VERDICT: PASS
|
||||
@@ -0,0 +1,9 @@
|
||||
# R5.6 active state — fresh verifier (Opus) on 9e87016, persisted by the orchestrator (verifier is read-only)
|
||||
|
||||
VERIFY — VERDICT: ECARTS(1)
|
||||
|
||||
- S1-S6, S8 MET; A1-A5 MET (applied code = verified patch + 4 declared owner adjustments; only sheet-ref.json 03e7f6ab…623b and report-migration-map.json 0e991bc2…e2db changed, value 12d7…74e2, block = registry; document.json, horse.svg, equine_draft.sql identical; no oracle/golden/n-1 change; --check 0 differences; import:equine = Rust, legacy = TS)
|
||||
- S7 NOT-MET: contract still required an inactive branch; activation authorised by the owner inline → settled by the 2026-10-25 activation Clarification of the contract
|
||||
- Probe (direct export API): v0 + canonical → SheetMismatch; mutated document / reference / id → refused; other sheet id → refused; reference set to the alias value after load → strict equality on a caller-declared value (documented trust boundary, no alias privilege); v1 block with aliases [] still recognises (declared list descriptive)
|
||||
- Notes: dangling citations BDR-070 / RUST.md §11.22 (added after); equine_draft.sql header still names packages/anatomy (frozen bytes); scripts/import-equine.sh:3 line long; CHANGELOG added after
|
||||
- FLOOR clean
|
||||
@@ -0,0 +1,127 @@
|
||||
# Security audit: R5.6 final switch state (M1 correction)
|
||||
|
||||
- Date: 2026-10-25. Mode: audit, fresh and blind (no prior verdict used).
|
||||
- Target: worktree `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/3571c861-cd6e-4751-a25e-c26da95beb85/scratchpad/r56final`,
|
||||
detached at 8e77e37 + `.claude/tasks/plans/r5-6-switch.patch`. The worktree diff is line-identical to the committed
|
||||
patch at baff882 (index lines aside). No code under `crates/ packages/ apps/ scripts/ supabase/` differs between
|
||||
943a25a, 8e77e37 and baff882, so the patch applies to the official code state.
|
||||
- Contract: `.claude/tasks/contracts/2026-10-25-rust-r5-6-rehearsal.md` S1-S8 and the 2026-10-25 Clarification.
|
||||
- Scope: the 30 files of the diff (9 Rust sources, Cargo.toml, 7 Rust tests + 2 fixtures, 4 TS tests/helpers,
|
||||
2 scripts, package.json, .prettierignore, CLAUDE.md, the 2 switched seed files).
|
||||
|
||||
## Attack on the alias decision
|
||||
|
||||
Every Rust site that decides "same document" was enumerated (grep of `equivalence`, `TRUSTED_ALIASES`,
|
||||
`TrustedAlias`, `same_document`, `recognises_historical_stamp`, `document_sha256` across `crates/core/src`,
|
||||
`crates/wasm/src`, `apps/app/src`, `packages/core-wasm/src`).
|
||||
|
||||
| Path | Decision | Result |
|
||||
|---|---|---|
|
||||
| `ops.rs:361` `archive.snapshot` → `snapshot_rules.rs:60-68` `build_sheet_snapshot` | alias only through `recognises_historical_stamp` | holds |
|
||||
| `pdf/write/input.rs:204-210` `assert_exportable` (wasm export session, `export_pdf`, `export_from_envelope`) | strict `sheet_id` + `document_sha256` equality, unchanged from 943a25a | holds |
|
||||
| `report/rebase.rs:42-46` (reached by `draft.restore`, `archive.import`, `import_pdf`, `restore_copy`) | exact equality (pre-existing) or `recognises_historical_stamp` with same sheet id | holds |
|
||||
| `archive/read.rs:116-119` `snapshot_matches` | strict equality report vs archived snapshot, unchanged | holds (mixed alias/canonical archive loses its snapshot, tested) |
|
||||
| `pdf/compare/lines.rs:350-376` report-compare | `recognises_historical_stamp`, both raw values printed through `esc` | holds |
|
||||
| `session.rs`, `ops.rs`, `facade.rs` (one arm), `crates/wasm` | no own comparison; route to the above | holds |
|
||||
| app TypeScript | no stamp comparison (only `provider.ts:52` copies the ref) | holds |
|
||||
|
||||
`recognises_historical_stamp` (`loaded.rs:163-171`) requires, in this order: a loader-verified `v1` value, that value
|
||||
still equal to `reference.document_sha256`, the registry pair (sheet id, stamp as alias, verified value as canonical,
|
||||
one direction only, `fingerprint.rs:63-75`), and a fresh `document_fingerprint(&self.document)` equal to the verified
|
||||
value. Because the registry canonical must equal a value recomputed from the current document, the privilege is
|
||||
granted only when the loaded content hashes to the registered equine canonical (SHA-256 preimage otherwise). A
|
||||
JS caller building a `SheetHandle` from a hand-made bundle gains nothing unless its document is that content.
|
||||
|
||||
Forging the private state:
|
||||
- Struct literal and functional update `..base`: impossible outside `loaded.rs` (private `verified_fingerprint`,
|
||||
`index`, `drawings`; E0451). No other constructor than `from_bundle` (grep: test helpers all go through it).
|
||||
- `Deserialize`, `Default`, `Serialize`: not derived on `LoadedSheet`. `unsafe_code = "forbid"` workspace-wide
|
||||
(`Cargo.toml:55`), no transmute. No `Cell`/`RefCell`.
|
||||
- `Clone`: copies a consistent state; mutating the clone's pub `document` or `reference` is caught by the
|
||||
recomputation and the `verified == reference` check (tested, `r5_6_switch.rs` `a_mutated_document_or_reference_never_keeps_the_alias`).
|
||||
- `SheetHandle` (`crates/wasm/src/lib.rs:26-37`) holds the sheet privately; no `&mut` access to it from JS.
|
||||
- Even a forged `verified_fingerprint` on a v0 sheet would only grant the alias when the content is the canonical
|
||||
one: a policy deviation from S3, never a content confusion.
|
||||
|
||||
Fingerprint block handling (`loaded.rs:39-57`, `191-210`): absent → v0, unverified, no privilege; explicit `null` →
|
||||
refused (`present_block`); string, unknown key, missing key → refused (`deny_unknown_fields`, required fields);
|
||||
convention other than `v1`, non 64-lowercase-hex value, value equal to a registered alias, unregistered alias in the
|
||||
list, value not recomputed from the document (including a document with integrity errors) → refused at load,
|
||||
closed `invalid_sheet` kind, detail never rendered (`useSheet.ts:60`). Duplicate JSON key: last wins, a trailing
|
||||
`null` is refused. A misspelt block key at `ref` level (no `deny_unknown_fields` on `BundledRef`) loads as v0:
|
||||
fail-closed (less privilege, old reports rebase onto identical content).
|
||||
|
||||
v1 with warnings (`fingerprint.rs:253-256`): errors refuse, warnings do not. This only widens which documents
|
||||
get a value; it grants no equivalence, and a warning-only document with another content gets another value
|
||||
(tested, it rebases old reports).
|
||||
|
||||
## Recursion and cost
|
||||
|
||||
- `canonical_json` (`fingerprint.rs:121-193`): containers refused at depth ≥ 128 with `CanonicalError::TooDeep`,
|
||||
tested at the bound (`sheet_fingerprint.rs:280-292`). Its only non-test caller is `document_fingerprint`, whose
|
||||
only caller is `loaded.rs` on the loaded sheet's document (the build-time bundle). Attacker data (archives,
|
||||
drafts, PDFs, rescue copies) never reaches `canonical_json`; archived snapshot finding depth stays capped at 8
|
||||
by `assert_snapshot` before shape. The new load-time `validate_sheet_document` + `serde_json::to_value` on the
|
||||
bundle are bounded by serde_json's parse limit (127 containers).
|
||||
- Recomputation cost: native release, a test with about 3 fingerprints, 7 loads and 1 import runs in 84-87 ms
|
||||
against 32 ms for 3 loads with none, so one fingerprint of the 296 KB document is in the low milliseconds.
|
||||
The recomputation runs only when the stamp already equals the registered alias (short-circuit order,
|
||||
`loaded.rs:167-170`; identical stamps never pay it), at most once per `archive.snapshot` (once per export,
|
||||
`export.ts:146`), per import (one candidate, `import.rs:24-30`) or per draft restore, plus one at each v1
|
||||
load. Import worker timeout 20 s (`controller.ts:19`). Repeated calls only cost the user's own tab: no DoS.
|
||||
|
||||
## Findings
|
||||
|
||||
No CRITICAL, no HIGH, no new security suppression.
|
||||
|
||||
- [LOW] hardening — `crates/core/src/domain/sheet/fingerprint.rs:63` — `equivalence` is a crate-public function
|
||||
that answers `TrustedAlias` from the registry alone, with no verified state and no recomputation. Its only
|
||||
caller today is `recognises_historical_stamp` (`loaded.rs:169`); a future caller (the Supabase sheet provider
|
||||
of Step 13, a new export path) could reopen the flaw of the previous round. Hint: `pub(crate)` or private to
|
||||
`loaded`, or a census test that pins its callers.
|
||||
- [LOW] by design, context — `crates/core/src/pdf/write/input.rs:204-210` — the export session accepts any
|
||||
self-consistent caller-supplied (report, snapshot) pair: a script in the page can write an archive whose report
|
||||
and snapshot both carry the alias with a hand-built snapshot. No equivalence is granted there (every reader
|
||||
re-decides against its loaded sheet), the alias adds nothing a forged canonical stamp could not already do,
|
||||
and this is the pre-switch behaviour for any stamp.
|
||||
- [LOW] context, pre-existing — `crates/core/src/domain/report/rebase.rs:43` — the exact-equality branch keeps a
|
||||
report of another sheet id unrebased when it carries the loaded value; unchanged since R2 (the new alias branch
|
||||
does check the sheet id). Equal SHA-256 means equal content, so no confusion.
|
||||
- [LOW] context — `crates/core/tests/r5_6_switch.rs:8` — new `#![allow(clippy::disallowed_methods,
|
||||
clippy::unwrap_used, clippy::expect_used)]`, test-only, the header 40 existing test files carry; not a
|
||||
security-tool suppression.
|
||||
- Out of diff, filtered: guardrails `WARN SB-POLICY-NOT-OWNER`
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177` (`species_select_active`, catalog read
|
||||
policy), pre-existing.
|
||||
|
||||
Non-security note: `r5_6_switch.rs:133-135` writes `<repo>/target/tmp/r5-6/` even when `CARGO_TARGET_DIR` points
|
||||
elsewhere (git-ignored, `.gitignore:65`).
|
||||
|
||||
## Proof
|
||||
|
||||
- semgrep 1.168.0, `p/security-audit`, `p/secrets`, `p/owasp-top-ten`, `--metrics=off`: 163 rules on the 30 diff
|
||||
files, 0 results, 0 errors. Caveat: these packs carry few Rust rules; the Rust review above is manual.
|
||||
`p/*` content is fetched at run time, not frozen by the CLI pin.
|
||||
- Checklist, 9 checks on the added lines and the tree: hardcoded secret/auth URL (0), SQL by concatenation (0),
|
||||
unsanitised render/eval (0), sensitive endpoint without authz (n/a, no endpoint in the diff), internal error to
|
||||
the user (0: `invalid_sheet` detail not rendered), secret/PII in a log (0), tracked `.env` (0, only
|
||||
`.env.example`), guardrails `--domain api,supabase` (0 FAIL, 1 WARN out of diff, filtered), new suppressions (0
|
||||
security; 1 test lint allow, LOW).
|
||||
- Hermetic tests run in the worktree, `CARGO_TARGET_DIR=<worktree>/target`, release, `--features cli`:
|
||||
`r5_6_switch` + `sheet_fingerprint` 25 passed; `pdf_import`, `pdf_compare`, `archive_rules`, `report_rules`,
|
||||
`sheet_contract` 78 passed.
|
||||
- Seed check: `draft/document.json` still hashes (bytes) to the registered alias 27c7…, `document.json`,
|
||||
`horse.svg`, `equine_draft.sql` unchanged; only `sheet-ref.json` and the map's `sheet.documentSha256` changed.
|
||||
- Nothing edited in either tree except this report; no network beyond the semgrep rule download, no Supabase.
|
||||
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 30 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] hardening: crate-public registry-only `equivalence` — crates/core/src/domain/sheet/fingerprint.rs:63
|
||||
- [LOW] by design: export accepts caller-supplied consistent snapshot — crates/core/src/pdf/write/input.rs:204
|
||||
- [LOW] pre-existing: exact-equality rebase branch without sheet id — crates/core/src/domain/report/rebase.rs:43
|
||||
- [LOW] test lint allow (crate convention) — crates/core/tests/r5_6_switch.rs:8
|
||||
PROOF: semgrep 163 rules on 30 files → 0 findings; checklist 9 checks → 4 findings (LOW), 1 out-of-diff WARN filtered
|
||||
SECURITY — VERDICT: PASS
|
||||
@@ -0,0 +1,14 @@
|
||||
# R5.6 final rehearsal — fresh verifier (Opus), persisted by the orchestrator (verifier is read-only)
|
||||
|
||||
Contract `.claude/tasks/contracts/2026-10-25-rust-r5-6-rehearsal.md` (S1-S8 + 2026-10-25 Clarification); final worktree r56final (8e77e37 + patch at baff882).
|
||||
|
||||
VERIFY — VERDICT: CONFORME (8/8)
|
||||
|
||||
- S1 MET: decision in snapshot_rules.rs build_sheet_snapshot via recognises_historical_stamp (private verified value, reference equality, registry one-way, recompute now); 9 scenarios tested (r5_6_switch 14/14, vitest N-1 2/2, app 2/2); rollback: unpatched build's report-compare reads the re-export, snapshot kept
|
||||
- S2 MET: input.rs, archive/read.rs, session.rs, ops.rs, crates/wasm, n-1 unchanged vs 943a25a; equality strict
|
||||
- S3 MET: probe through facade::load_sheet: v0 canonical = no privilege; null/string/{}/array/missing/extra/v0/upper/alias/63-hex/other/forged/self-alias/content-changed/foreign-id refused; warnings-only v1 loads verified
|
||||
- S4 MET: only the two seed files differ; document.json, horse.svg, equine_draft.sql byte-identical; no oracle/golden/ledger/n-1 in patch; original contexts + post-switch twins
|
||||
- S5 MET (12-hex prefixes, the tool's style); S6 MET; S7 MET (main tree clean, seed 27c7, patch applies); S8 MET (rust:check rc 0, 1475 passed; check:all 01:53:59 cited)
|
||||
- Direct-API probe: v1 + old report → 27c7/27c7 exported; v0 canonical → SheetMismatch; mutated document / reference value / reference id → refused; other sheet id → refused; forged stamp → refused; canonical_json 128 OK, 129 TooDeep, 5000 TooDeep without overflow
|
||||
- Observations: stale comments fingerprint.rs (6, 15, 246-247); RUST.md §11.20 superseded mechanism (superseded by §11.21); null block → invalid_shape in facade (refusal either way)
|
||||
- FLOOR clean
|
||||
@@ -0,0 +1,142 @@
|
||||
# Security audit: R5.6 fingerprint switch rehearsal (worktree 943a25a + `r5-6-switch.patch`, seed switched)
|
||||
|
||||
- date: 2026-10-25 | mode: audit (delta), fresh run | contract `.claude/tasks/contracts/2026-10-25-rust-r5-6-rehearsal.md` (S1-S8)
|
||||
- scope: `git -C <worktree> diff HEAD`, 28 files (9 Rust sources, Cargo.toml, 8 Rust test files and 2 fixtures,
|
||||
5 TypeScript test files, package.json, rust-check.sh, .prettierignore, the two switched seed files). Worktree:
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/3571c861-cd6e-4751-a25e-c26da95beb85/scratchpad/r56wt`.
|
||||
`git apply --check -R` of the stored patch is clean there, so the worktree is exactly the patch.
|
||||
- isolation (S7) checked: main tree at 6ade917, seed `sheet-ref.json` still `27c7…`, `fingerprint.rs` still
|
||||
"PREPARED, NOT ACTIVE", the main-tree diff since 943a25a touches only `.claude/` and RUST.md.
|
||||
- untouched, checked file by file: `domain/archive/*` (incl. `read.rs`), `domain/session.rs`, `domain/report/read.rs`,
|
||||
`pdf/read/*`, `pdf/import.rs`, `pdf/json_js.rs`, `packages/core-wasm/tests/n-1`, `crates/core/tests/oracles`,
|
||||
`apps/app/tests/fixtures`, `tests/pdf/LEDGER.tsv`, `Cargo.lock`, `deny.toml`. Seed `document.json`, `horse.svg`,
|
||||
`equine_draft.sql` identical to HEAD; `sha256sum document.json` = `27c7383c…2660` (the registered alias).
|
||||
- hermetic runs in the worktree (`CARGO_TARGET_DIR=<worktree>/target`): `--features cli` r5_6_switch 8,
|
||||
sheet_fingerprint 10, pdf_compare 16, pdf_import 4; `--features curation` curation_draft 20, curation_review 12;
|
||||
vitest app `r5-6.switch` 2, core-wasm `r5-6.n-1` 1 (resolves the worktree's `packages/core-wasm`). All green.
|
||||
- not done: no service key, no Supabase, no transfer tool, no delete, no code edited, no `pnpm format`. The only
|
||||
network access was semgrep fetching its three pinned packs (metrics off).
|
||||
|
||||
## Focus answers
|
||||
|
||||
**Forged, unknown, foreign-sheet or unverified stamp counted as the same document?** Not in the readers.
|
||||
`equivalence` (`fingerprint.rs:63-75`) admits only the registry pair, one way (alias stamped, canonical current),
|
||||
bound to the sheet id. A forged 64-hex stamp is `Different` and rebases (`r5_6_switch.rs:169-179`); a case-changed
|
||||
one is refused by the v3 contract; a foreign sheet id matches no entry and fails the sheet-id checks
|
||||
(`rebase.rs:43-48`, `input.rs:212`). `check_reference` (`loaded.rs:161-180`) recomputes `v1` from the same typed
|
||||
document the index is built from, so `fingerprint_verified` implies the loaded content is the registered content.
|
||||
The pair itself is bound to one content: sha256 of the bytes is the alias, `v1` of the same bytes is the canonical
|
||||
value (curation_draft `the_document_fingerprint_is_v1_and_its_bytes_keep_the_historical_alias`, plus my sha256sum).
|
||||
Rebase and report:compare grant the alias only on a verified sheet (`rebase.rs:43-48`, `lines.rs:356`). The export
|
||||
check is the exception: see M1.
|
||||
|
||||
**Does M1 write a snapshot that is not the trusted document, or loosen archive checks?** Through the application,
|
||||
no: `export.ts:146` always builds the snapshot with `archive.snapshot` from the loaded sheet, and an old stamp only
|
||||
survives to export when that sheet was verified (otherwise rebase moved the report first). `archive/read.rs:116-119`
|
||||
is unchanged and strict; the mixed envelope is still rejected (`r5_6_switch.rs:185-188`). At the core boundary the
|
||||
binding is not enforced (M1).
|
||||
|
||||
**Can a v0 reference gain alias privilege?** At load, no: no block means `fingerprint_verified = false`, and a v0
|
||||
reference carrying the canonical value still rebases an old report (`r5_6_switch.rs:255-263`). At export, yes,
|
||||
if a caller pairs an old report with a snapshot cut from such a sheet (M1).
|
||||
|
||||
**Do the declared test rules hide a content difference?** No. Each rule is targeted and fails loudly on anything
|
||||
else. `curation_draft` `r5_6_rule` asserts the pre-switch value is the alias, moves only that value, adds the block,
|
||||
then compares whole bytes through the generator's formatter; the pre-switch copies are pinned to the frozen R5.1
|
||||
ledger (sha and length). `pdf_compare` `r5_6_rule` rewrites only the "matches the bundled sheet" line of
|
||||
non-migrated blocks into the alias line; a `!!` line or any other drift still mismatches, and the alias count is
|
||||
pinned (77). O7, O10, the R2 Zod oracle, the R5.1 ledger and `seed.golden` read the pre-switch files; their
|
||||
post-switch twins (`o7_after_r5_6_…`, `o10_on_the_switched_seed_…`, `the committed migration map after R5.6`)
|
||||
allow the stamp alone to move. `migratedBeforeR56` changes only `report.sheet.documentSha256`; observations
|
||||
still go through the switched map, so a mapping change would break the byte goldens.
|
||||
|
||||
**Wasm sha2 and the load-time fingerprint: cost or DoS?** The hasher is reached only from `loaded.rs:175` and the
|
||||
pipeline (`pipeline.rs:174`). Archive snapshots and imported files are never fingerprinted, and the bundle is
|
||||
build-time content. The cost is one validation, one JCS and one SHA-256 of a 296 KB document per sheet handle
|
||||
(page plus each import worker): the 8-test switch suite with its dozen loads runs in 0.46 s in a debug build.
|
||||
`sha2 0.11.1` was already locked; the worker keeps its own dependency. No attacker-driven DoS. Two LOWs below
|
||||
(L4, L5).
|
||||
|
||||
## Findings
|
||||
|
||||
### M1 [MEDIUM] The export grants the alias on the snapshot's word, with no verified sheet behind it
|
||||
|
||||
`crates/core/src/pdf/write/input.rs:205-213` (`assert_exportable`) and `:238-248` (`keep_the_report_stamp`)
|
||||
call `equivalence` with the snapshot's own `reference.document_sha256` as "current". The export never sees a
|
||||
`LoadedSheet`, so `fingerprint_verified` is not consulted. The snapshot document is a lean cut and cannot be
|
||||
re-hashed. As a result:
|
||||
|
||||
- `export_from_envelope` (`pdf/write/export.rs:117-136`, cargo only) took a mixed envelope (report `27c7…`,
|
||||
snapshot `12d7…`, any lean document) as `sheet_mismatch` before the patch. It now accepts it and writes the
|
||||
snapshot under `27c7…`, a self-consistent envelope that the strict reader keeps. The writer normalises what
|
||||
the reader rejects.
|
||||
- A snapshot cut from an unverified v0 reference that carries the canonical value, exported with an old report,
|
||||
gets the alias that rebase and report:compare deny for the same sheet.
|
||||
- `packages/core-wasm/tests/r5-6.n-1.test.ts:46-49` feeds a hand-edited snapshot and the core accepts it, which
|
||||
shows there is no provenance check.
|
||||
- RUST.md §11.20 (main tree, lines 3538-3545) says M1 "needs the loaded sheet's `fingerprint_verified`". At this
|
||||
site it does not. Contract S1's "with the loaded sheet's `v1` value recomputed … at load" holds only as an
|
||||
application-flow precondition.
|
||||
|
||||
Not exploitable today. The only shipped caller builds the snapshot from the verified loaded sheet. Archives are
|
||||
unsigned, so anyone can already write a `27c7/27c7` envelope around arbitrary content: no capability is gained.
|
||||
This is a trust-boundary gap, not a vulnerability.
|
||||
Hint: make the M1 decision where verification is known. `build_sheet_snapshot` / `archive.snapshot` can stamp the
|
||||
report's value when `sheet.fingerprint_verified` and the pair is `TrustedAlias`, and `assert_exportable` goes back
|
||||
to strict equality (that also keeps `export_from_envelope` strict). This needs an S2 scope amendment
|
||||
(`snapshot_rules.rs`). Otherwise, state the precondition at `input.rs:232-237` and correct RUST.md §11.20.
|
||||
|
||||
### L1 [LOW] `"fingerprint": null` reads as no block (v0)
|
||||
`crates/core/src/domain/sheet/loaded.rs:38-39`: `#[serde(default)] Option<…>` maps `null` to `None`. A present but
|
||||
null block is not refused, against S3's "a present block is strict". It fails closed: no privilege, old reports
|
||||
rebase. Hint: refuse an explicit `null`, for example with a custom deserializer.
|
||||
|
||||
### L2 [LOW] `BundledRef` accepts unknown keys, so a misspelt block loads as v0
|
||||
`loaded.rs:30-40`: `DeclaredFingerprint` is strict, but its parent is not. A typo such as `fingerprints` silently
|
||||
yields an unverified sheet. This fails closed, and the struct shape predates the patch. Hint:
|
||||
`deny_unknown_fields` on `BundledRef`, if every bundle producer (vite plugin, report_compare `SeedRef`, tests)
|
||||
emits only known keys.
|
||||
|
||||
### L3 [LOW] Verification state and verified value are public mutable fields
|
||||
`loaded.rs:53` (`reference`) and `:66` (`fingerprint_verified`) are `pub`. Crate code holding `&mut LoadedSheet`
|
||||
could change one without the other. No such writer exists today. Hint: private fields with getters.
|
||||
|
||||
### L4 [LOW] A warning the drafts allow makes a v1 bundle unloadable and the generator fail
|
||||
`fingerprint.rs:228-231` refuses any issue, warnings included. `validate.rs:366-380` (`annotation.no_findings`,
|
||||
documented "allowed in a draft") and `:440-455` (`finding_zone.unreferenced`) are warnings. Since the patch,
|
||||
`loaded.rs:175-179` turns that into `invalid_sheet` at load, and `pipeline.rs:174-175` refuses to render the draft.
|
||||
The behaviour fails closed and follows the frozen `v1` rule, but a content-only curation step can now take the
|
||||
sheet out of the application. Hint: owner decision. Either keep it (and document it next to the curation workflow)
|
||||
or have `v1` refuse errors only, which would need a new convention version.
|
||||
|
||||
### L5 [LOW] `canonical_json` recursion has no depth bound
|
||||
`fingerprint.rs:129-170` recurses on the typed document. Only the build-time bundle reaches it today. Hint: bound
|
||||
the depth, or reuse the import prescan caps, before any user- or database-supplied sheet goes through
|
||||
`from_bundle` (Step 13).
|
||||
|
||||
### L6 [LOW, context] New test file carries the standard clippy allow
|
||||
`crates/core/tests/r5_6_switch.rs:8` has `#![allow(clippy::disallowed_methods, clippy::unwrap_used, …)]`, the
|
||||
same header as all 40 integration tests. It is test-only, not shipped, and not a security-rule suppression (no
|
||||
`nosemgrep`, `nosec` or `eslint-disable` in the diff). Not blocking.
|
||||
|
||||
Out of scope, filtered: guardrails WARN `SB-POLICY-NOT-OWNER` at `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`
|
||||
(not in the diff, unchanged).
|
||||
|
||||
## Verdict block
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 28 files
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] M1 export grants the trusted alias on the snapshot's unverified reference; export_from_envelope now normalises a mixed envelope; RUST.md §11.20 claims a fingerprint_verified guard absent at this site — crates/core/src/pdf/write/input.rs:205-248, crates/core/src/pdf/write/export.rs:117-136
|
||||
- [LOW] L1 "fingerprint": null read as v0 instead of refused — crates/core/src/domain/sheet/loaded.rs:38-39
|
||||
- [LOW] L2 BundledRef accepts unknown keys, misspelt block loads as v0 — crates/core/src/domain/sheet/loaded.rs:30-40
|
||||
- [LOW] L3 reference and fingerprint_verified public mutable fields — crates/core/src/domain/sheet/loaded.rs:53,66
|
||||
- [LOW] L4 draft-allowed warning makes a v1 bundle unloadable and the generator fail — crates/core/src/domain/sheet/fingerprint.rs:228-231, crates/core/src/domain/sheet/loaded.rs:175-179
|
||||
- [LOW] L5 canonical_json recursion unbounded (trusted input only today) — crates/core/src/domain/sheet/fingerprint.rs:129-170
|
||||
- [LOW] L6 standard clippy allow in new test file, not a security suppression — crates/core/tests/r5_6_switch.rs:8
|
||||
PROOF: semgrep 163 rules on 28 files → 0 findings (0 errors); checklist 9 checks (secrets, SQL concat, unsafe render, authz, error exposure, secret in log, tracked .env, guardrails 0 FAIL / 1 WARN out of diff, new suppressions) → 1 finding (L6); targeted review 5 focus areas → 1 MEDIUM, 5 LOW; worktree runs: cargo r5_6_switch 8, sheet_fingerprint 10, pdf_compare 16, pdf_import 4, curation_draft 20, curation_review 12, vitest r5-6.switch 2, r5-6.n-1 1, all green
|
||||
```
|
||||
@@ -0,0 +1,13 @@
|
||||
# R5.6 rehearsal — fresh verifier (Opus), persisted by the orchestrator (verifier is read-only)
|
||||
|
||||
Contract `.claude/tasks/contracts/2026-10-25-rust-r5-6-rehearsal.md`, patch at 82e2e43, worktree switched.
|
||||
|
||||
VERIFY — VERDICT: ECARTS(3)
|
||||
|
||||
- S1 NOT-MET: export path (`pdf/write/input.rs` assert_exportable :205, keep_the_report_stamp :238) grants the pair from the snapshot's claimed value, never reads `fingerprint_verified`. Probe: v0 sheet-ref (no block) + demo report 27c7 + core build_sheet_snapshot → export succeeds 27c7/27c7 (pre-patch: sheet_mismatch). App store flow cannot reach it (rebase checks the flag). Rest of S1 met (one direction, same sheet, registry only, new report canonical, 9 scenarios tested).
|
||||
- S2 MET (8 Rust files in scope, archive/read.rs and session.rs unchanged, archive check strict, invalid_sheet / sheet_mismatch kept, no n-1 change)
|
||||
- S3 NOT-MET: load conforms; v0 snapshot gets alias privilege at export (same root cause)
|
||||
- S4 MET (2 files differ, value + block v1, document/horse.svg/SQL identical, no oracle/golden changed, original contexts + post-switch twins, O10 77 rows pinned)
|
||||
- S5 MET, S6 MET, S7 MET (main tree inert, patch applies on HEAD)
|
||||
- S8 NOT-MET: alias-security gate fails for the export consumer; everything else green (rust:check RC 0, suites listed, vitest 41/41, rollback report-compare reads the re-export with snapshot kept)
|
||||
- FLOOR clean. Notes: stale `pnpm import:equine` comments (scripts/import-equine.sh:8, CLAUDE.md); verifier overwrote its own scratch probe folder (session scratch only).
|
||||
@@ -0,0 +1,48 @@
|
||||
# R5.7 pre-merge checks (2026-10-25)
|
||||
|
||||
Owner brief « Validation R5.7 et dernière vérification pré-fusion ». State checked: `feature/rust-r5-pipeline-core` at
|
||||
`27bf59c` (code and tests); the commit that adds this file changes docs and `.claude/` records only (SUPABASE.md layout,
|
||||
the `.bin/tsx` wording in RUST.md §11.23 and in the contract clarification, TODO, registries).
|
||||
|
||||
## Decision C6
|
||||
|
||||
Owner accepted `tsx` in `pnpm-lock.yaml` as vite's optional peer only. Contract C6 amended with a Clarification section.
|
||||
`pnpm why tsx -r`: `tsx` 4.23.15 reached only as `vite` 7.3.6's peer (`peerDependenciesMeta.tsx.optional: true`) through the
|
||||
devDependencies of `apps/app`, `packages/core-wasm`, `packages/supabase`; no importer, no manifest, no script; snapshot
|
||||
`optional: true`. No lockfile hand edit, no override.
|
||||
|
||||
## Checks run on 27bf59c
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `pnpm install --frozen-lockfile` | rc 0, "Lockfile is up to date", lockfile sha256 unchanged |
|
||||
| `pnpm dedupe --check` | rc 0 |
|
||||
| `pnpm check:all` (starts with `scripts/rust-check.sh` = `pnpm rust:check`) | green 13:37:52 → 13:41:41: fmt, clippy, 177 cargo result lines all ok (1548 passed, 0 failed, 3 ignored, none a ledger target), ts-rs drift none, cargo-deny + advisories ok, wasm, typecheck, vitest app 446 / core-wasm 60 / supabase 40, lint, build, app image, worker sandbox tests, worker runtime image |
|
||||
| `pnpm import:equine --check` | legacy-import 6 files, draft 12 files, 0 differences (app-main under the declared R3 rule) |
|
||||
| `pnpm report:compare` on `apps/app/tests/fixtures/*.pdf` and `fixtures/pdf` | rc 0 |
|
||||
| `pnpm audit` | 57 advisories (2 critical, 33 high, 21 moderate, 1 low), every version identical on `develop`: none introduced by R5. Runtime-reachable: zod 3.25.76 (GHSA-mrhx-c7xc-m3w5, patched only in zod 4). Rest dev tooling (pnpm 10.13.1, tinypool via vitest, vite 7.3.6 dev server, happy-dom, brace-expansion, js-yaml, nanoid, postcss, source-map-js). Separate chore, owner decision (majors excluded by the brief) |
|
||||
| Suites named by the brief | RFC 8785 and aliases `sheet_fingerprint` 11, `r5_6_switch` 14, app `r5-6.switch`; reports v2/v3 `report_rules` 17, `report_contract` 4, `archive_rules` 38, `characterisation` 4; N-1 `n-1.reader`, `n-1.scanner`; PDF `pdf_*` (import 4, compare 16, session 6, …), app `pdf.roundtrip`, core-wasm `pdf-import`; rebase and history: `archive_rules`, the 612 leaves (`curation_leaves` 8, app `migration.leaves`); seed and sheet `curation_*`, `sheet_rules` 25, `sheet_conformance` 3; census `r4.census` 8, `r5.census` 10 |
|
||||
| Moved fixtures | sha256 `e96b0882…729c`, `ba79698c…fde7`, equal to 756b7ef |
|
||||
| Old packages required nowhere | `git grep` over scripts, docker, `.gitea`, Dockerfile, compose, configs, tests, package sources: comments and census bans only |
|
||||
|
||||
## Ledger
|
||||
|
||||
407 rows: 256 ported, 149 replaced, 2 deleted (JS aliasing check with no Rust meaning; TS generator of
|
||||
`fixtures/horse-slice.json`, read by 4 Rust suites). Replaced rows active: every cargo target a non-ignored `#[test]`
|
||||
(census now refuses `#[ignore]` after `#[test]`, cd69f0e), every cited oracle read by a test.
|
||||
|
||||
## Gates
|
||||
|
||||
- Fresh verifier: CONFORME, C1 to C10 MET (C6 under the clarification), 14 scratch reintroduction probes all fail the
|
||||
census. Precisions it raised, fixed in the docs commit: this file did not yet exist; SUPABASE.md §1 still drew
|
||||
`packages/domain`; `.bin/tsx` is linked under the three packages (not the root).
|
||||
- Fresh security audit: PASS (`2026-10-25-rust-r5-7-security-final.md`). MEDIUM pre-existing `pnpm audit`; LOW census
|
||||
regex accepts `#[ignore]` before `#[test]` or `#[cfg_attr(…, ignore)]` (no instance; left for the owner, a fix would
|
||||
invalidate this `check:all`); LOW clippy allow header; LOW pre-existing SQL policy warning outside the diff.
|
||||
|
||||
## Git
|
||||
|
||||
Branch clean, equal to `origin/feature/rust-r5-pipeline-core`; `develop` = `origin/develop` = `9c0117a`, ancestor of the
|
||||
branch (76 commits ahead, 0 behind): `git merge-tree` clean, a `--no-ff` merge has no conflict. No secret pattern in the
|
||||
added lines; no env file, key or binary added; largest added files are the R5.1 oracles `numbers.json` (1.8 MB) and
|
||||
`pipeline.json` (1.4 MB). `main` 605ee7c and `release/0.2.0` f8d9425 untouched.
|
||||
@@ -0,0 +1,187 @@
|
||||
# Security gate, final: Rust R5.7 (deletion of packages/anatomy and packages/domain)
|
||||
|
||||
Date 2026-10-25. Branch `feature/rust-r5-pipeline-core`. Scope `756b7ef..27bf59c`
|
||||
(152 paths: 8 added, 32 modified, 2 renamed, 110 deleted; 2 055 insertions, 17 542 deletions,
|
||||
7 commits `0da7d3a`..`27bf59c`). Fresh auditor; the previous report
|
||||
`2026-10-25-rust-r5-7-security.md` was read only for the status check the brief asked for.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 42 files (the existing files of the diff; 110 deleted paths inspected through git)
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] pnpm-audit (pre-existing, no version touched by the diff) — pnpm-lock.yaml — 57 advisories on unchanged versions: 2 critical (tinypool 1.1.1 via vitest, dev), 33 high (pnpm 10.13.1 itself x17, brace-expansion, js-yaml, nanoid, postcss, source-map-js, happy-dom, zod 3.25.76), 21 moderate, 1 low
|
||||
- [LOW] census-regex-gap — apps/app/tests/unit/r5.census.test.ts:300 — `#[ignore]` placed before `#[test]`, or `#[cfg_attr(…, ignore)]`, still passes the ledger-target check (no instance in the tree)
|
||||
- [LOW] new-suppression (clippy allow, not a security-tool suppression) — crates/core/tests/curation_leaves.rs:6 — identical header in all 41 files under crates/core/tests
|
||||
- [LOW] SB-POLICY-NOT-OWNER (guardrails WARN, pre-existing, file not in the diff) — supabase/migrations/20261008110000_anatomical_sheets.sql:1177
|
||||
PROOF: semgrep 158 rules (725 loaded, 3 packs) on 42 files → 0 findings, 0 errors, 0 skipped; checklist 8 checks → 0 findings; guardrails scan.py api,supabase → 0 FAIL, 1 WARN (outside diff), 1 MANUAL (not reported); anti-gaming 1 new clippy allow (LOW), 0 removed; cargo deny check advisories → ok; pnpm audit → 57 advisories, 0 introduced; ledger 407 rows / 566 cargo targets → 566 active #[test], 0 #[ignore]
|
||||
```
|
||||
|
||||
## Semgrep
|
||||
|
||||
`semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten
|
||||
--metrics=off --quiet --json <42 paths>`: 0 results, 0 errors, 0 skipped paths, stderr empty.
|
||||
The verbose rerun reports 725 rules loaded, 158 applicable to the languages present, 42 files.
|
||||
|
||||
## Checklist (8 checks, 0 findings)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Hardcoded secret, token, key, auth URL in added lines | None. Hits are variable names in prose (`SUPABASE_SERVICE_ROLE_KEY` in CLAUDE.md and the previous audit), no value. URLs in added lines are all LEDGER.tsv case descriptions (`https://x.test/…`, `http://localhost:5173`, `https://faunosteo.com`, the partner credit, `https://zenquality.fr`): test and product data, no credential. No IP, no `/home/`, no `/Users/`. |
|
||||
| SQL by concatenation | No new SQL. `seed.rs` keeps `sql_text` quote doubling and the `$equine_doc$` dollar tag with the refusal at line 94 (`DOCUMENT_DOLLAR_TAG`). The diff on that file is a 3-line doc comment. |
|
||||
| Unsanitized render (innerHTML, eval, raw) | None. |
|
||||
| Sensitive endpoint without authz | No endpoint, policy or RPC in the diff. |
|
||||
| Stack trace / internal path to the user | None. |
|
||||
| Secret or PII to a log | None. |
|
||||
| Tracked `.env` or credential file | Only `.env.example` is tracked (`git ls-files`); `.gitignore` and `.dockerignore` exclude `.env`, `.env.*`, the gateway htpasswd and the Mailpit password. No binary in the diff (`git diff --numstat` shows no `-` row); the largest added file is `LEDGER.tsv` (173 KB, UTF-8 text). |
|
||||
| Guardrails scanner (api,supabase, 1219 files) | 0 FAIL. 1 WARN `SB-POLICY-NOT-OWNER` on `species_select_active` (pre-existing, file not in the diff). 1 MANUAL `SB-RLS-LIVE`, not reported here. |
|
||||
|
||||
## Anti-gaming
|
||||
|
||||
One new suppression-like marker in the added lines:
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` at
|
||||
`crates/core/tests/curation_leaves.rs:6`. It is a clippy allow, not a semgrep, bandit or
|
||||
ESLint-security suppression, and every one of the 41 files under `crates/core/tests/` carries the
|
||||
same single header (`disallowed_methods` forbids I/O and clock in the library; integration tests
|
||||
read fixtures). LOW, not blocking. No `nosemgrep`, `nosec`, `eslint-disable`, `@ts-ignore`,
|
||||
`@ts-expect-error`, `.skip` or `#[ignore]` added; no suppression removed.
|
||||
|
||||
## Supply chain
|
||||
|
||||
### Lockfile
|
||||
|
||||
`pnpm-lock.yaml`: 1 insertion, 69 deletions.
|
||||
|
||||
- Removed: importers `packages/anatomy` and `packages/domain`; the `packages:` and `snapshots:`
|
||||
entries of `@xmldom/xmldom@0.9.12`; the `deprecated:` metadata line of `eslint@9.39.4`
|
||||
(registry metadata, same version).
|
||||
- Added: the single line `optional: true` on the `tsx@4.23.15` snapshot. `tsx` appears in no
|
||||
manifest (`git grep '"tsx"' -- '*/package.json'` → none); it stays resolved only as vite's
|
||||
optional peer, per the owner decision C6 (`27bf59c`).
|
||||
- No new package, no version change, no `overrides`, `patchedDependencies`, `onlyBuiltDependencies`
|
||||
change (`pnpm-workspace.yaml` untouched).
|
||||
|
||||
### cargo deny
|
||||
|
||||
`. scripts/rust-env.sh` (read first: appends `~/.cargo/bin` to PATH when no cargo is found,
|
||||
installs nothing) then `cargo deny check advisories` (cargo-deny 0.20.2, `deny.toml`
|
||||
`yanked = "deny"`, `ignore = []`): `advisories ok`. No build, no `target/` access.
|
||||
|
||||
### pnpm audit (read-only)
|
||||
|
||||
`pnpm audit --json` with the local pnpm 10.13.1: 57 advisories, `{critical: 2, high: 33,
|
||||
moderate: 21, low: 1}`. Every advisory sits on a package version the lockfile diff did not
|
||||
change, so none is introduced by this scope; they are reported, not gated, and belong to a
|
||||
separate `chore/*`. Breakdown by reachable surface:
|
||||
|
||||
| Package (version) | Advisories | Surface |
|
||||
|---|---|---|
|
||||
| pnpm 10.13.1 (root devDependency, `packageManager`, Dockerfile `corepack prepare pnpm@10.13.1`) | 17 high + 8 moderate, patched in 10.26 … 10.34.5 (lifecycle-script bypass, lockfile integrity bypass, env-var command injection, manifest identity spoof running attacker lifecycle, secrets exfiltration through `pnpm-workspace.yaml` proxy placeholders, several path traversals) | developer machine and the Docker builder stage; the Docker build runs `pnpm install --frozen-lockfile` on a repository-controlled lockfile. The CLAUDE.md pin rationale ("10.x line stays pinned") is compatible with 10.34.5. |
|
||||
| tinypool 1.1.1 (vitest) | 2 critical (prototype-pollution RCE in worker options) | dev only (test runner) |
|
||||
| brace-expansion 1.1.14 / 5.0.6 (eslint, typescript-eslint) | 8 high + 2 moderate (DoS) | dev only |
|
||||
| js-yaml 4.3.0 (eslint) | 2 high (CPU DoS) | dev only |
|
||||
| vitest 3.2.7, @vitest/mocker | 3 moderate | dev only |
|
||||
| happy-dom 20.10.6 | 1 high | dev only |
|
||||
| vite 7.3.6 | 1 high, 4 moderate, 1 low (launch-editor, `server.fs.allow`) | dev server and build |
|
||||
| postcss 8.5.14, nanoid 3.3.12, source-map-js 1.2.1 (vite, tailwind) | 2 high + 1 moderate, 2 high, 1 high | build time |
|
||||
| zod 3.25.76 (apps/app, core-wasm, supabase runtime dependency) | 1 high "memory exhaustion on oversized constrained collections", patched in zod 4.6.6 only | runtime, browser-side form validation of the user's own input: self-DoS of one's own tab, no server impact |
|
||||
|
||||
## No lost validation
|
||||
|
||||
`crates/core/tests/curation/LEDGER.tsv`: 407 rows (144 anatomy, 263 domain), dispositions
|
||||
`ported` 256, `replaced` 149, `deleted` 2, `kept` 0. Targets: 566 `cargo …::<test>` parts (154
|
||||
distinct functions), 8 `vitest …` parts, 25 `import-equine --check`.
|
||||
|
||||
Independent resolution (script in the auditor's scratchpad, not the census): for each cargo target
|
||||
every `fn <name>(` under the named binary's files was located and its whole contiguous attribute
|
||||
block inspected, in any attribute order. Result: 566/566 resolve to a function carrying `#[test]`
|
||||
with no `ignore` token in the block; 0 unresolved, 0 non-test, 0 ignored. The 8 vitest files
|
||||
exist. The one `#[ignore]` in the crate (`pdf_serialize.rs:811`, release-only timing) predates
|
||||
the diff (`babe7b7`, 2026-10-18, present at `756b7ef`) and is not a ledger target.
|
||||
|
||||
| Control named in the brief | Ledger rows | Rust test (active `#[test]`) | Implementation |
|
||||
|---|---|---|---|
|
||||
| Sheet integrity rules | 69 (40 ported, 29 replaced) | `sheet_rules::{view_main_count, view_id_duplicate, view_parent_missing, view_parent_cycle_*, view_unreachable, view_asset_missing_*, annotation_id_duplicate_across_views, …}`, `sheet_rules::the_curated_draft_validates_with_no_issue`, `sheet_rules::the_horse_slice_has_no_issue` | `crates/core/src/domain/sheet` |
|
||||
| Changeset refusals | 174 (120 ported, 53 replaced, 1 deleted: purity, see below) | `curation_draft::the_synthetic_cases_match_the_typescript_or_are_refused_alike` + oracle `oracles/r5/curation-synthetic.json`, plus the per-case ported tests in `curation_draft.rs` | `crates/core/src/curation/draft/{apply,ops,targets,fields}.rs` |
|
||||
| Reserved ids | 4 (3 ported, 1 replaced) | `curation_draft::a_changeset_reusing_the_reserved_asset_change_id_is_refused`, `report_rules::report_v3_schema_defaults_and_refusals` | `curation/draft`, `domain/report` |
|
||||
| Seed dollar-quote guard | 1 ported | `curation_draft::a_document_holding_the_dollar_quote_tag_is_refused_by_the_seed` (`curation_draft.rs:1075`, asserts the error names `$equine_doc$`) | `curation/draft/seed.rs:13,94` |
|
||||
| SVG sanitising | 24 (10 ported, 12 replaced, 2 deleted: not sanitiser cases, matched on wording) | `svg_sanitiser::{every_hostile_case_gives_the_typescript_verdict_or_a_declared_one, every_generated_case_…, the_product_svgs_come_out_untouched, a_rejection_names_a_closed_reason}` iterating the 42 `fixtures/svg/hostile/*.svg` against `oracles/r3/sanitiser.json` (452 KB); `sheet_shape::refuses_what_zod_refuses` for the `javascript:` reference | `crates/core/src/anatomy/svg/{prescan,sanitize,refs}.rs` |
|
||||
|
||||
The two `deleted` rows carry no control: "applyChangeset never mutates its inputs" is a JavaScript
|
||||
aliasing check with no Rust meaning (`apply_changeset(baseline: &Value, …)`, `apply.rs:210`);
|
||||
"horse slice JSON fixture equals the TypeScript lean input" was the TS generator of
|
||||
`fixtures/horse-slice.json`, now the source file, read by `sheet_index`, `sheet_rules` and
|
||||
`sheet_shape`.
|
||||
|
||||
Declared functional loss, confirmed as not a security loss: the xmldom re-cleaning of a legacy SVG
|
||||
(`packages/anatomy/src/curation/assets.ts`, FILES.tsv row `replaced`) is not ported; the pipeline
|
||||
reads the committed `draft/assets/horse.svg` and the application loads every drawing through
|
||||
`anatomy::svg::read_svg` regardless of origin.
|
||||
|
||||
The four `crates/core/src` changes in the diff (`anatomy/mod.rs`, `curation/mod.rs`,
|
||||
`domain/mod.rs`, `curation/draft/seed.rs`) are doc comments only (11 insertions, 2 deletions,
|
||||
no code line).
|
||||
|
||||
## Dockerfile and build
|
||||
|
||||
`Dockerfile`: the two `COPY packages/{domain,anatomy}/package.json` lines are removed; the
|
||||
builder stage copies exactly the four manifests the lockfile has importers for (`.`, `apps/app`,
|
||||
`packages/supabase`, `packages/core-wasm`) before `pnpm install --frozen-lockfile`, then
|
||||
`COPY . .`. The folders `packages/anatomy`, `packages/domain` and `packages/pdf` still exist on
|
||||
disk with a git-ignored `node_modules/` only (0 tracked files each) and `.dockerignore` excludes
|
||||
`**/node_modules/`, so nothing of them enters the image. `crates/worker/Dockerfile`,
|
||||
`docker/rust-builder.Dockerfile`, `scripts/`, `.gitea/workflows`, tsconfig, vite and vitest
|
||||
configs: no reference to the deleted paths (`git grep` over tracked non-doc files returns only the
|
||||
ESLint ban list, the two census tests and Rust `//!` provenance comments).
|
||||
|
||||
`package.json`: `import:equine:legacy` removed; `packages/supabase/package.json`:
|
||||
`assets:upload:legacy` removed with its script `scripts/upload-sheet-assets.ts` (the Rust
|
||||
`upload-sheet-assets` binary remains the operator path). `apps/app/eslint.config.js` adds
|
||||
`@faunosteo/anatomy` and `@xmldom/xmldom` to the `no-restricted-imports` ban.
|
||||
|
||||
## Census tests cannot pass vacuously
|
||||
|
||||
`r5.census.test.ts` asserts positives before negatives: the manifest list must contain the four
|
||||
remaining packages; the lockfile importer set must equal the manifest set exactly;
|
||||
`packages.length > 100` and a `zod@` key; more than 100 scanned sources containing the
|
||||
`@faunosteo/core-wasm` specifier; the Dockerfile `COPY` list must contain the two package
|
||||
manifests and each must exist; FILES.tsv > 80 rows with every counterpart present (112 rows:
|
||||
55 ported, 48 replaced, 9 deleted, all package tooling or test helpers); LEDGER.tsv exactly 407
|
||||
rows (144 + 263); every oracle path present; every cargo target resolved to a `#[test]` fn; the
|
||||
two moved fixtures pinned by SHA-256 (verified: `e96b0882…b729c` and `ba79698c…fde7` match
|
||||
`sha256sum`). `r4.census.test.ts` extends the bans and checks the importer split finds
|
||||
`['.', 'apps/app', 'packages/core-wasm']` before asserting absences.
|
||||
|
||||
Residual gap (LOW): the ledger regex at `r5.census.test.ts:300-302` anchors on `#[test]` and
|
||||
then refuses attributes starting with `ignore`. Probed with Node against the exact regex:
|
||||
|
||||
| Input | Census |
|
||||
|---|---|
|
||||
| `#[test]` / `fn foo(` | accepted |
|
||||
| `#[test]` / `#[ignore]` / `fn foo(` | refused (correct) |
|
||||
| `#[test]` / `#[ignore = "r"]` / `fn foo(` | refused (correct) |
|
||||
| `#[ignore]` / `#[test]` / `fn foo(` | accepted (bypass: rustc accepts both orders, `cargo test` skips it) |
|
||||
| `#[test]` / `#[cfg_attr(debug_assertions, ignore)]` / `fn foo(` | accepted (bypass) |
|
||||
| `fn foo(` alone, or `#[test] fn bar` then `fn foo` | refused (correct) |
|
||||
|
||||
No ledger target is in either shape today (independent order-agnostic check above). Hint: scan the
|
||||
whole attribute block above `fn <test>(` and refuse any `ignore` token in it, or assert
|
||||
`cargo test -- --list` output in `check-all.sh`.
|
||||
|
||||
## Status of the previous audit's two LOW findings
|
||||
|
||||
| Previous finding | Status at 27bf59c |
|
||||
|---|---|
|
||||
| LOW clippy allow header, `curation_leaves.rs:6` | Unchanged. Same header as the 41 sibling test files; still LOW, still not a security-tool suppression. |
|
||||
| LOW census accepted any `fn <name>(` | Fixed. `475a41d` requires `#[test]` with only attributes or doc lines between (`#\[test\]\s*(?:(?:#\[[^\]]*\]\|///[^\n]*)\s*)*fn`), `cd69f0e` adds `(?!ignore)` on those attributes. Residual gap above (attribute order, `cfg_attr`), LOW, no instance in the tree. |
|
||||
|
||||
## Not done (out of scope or forbidden)
|
||||
|
||||
No command executed from the repository other than semgrep, the guardrails scanner, `pnpm audit`
|
||||
(read-only, registry query), `cargo deny check advisories` (no build) and read-only git, grep,
|
||||
sha256sum. No `cargo clean`, `pnpm install`, `pnpm format`, no test suite run (a `pnpm check:all`
|
||||
ran concurrently in the tree). The runtime behaviour of the ported Rust tests is attested by that
|
||||
gate, not by this audit; this audit attests they exist, carry `#[test]`, are not ignored, and map
|
||||
each ledger row.
|
||||
@@ -0,0 +1,149 @@
|
||||
# Security gate: Rust R5.7 (deletion of packages/anatomy and packages/domain)
|
||||
|
||||
Date 2026-10-25. Branch `feature/rust-r5-pipeline-core`. Scope `756b7ef..4e0a6c0`
|
||||
(149 paths in the diff: 110 deletions, 2 renames, 37 modified or added files).
|
||||
Fresh auditor, no prior verdict consulted.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: PASS
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 39 files (the existing files of the diff; 110 deleted paths inspected through git)
|
||||
BLOCKING:
|
||||
(none)
|
||||
REPORTED (non-blocking):
|
||||
- [LOW] new-suppression (clippy allow, not a security-tool suppression) — crates/core/tests/curation_leaves.rs:6
|
||||
- [LOW] census-weakness (ledger target resolved by `fn <name>(` presence, not `#[test]`) — apps/app/tests/unit/r5.census.test.ts:103
|
||||
PROOF: semgrep 158 rules (725 loaded, 3 packs) on 39 files → 0 findings; checklist 8 checks → 0 findings; guardrails scan.py api,supabase → 0 FAIL, 1 WARN outside the diff (filtered), 1 MANUAL (not reported); anti-gaming 1 new allow (LOW)
|
||||
```
|
||||
|
||||
## Semgrep
|
||||
|
||||
`semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten
|
||||
--metrics=off --quiet --json <39 paths>`: 0 results, 0 errors, 0 skipped. The verbose
|
||||
rerun confirms 725 rules loaded from the 3 packs, 158 applicable to the languages present
|
||||
(TypeScript, Rust, Dockerfile, JSON, YAML, Markdown, shell), 39 files scanned.
|
||||
|
||||
## Checklist (8 checks, 0 findings)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Hardcoded secret, token, key, auth URL in added lines | None. The only hit is the variable name `SUPABASE_SERVICE_ROLE_KEY` in a CLAUDE.md command description, no value. |
|
||||
| SQL by concatenation | No new SQL. `crates/core/src/curation/draft/seed.rs` (doc comment only in this diff) keeps `sql_text` quote doubling and `$equine_doc$` dollar quoting; the deleted `importer.guards.test.ts` case "refuses a document that contains the dollar-quote tag" is ported as `a_document_holding_the_dollar_quote_tag_is_refused_by_the_seed` (curation_draft.rs). |
|
||||
| Unsanitized render (innerHTML, eval, raw) | None in the diff. |
|
||||
| Sensitive endpoint without authz | No endpoint, policy or RPC in the diff. |
|
||||
| Stack trace / internal path to the user | None. |
|
||||
| Secret or PII to a log | None. |
|
||||
| Tracked `.env` or credential file | Only `.env.example` is tracked; `.dockerignore` excludes `.env`, `.env.*`, the gateway htpasswd and Mailpit password. |
|
||||
| Guardrails scanner (`~/.claude/skills/guardrails/scripts/scan.py`, domains api,supabase) | 0 FAIL. 1 WARN `SB-POLICY-NOT-OWNER` on `supabase/migrations/20261008110000_anatomical_sheets.sql:1177` (`species_select_active`): pre-existing, file not in the diff, filtered per gate rule. 1 MANUAL `SB-RLS-LIVE`, not reported here. |
|
||||
|
||||
Local paths (`/home/`, `/Users/`), user names, e-mail addresses and private IP ranges:
|
||||
none in the added lines. The URL `https://www.instagram.com/paws.iin.harmony` present in the
|
||||
moved fixtures is the product's existing partner credit (`apps/app/src/pdf/branding.ts`,
|
||||
`components/brand/Footer.tsx`), product data, not personal data introduced by this change.
|
||||
|
||||
## Anti-gaming
|
||||
|
||||
One new suppression-like marker in the added lines:
|
||||
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` at
|
||||
`crates/core/tests/curation_leaves.rs:6`. It is a clippy allow, not a semgrep, bandit or
|
||||
ESLint-security suppression, and it is the identical header the 40 sibling integration tests
|
||||
under `crates/core/tests/` already carry (`disallowed_methods` forbids I/O and clock in the
|
||||
core library; integration tests read fixtures from disk). Reported LOW, not blocking. No
|
||||
removed suppression, no `nosemgrep`, `nosec`, `eslint-disable`, `@ts-ignore`, `.skip` or
|
||||
`#[ignore]` added.
|
||||
|
||||
## Focus (1): controls still needed by active paths
|
||||
|
||||
Pre-state references to the deleted code in production paths (`apps/app/src`,
|
||||
`packages/core-wasm/src`, `packages/supabase/src`, `Dockerfile`, `docker/`, vite and tsconfig
|
||||
files, `scripts/`, `.gitea/`): exactly two, both removed by the diff.
|
||||
|
||||
- `Dockerfile:39-40`: `COPY packages/domain/package.json` and `COPY packages/anatomy/package.json`.
|
||||
- `packages/supabase/package.json:21`: script `assets:upload:legacy` → `node scripts/upload-sheet-assets.ts`.
|
||||
|
||||
Post-state: zero references in those paths (the `scripts/r4-ledger.mjs` hits are `.tsx`
|
||||
file names). `pnpm-workspace.yaml` globs `packages/*`; the lockfile importers are now exactly
|
||||
`.`, `apps/app`, `packages/core-wasm`, `packages/supabase`, and the Dockerfile builder stage
|
||||
copies exactly those four manifests before `pnpm install --frozen-lockfile`, then `COPY . .`.
|
||||
The image build is coherent with the lockfile.
|
||||
|
||||
Each control the brief names lives in the Rust core and is untouched (the four `crates/core/src`
|
||||
changes are doc comments only):
|
||||
|
||||
| Control | Where it runs now | Present at 4e0a6c0 |
|
||||
|---|---|---|
|
||||
| PDF import (byte scan, capped inflation, layered read) | `crates/core/src/pdf/{read,import.rs}` | yes |
|
||||
| SVG sanitising (caps, allowlist, reference graph) | `crates/core/src/anatomy/svg/{prescan,sanitize}.rs` | yes |
|
||||
| Sheet integrity rules | `crates/core/src/domain/sheet` | yes |
|
||||
| Seed SQL generation (quoting, dollar-tag refusal) | `crates/core/src/curation/draft/seed.rs` | yes |
|
||||
| Upload tool (local-only URL, private-bucket handshake, no overwrite, key never printed) | `crates/core/src/curation/upload.rs`, `crates/core/src/bin/upload_sheet_assets`, `scripts/upload-sheet-assets.sh` (`env -u SUPABASE_SERVICE_ROLE_KEY cargo build`, then exec) | yes |
|
||||
| Archive readers | `crates/core/src/domain/archive` | yes |
|
||||
| N-1 Zod reader (frozen copy of the deleted domain) | `packages/core-wasm/tests/n-1/domain` | yes |
|
||||
|
||||
Declared functional loss, not a security loss: the xmldom-based re-cleaning of a legacy SVG
|
||||
(`packages/anatomy/src/curation/assets.ts`) is not ported; the pipeline reads the committed
|
||||
cleaned `draft/assets/horse.svg` bytes and verifies them through the core reader
|
||||
(`FILES.tsv` row, BDR-068). Every drawing the application loads still goes through
|
||||
`anatomy::svg::read_svg` at sheet load regardless of its origin.
|
||||
|
||||
## Focus (2): lockfile and supply chain
|
||||
|
||||
`pnpm-lock.yaml`: 1 insertion, 69 deletions.
|
||||
|
||||
- Removed: importers `packages/anatomy` and `packages/domain`; package and snapshot entries of
|
||||
`@xmldom/xmldom@0.9.12`; the `deprecated:` metadata line of `eslint@9.39.4` (registry
|
||||
metadata refreshed at install, same version).
|
||||
- Added: `optional: true` on the `tsx@4.23.15` snapshot. `tsx` is declared in no manifest
|
||||
(`git grep tsx -- '*/package.json'` → none) and stays resolved only as the optional peer of
|
||||
vite/vitest.
|
||||
- No new package, no version change on any package.
|
||||
|
||||
## Focus (3): secrets, local paths, personal data
|
||||
|
||||
See the checklist rows above: none in the 20,666-line diff's added lines. The `.claude/tasks`
|
||||
contract and plan files added by the diff were in the semgrep scope and the grep; clean.
|
||||
|
||||
## Focus (4): the new tests
|
||||
|
||||
New or extended Rust tests (`curation_leaves.rs` new, 263 lines; additions in `curation_draft`,
|
||||
`curation_format`, `curation_review`, `anatomy_compose`, `report_contract`, `archive_rules`,
|
||||
`archive_support`, `report_rules`): fixture access is read-only through `common::repo_path` /
|
||||
`repo_file` (`CARGO_MANIFEST_DIR/../../<relative>`) and the frozen oracles; `files_under` walks
|
||||
the two seed folders with `read_dir` only. No network, no `std::process`, no `std::env`
|
||||
read, no write, no delete in the added code.
|
||||
|
||||
The one test in these files that spawns a process and writes (`import_equine`, `out_fixtures`,
|
||||
`out_refuses_every_unsafe_destination_and_writes_nothing`, `curation_draft.rs` ~768-816) is
|
||||
pre-existing (before the diff's second hunk at line 963): it writes fixed fixtures under
|
||||
`CARGO_TARGET_TMPDIR/import-equine-out` only and asserts the binary refuses every unsafe
|
||||
`--out` destination (symlink, file, non-empty folder, `..`, absent parent) with exit code 2
|
||||
and writes nothing. Not changed by this diff.
|
||||
|
||||
`archive_rules.rs` and `archive_support/mod.rs` only repoint the two moved fixtures
|
||||
(`crates/core/tests/fixtures/report-v2.{full,minimal}.json`, R100 renames) and a doc comment.
|
||||
|
||||
## Focus (5): census tests not vacuous
|
||||
|
||||
`apps/app/tests/unit/r5.census.test.ts` (new) asserts positive facts before the negatives:
|
||||
the manifest list must contain the four remaining packages; the lockfile importer set must
|
||||
equal the manifest set exactly; `packages.length > 100` and a `zod@` key must be found; the
|
||||
scanned source set must exceed 100 files and must contain the `@faunosteo/core-wasm`
|
||||
specifier; the Dockerfile `COPY <manifest>` list must contain the two package manifests and
|
||||
each copied manifest must exist; `FILES.tsv` must hold more than 80 rows with each counterpart
|
||||
present; `LEDGER.tsv` must hold exactly 407 rows (144 anatomy, 263 domain), each oracle path
|
||||
present and each `cargo <bin>::<test>` target resolved to a `fn <test>(` in the right file;
|
||||
the two moved fixtures are pinned by SHA-256; ten oracle and N-1 paths must exist.
|
||||
`r4.census.test.ts` gains `@faunosteo/anatomy` and `@xmldom/xmldom` in the bans and replaces
|
||||
the "xmldom only under packages/anatomy" assertion by an importer-list assertion with its own
|
||||
non-vacuity check (`arrayContaining(['.', 'apps/app', 'packages/core-wasm'])`).
|
||||
|
||||
Weakness reported LOW: `cargoTargetFile('faunosteo_core')` accepts a `fn <name>(` anywhere
|
||||
under `crates/core/src`, so a ledger row pointing at a non-test function would pass. The ban
|
||||
regexes and the specifier scanner cover `from`, `import()` and `require()` forms and relative
|
||||
`anatomy/src` / `domain/src` paths.
|
||||
|
||||
## Not done (out of scope or forbidden)
|
||||
|
||||
No command executed from the repository other than semgrep, the guardrails scanner and
|
||||
read-only git / grep. No test suite run, no `cargo` invocation, no Supabase access. The
|
||||
runtime behaviour of the ported tests is attested by the dev's own gate, not by this audit.
|
||||
@@ -0,0 +1,20 @@
|
||||
# R5.7 fresh verifier, 2026-10-25 (diff 756b7ef..4e0a6c0, contract `.claude/tasks/contracts/2026-10-25-rust-r5-7-cleanup.md`)
|
||||
|
||||
VERDICT: ECARTS(2). C1-C5, C7, C9, C10 MET; C6 NOT-MET; C8 NOT-MET.
|
||||
|
||||
- C6: `tsx@4.23.15` still in pnpm-lock.yaml (`packages:`, `snapshots:` with the diff's one added line `optional: true`,
|
||||
peer suffix of vite@7.3.6 / vitest@3.2.7), installed under node_modules/.pnpm; declared by no manifest; documented
|
||||
(RUST.md §11.23, CHANGELOG, r5.census asserts `optional: true`); `pnpm dedupe --check` clean. Contract says "tsx gone",
|
||||
no Clarification → owner decision (accept residue, or a lockfile change).
|
||||
- C8: RUST.md §11.23 abbreviated report-v2.minimal.json digest as `ba79698c…dfe7` (actual `…7399c33cfde7`). Fixed after
|
||||
the pass (RUST.md, FILES.tsv note): `…fde7`.
|
||||
- Evidence of the pass: cargo test core cli,upload 382 passed 1 ignored (pre-existing release-only timing); 154/154 ledger
|
||||
targets in `--list`; vitest app 446, core-wasm 60, supabase 40, census 18/18; typecheck, lint rc 0; fmt, clippy ×2, deny
|
||||
ok; `pnpm import:equine --check` 0 differences; fixtures sha256 equal before/after; no diff under oracles, goldens,
|
||||
conformance, pdf ledger, app fixtures, supabase/seed, migrations, n-1, types; 18 seed hashes equal the §11.23 table;
|
||||
every "ported at R5.7" ledger row checked TS-vs-Rust; random sample of other rows consistent.
|
||||
- FLOOR: 42 `DELETED_TEST` hits, all files of the two deleted packages; covered by the contract's deletion clause and
|
||||
LEDGER.tsv (verifier reading; orchestrator may require an explicit Clarification).
|
||||
- Notes: leftover scratch worktrees (ts-repro, prev-59521dc, r56final, r56wt) and git-ignored
|
||||
`packages/{anatomy,domain}/node_modules` = owner housekeeping.
|
||||
- After the pass (not re-verified): security LOW "census accepts any `fn`" fixed, ledger cargo targets must be `#[test]` fns.
|
||||
@@ -0,0 +1,204 @@
|
||||
# R5 census: sheet fingerprint `documentSha256`
|
||||
|
||||
Date 2026-10-22, branch feature/rust-r5-pipeline-core (cut at 9c0117a). Read-only census by an analyzer agent, saved by the orchestrator. Facts only, no recommendation.
|
||||
Names covered: `documentSha256` (camelCase contracts), `document_sha256` (sheet-ref.json, bundle payload, Rust fields), `SheetRef` / `SheetSnapshotRef` / `BundledRef`, `sheet-ref.json`.
|
||||
|
||||
## Summary
|
||||
|
||||
1. There is exactly one producer: `sheetRef()` in packages/anatomy/src/curation/pipeline.ts:159-165. It computes sha256 over `formatJson(document)`, which is the Prettier 3.8.3 json printer run over `JSON.stringify(v,null,2)`.
|
||||
2. Nothing in Rust, SQL, apps/app, packages/supabase or scripts computes a document fingerprint. In crates/core, `sha2` is a dev-dependency only.
|
||||
3. SQL has no fingerprint column, CHECK, trigger or RPC. Measured on PGlite (PG 17.5), `jsonb::text` of the draft hashes to eaf891f4…, not 27c7383c…. jsonb reorders keys (length, then bytes) and keeps numeric scale (`1.50`, `1.0`).
|
||||
4. One real value exists: 27c7383c445c73dc2001ebf4794a3ea63d416910bbbaa0d7848dd720de7c2660. It equals sha256(draft/document.json, 302,794 B), has been unchanged since 0a17e39 (2026-10-10), and belongs to sheetId 503dc301-… (uuid derived from `faunosteo:sheet:equine:v1`) with versionId null.
|
||||
5. That value sits in sheet-ref.json, in report-migration-map.json, in 8 app JSON fixtures, in 6 Rust oracle files, and inside the attachments of 99 of the 216 committed PDFs.
|
||||
6. The rebase decides on the fingerprint alone (rebase.rs:40). The envelope invariant (read.rs:116-119) and the export (input.rs:206-207) compare sheetId and fingerprint. versionId is never compared anywhere.
|
||||
7. Mismatch on import or draft restore: the report is rebased. `report.sheet` takes the loaded ref, unknown zones and findings get `unresolved: true`, labels are filled only where absent, nothing is deleted. Messages: `home.import.rebased`, `editor.draft_rebased`.
|
||||
8. Same fingerprint with a different sheetId: no rebase, so the report keeps the foreign sheetId and a later export is refused with `sheet_mismatch`, shown as the generic `editor.export_failed`.
|
||||
9. Both v2 imports and new reports get their stamp from the copies the pipeline writes (map.sheet, sheet-ref.json). No runtime check verifies that those copies agree or match document.json.
|
||||
10. `BundledRef.document_sha256` is loaded without any format check (loaded.rs:29-36). Report and snapshot refs are checked against `^[0-9a-f]{64}$`.
|
||||
11. The hashed bytes depend on Prettier's layout. The file differs from `JSON.stringify(v,null,2)+LF` from line 17 on, because Prettier collapses short arrays. It also differs from the parsed form: Zod or serde fill defaults, 173,136 vs 245,636 UTF-16 units compact.
|
||||
12. Draft data: 208 numbers (29 non-integer, none negative, no exponent, no `-0`, all already in ECMAScript `Number#toString` form). No escape of any kind, no astral character, no empty container. Every key is ASCII, and key order is schema order, not sorted.
|
||||
13. The docs disagree with each other. BDR-045 / Annex C says "existing fingerprints untouched". RUST.md:2092 and TODO:307 still say "will change once".
|
||||
14. Doc errors: RUST_ARCHITECTURE_REVIEW.md:145 says "lean document" (the fingerprint covers the full document) and cites the deleted `packages/pdf export.ts:88`. STEP6:89 says "every run changes" the fingerprint, but the pipeline is deterministic and the golden test pins it.
|
||||
15. Open: which form gets canonicalised (input file or parsed), what "compatibility" with 27c7… means concretely, who computes the fingerprint for database-served sheets (Step 13), and whether versionId or sheetId should take part in the rebase decision.
|
||||
|
||||
## 1. Producers (who computes a fingerprint)
|
||||
|
||||
| Layer | Site | What it does |
|
||||
|---|---|---|
|
||||
| TS pipeline | packages/anatomy/src/curation/pipeline.ts:158-165 `sheetRef()` | `documentSha256: sha256Hex(documentJson)`, with `sheetId: draft.sheetId` and `versionId: null` |
|
||||
| | pipeline.ts:185-186 | `documentJson = await formatJson(draft.document)`. The same text is written as `document.json` (:203) and embedded verbatim in the seed SQL (:216-221, seed.ts:21-25 and :73-80 dollar-quoted `::jsonb`) |
|
||||
| | pipeline.ts:205-211 | writes `sheet-ref.json` `{sheet_id, version_id, document_sha256, schema_version, species_id}` |
|
||||
| | pipeline.ts:187-194 → curation/migration-map.ts:24 | the same `ref` object becomes `report-migration-map.json` `.sheet` |
|
||||
| | packages/anatomy/src/legacy/ids.ts:15-17 | `sha256Hex = createHash('sha256').update(data).digest('hex')` (node:crypto, UTF-8 of the string) |
|
||||
| | packages/anatomy/src/format.ts:9-26 | `formatJson(v) = prettier.format(JSON.stringify(v,null,2), {parser:'json', printWidth:80, …})`. Prettier `^3.4.2` (packages/anatomy/package.json:31), locked 3.8.3 (pnpm-lock.yaml:1894) |
|
||||
| TS legacy importer | legacy/artifacts.ts:17 | writes legacy-import/document.json through `formatJson`. It produces no fingerprint and no sheet-ref |
|
||||
| TS app | none | apps/app/vite/sheet-bundle.ts:44-75 `JSON.parse`s document.json and re-serialises the payload (`JSON.stringify(payload)`, :99). The browser never holds the hashed bytes. provider.ts:46-57 copies `ref` |
|
||||
| TS packages | none | packages/domain, packages/supabase and packages/core-wasm/src have no document hash |
|
||||
| Rust | none | crates/core: `sha2` is a `[dev-dependencies]` entry (Cargo.toml:41-44, oracle digests). crates/worker: sha2 hashes WebP output only (job.rs:8, :210, :264). crates/wasm: nothing |
|
||||
| SQL | none | no column, CHECK, trigger or RPC hashes `content`. sha256 columns exist only for assets: `anatomical_sheet_assets.sha256` (20261008110000_anatomical_sheets.sql:154-164) and `public.assets.sha256` (20261013100000_files_foundation.sql:74-75) |
|
||||
| Scripts | none | scripts/browser/lib/core-module.mjs:45 and scripts/r4-j2-kit.mjs:51 pass `sheet-ref.json` through. wasm-stamp.mjs and r4-afm-data.mjs hash other inputs |
|
||||
|
||||
On the SQL side, `anatomical_sheet_versions.content` is `jsonb` (anatomical_sheets.sql:105-108, 2 MiB cap). Its schema version is pinned by CHECK (`document_schema_version = 2`, 20261010100000_sheet_document_schema_v2.sql:11-12). Measured on PGlite 0.3.16 (PostgreSQL 17.5, in-memory), one run:
|
||||
|
||||
```
|
||||
jsonb_text_sha: eaf891f46d3149ecc8734c51ccca57a5459c9c1e5d81776445b25bba351d5d54 (file: 27c7383c…)
|
||||
jsonb_text_bytes: 184105 stable re-cast: true
|
||||
head: {"name": {"en": "Horse", "fr": "Cheval"}, "views": [{"id": "main", "kind": "main", ...
|
||||
numeric samples [1.50, 1e2, -0, 743.0451049804688, 1.0] -> [1.50, 100, 0, 743.0451049804688, 1.0]
|
||||
key order {"fr","en","schema_version","id"} -> {"en","fr","id","schema_version"} duplicate key -> last wins
|
||||
```
|
||||
|
||||
jsonb::text orders keys by length then bytes and uses `", "` / `": "` separators. It keeps numeric scale (`1.50`, `1.0`), unlike JS and JCS. `1e2` becomes `100` and `-0` becomes `0`.
|
||||
|
||||
## 2. Consumers (read / compare)
|
||||
|
||||
| Site | file:line | Behaviour |
|
||||
|---|---|---|
|
||||
| Contract `SheetRef` | crates/core/src/domain/report/v3.rs:51-71 | `{sheetId, versionId\|null, documentSha256}`. `deserialize_with = "sha256"` → `js::is_sha256_hex` (js.rs:186-188, `^[0-9a-f]{64}$`). No `deny_unknown_fields` |
|
||||
| Contract `SheetSnapshotRef` | archive/snapshot.rs:39-55 | adds schemaVersion and speciesId, `deny_unknown_fields`, same sha256 check |
|
||||
| Bundle load | domain/sheet/loaded.rs:29-36, :104-111 | `BundledRef.document_sha256: String` copied into `LoadedSheet.reference` without format check or recomputation |
|
||||
| Index to TS | sheet/index_data.rs:25, :112 | exposes `ref` (`SheetIndexData.ref`) → apps/app/src/services/sheetIndex.ts:28, :63-66 |
|
||||
| New report | ops.rs:53-60, :117-125 → editor/report_ops.rs:56 | stamps the `SheetRef` it receives (validated). App: NewReportRoute.tsx:77 `sheet.sheet.ref` (from provider.ts `availabilityOf`, raw bundle) → reportStore.ts:354-366 → reportOps.ts:21-29 `report.new` |
|
||||
| v2 migration | domain/report/read.rs:91 | `sheet: map.sheet.clone()`: a migrated v2 report takes the migration map's ref |
|
||||
| Rebase | domain/report/rebase.rs:40-47 | only `report.sheet.document_sha256 == sheet.reference.document_sha256`: equal → report returned untouched, `rebased:false` |
|
||||
| | rebase.rs:48-62 | otherwise `rebased.sheet = sheet.reference.clone()` (sheetId, versionId and sha all replaced), observations rebased, `reparse` through the v3 schema |
|
||||
| Envelope invariant | domain/archive/read.rs:94-119 | `snapshot.ref.sheet_id == report.sheet.sheet_id && snapshot.ref.document_sha256 == report.sheet.document_sha256`. If false: `SnapshotRejection::SheetMismatch`, envelope dropped, report kept (:86-89) |
|
||||
| Session entries | domain/session.rs:76-101 (draft), :111-135 (import) | read, then rebase with the archived snapshot. A rebase failure becomes `RestoreFailure::Rebase` → `rebase_failed` (ops.rs:327) or `ImportFailure::Invalid` → `invalid` (ops.rs:351-357) |
|
||||
| Snapshot build | archive/snapshot_rules.rs:54-66 | copies the loaded ref into `sheetSnapshot.ref`. The snapshot `document` is the lean document, so the ref names the full document, not the archived one |
|
||||
| Export check | pdf/write/input.rs:204-210 | `sheet_id !=` or `document_sha256 !=` → `ExportErrorKind::SheetMismatch` (`sheet_mismatch`, :50, :66) |
|
||||
| Export envelope | pdf/write/export.rs:26-37 | writes `report` (with its `sheet`) and `sheetSnapshot` (with `ref`) as given, `JSON.stringify(…,null,2)` form. `export_from_envelope` (:116-137) re-exports from the envelope's own report and snapshot. The PDF layout draws no ref |
|
||||
| PDF import | pdf/import.rs:22-30, facade.rs:66-77 | byte scan → `import_archive` (same rebase) |
|
||||
| Rescue copy | facade.rs:80-100 | `restore_draft` semantics with no archive |
|
||||
| `sheet_ref` op | facade.rs:43 | echo of the contract (tests) |
|
||||
| report-compare | bin/report_compare.rs:32-66 | loads `sheet-ref.json` as-is. compare/mod.rs:133 reads the archive with no rebase. compare/lines.rs:347-367: sheetId and sha equal → `sheet … (draft, matches the bundled sheet)`, else `sheet … !! written on another sheet document (sha <12 hex>…)`. Exit code unaffected (exit 2 only on unreadable entries, run.rs:252) |
|
||||
| App, import messages | apps/app/src/app/routes/HomeRoute.tsx:41-100, :205-229 | see §4 |
|
||||
| App, draft restore | state/reportStore.ts:262-293; EditorRoute.tsx:58-67, :104-139 | see §4 |
|
||||
| App, error mapping | core/worker/outcome.ts:65-67 | `invalid` (rebase refused) → `ImportError('generic')`. EditorRoute.tsx:262: every export failure, `sheet_mismatch` included → `editor.export_failed` |
|
||||
| packages/domain (frozen; only packages/anatomy imports it, type-only for SheetRef) | report/v3.ts:31-38; rebase.ts:114-135; archive/envelope.ts:96-101; archive/snapshot.ts:24, :60-68, :262-276; report/migration-map.ts:17; report/migrate.ts:121; anatomical-sheet/sheet-index.ts:23 | same semantics as the Rust (rebase on sha only, invariant on sheetId+sha) |
|
||||
| N-1 reader | packages/core-wasm/tests/n-1/domain/** | byte-identical to packages/domain/src (`diff -r` empty). Used by pdf-export.test.ts:16, :192 and measure.test.ts:14 |
|
||||
| packages/supabase | none | publish gate anatomical-sheets.ts:74-98 checks shape and integrity only. scripts/upload-sheet-assets.ts:58-59 hashes asset bytes |
|
||||
| SQL RPCs | anatomical_sheets.sql:735-762 `create_sheet_draft` copies `content` verbatim from a non-draft version (same document, new `gen_random_uuid` id). :764-805 `publish_sheet_version` runs `sheet_document_issues` and `unknown_assets`, no hash. No reports table exists |
|
||||
|
||||
## 3. Stored values
|
||||
|
||||
Only one non-synthetic fingerprint has ever been committed. `git log` on sheet-ref.json shows one commit, 0a17e39 (2026-10-10), and document.json has not changed since that commit.
|
||||
|
||||
| Value | Where (file:line) | Identifies |
|
||||
|---|---|---|
|
||||
| `27c7383c445c73dc2001ebf4794a3ea63d416910bbbaa0d7848dd720de7c2660` | supabase/seed/anatomical-sheets/equine/draft/sheet-ref.json:4; draft/report-migration-map.json:6 | sha256 of the committed draft/document.json bytes (302,794 B, verified with `sha256sum`). sheetId `503dc301-a8b1-50d4-9e44-0919b2fc2916` = `deterministicUuid('faunosteo:sheet:equine:v1')` (verified; importer.ts:329), versionId null, curated draft schema v2 |
|
||||
| same | apps/app/tests/fixtures/report-v3.demo.json:24; r4/reports/L.json:24; r4/reports/CAP.json:24; pdf/r3/store-path.report.json:22 | Report v3 fixtures stamped on the bundled draft |
|
||||
| same | apps/app/tests/fixtures/goldens/r2/envelope.demo.fr.nologos.json:29,279; envelope.v2full…:29,160; envelope.v2unresolved…:24,763 | report.sheet + sheetSnapshot.ref of frozen envelopes |
|
||||
| same | crates/core/tests/oracles/editor.json (18 lines, 1669…2475); migration-map.draft.json:1; report-v3.demo.json:1 | R2 oracles frozen from the TS |
|
||||
| same | oracles/r4/compare/compare.json:9; oracles/r4/tolerance.json:6 | sheet pins, asserted equal to the loaded ref (pdf_compare.rs:266-270) |
|
||||
| same | oracles/r4/text.json:17807-17808 | an AFM width sample string, not a fingerprint use |
|
||||
| same | attachments of 99 committed PDFs (pdf/r3 13, r3/real 1, r4/read 21, r4/tolerance 50, step6 3, fixtures root 1, compare scenarios 10). Visible uncompressed in r4/read/scan.{crlf-unspaced-type-params,indirect-length,wrong-direct-length}.pdf | exported archives of the draft |
|
||||
| `fff…f` (64) | 1 PDF in r4/read; archive_rules.rs:777; domain archive.envelope.test.ts:189; r2-characterisation.test.ts:676; anatomy curation.migration-map.test.ts:15 | synthetic: snapshot of another sheet |
|
||||
| `bbb…b` | 1 PDF in compare scenarios; report_rules.rs:397; reportStore.test.ts:201; HomeRoute.archive.test.tsx:69; domain tests | synthetic "other sheet" |
|
||||
| `aaa…a`, `ccc…c`, `eee…e`, `000…0` | oracles/characterisation.json, oracles/verdicts.json (≈600 `a`), Rust tests (`"0".repeat(64)` in sheet_index.rs:120, editor_rules.rs:43, sheet_contract.rs:35, anatomy_canvas_op.rs:21, pdf_layout.rs:295), app tests (`'c'.repeat(64)` in heldSession/quarantine/rescueCopy, `'0'` in EditorRoute.test.tsx:223) | synthetic. Part of the `c` values are asset hashes |
|
||||
| `91e459f4c0ac8043831242c9fccde01f75f1023464db909f1c29571aa8a414c2` | not stored: sha256 of legacy-import/document.json, computed for this census | legacy baseline has no sheet-ref |
|
||||
| `eaf891f4…5d54` | not stored: sha256 of the draft's `jsonb::text` (PGlite) | §1 |
|
||||
|
||||
The other 64-hex values in the repo (89 files) are unrelated to the fingerprint: asset hashes (draft horse.svg `0ea5202f…` in draft/assets.json and equine_draft.sql:18, two legacy-import asset hashes), PDF and PNG digests in oracles, R4 golden digests, wasm and build digests under scripts/browser/results, and AFM data headers.
|
||||
|
||||
The PDF scan was a regex over raw bytes plus zlib inflation of every `stream`. 116 PDFs carry no fingerprint: r4/read hostile and v2 cases (110), 3 compare scenarios, 2 legacy v2 PDFs and 1 tolerance case.
|
||||
|
||||
## 4. Semantics on mismatch
|
||||
|
||||
Import (PDF or attachment JSON), session.rs:111-135, in this order.
|
||||
|
||||
1. The report is read first. A v3 report is kept as written. A v2 report is migrated and stamped with `map.sheet` (read.rs:91), which is 27c7…/503dc301 today.
|
||||
2. Envelope v1: the snapshot passes the strict gate, then the invariant. The snapshot ref and the report ref of the same file must agree on sheetId and fingerprint (read.rs:116-119). If they don't, the envelope is dropped (`snapshotRejected: 'sheet_mismatch'`) and the report is kept. Message: `home.import.snapshot_rejected` (fr.ts:46-47, en.ts:44-45) with `home.import.snapshot_reason.sheet_mismatch` ("fiche différente de celle du bilan", fr.ts:53-54; "sheet differs from the report's", en.ts:51-52). Headline `home.import.degraded`.
|
||||
3. Rebase onto the loaded sheet (rebase.rs). If the fingerprint is equal, nothing changes. If it differs, `report.sheet` becomes the loaded ref.
|
||||
- A zone becomes `unresolved: true` when its `annotationId` or `viewId` is unknown (:85-101). Its label snapshots are filled from the archived snapshot only when absent (`??=`, :103-112).
|
||||
- A finding becomes `unresolved: true` when `(findingZoneId, path)` is unknown (:137-153).
|
||||
- Already-flagged items are left alone and not re-counted. Ids, paths, severity, comments and labels are all kept. "unresolved" means "the loaded sheet cannot place this item", not "deleted".
|
||||
- The app shows `home.import.rebased` with the count: "Bilan ouvert sur la fiche actuelle, différente de celle de l'archive. Éléments non retrouvés : {count} (conservés avec leurs libellés)." (fr.ts:55-56, en.ts:53-54).
|
||||
- The headline is `home.import.complete` (fr.ts:39, en.ts:37) when 0 items are lost, else `home.import.degraded` (fr.ts:40-41, en.ts:38-39) (HomeRoute.tsx:70-84). An accepted envelope adds `home.import.archive_ok` (fr.ts:44-45).
|
||||
- The imported envelope is kept unchanged (`osteo:archive`) with its original refs. Only the working report is rebased.
|
||||
4. A sheetId that differs while the fingerprint is equal triggers no rebase, because rebase.rs:40 ignores sheetId. The foreign sheetId stays, and a later export is refused (`sheet_mismatch`, input.rs:206) as the generic `editor.export_failed` (EditorRoute.tsx:262). A different sheetId with a different fingerprint is rebased and the sheetId replaced.
|
||||
|
||||
Draft restore (reportStore.ts:262-293 → core `draft.restore`) uses the same rebase. When the draft was rebased, items were flagged and the practitioner has not confirmed replacement, the raw draft goes to `osteo:quarantine:draft-pre-rebase`, or is held if that slot cannot be written. The editor then shows `editor.draft_rebased` (fr.ts:89-90; en.ts:83-84). A rescue copy (HomeRoute.tsx:222-229) gets the same message. If the rebased report fails the schema (a defect of the loaded ref, such as a malformed bundle sha), the result is `rebase_failed`: the draft stays where it is and the app answers `core_unavailable` (reportStore.ts:266-271).
|
||||
|
||||
Export: the snapshot ref and the report ref must agree on sheetId and fingerprint. After any rebase both equal the loaded ref. The envelope carries the fingerprint twice (`report.sheet` and `sheetSnapshot.ref`, docs/PDF-FORMAT.md:60-70, :121-122), and the visible PDF text carries none. report-compare prints a header line only (§2) and never rebases.
|
||||
|
||||
## 5. Document data shape (script output)
|
||||
|
||||
```
|
||||
== draft/document.json bytes 302794 sha256 27c7383c…2660 BOM False CR False tabs False trailing LF True lines 8284
|
||||
top-level keys: schema_version species_id name description scope_disclaimer attribution vertebral_config references categories views annotations finding_zones
|
||||
numbers: {'int': 179, 'float': 29} (no negative, no exponent, no -0) float paths: geometry.points 28, geometry.cx 1
|
||||
float examples 743.0451049804688 1418.5 556.2 … max 16 significant digits; ending .0: 0; int range 2..1780
|
||||
raw escapes in file: \u 0 \n 0 \" 0 \\ 0 \/ 0 \t 0
|
||||
non-ASCII code points 1831, distinct 18: é 1354, è 139, à 106, ï 58, ° 45, ê 37, ç 22, â 19, « 15, » 15, – U+2013 6, ô 5 …; astral 0
|
||||
strings 3724, non-ASCII 788, control chars 0; max string length 521 code points ($.scope_disclaimer.fr)
|
||||
non-ASCII keys 0; duplicate-key objects 0; objects 1951, arrays 370
|
||||
objects with >=2 keys: sorted (UTF-16) 560, not sorted 1390 (e.g. $ top level; every {fr, en}); single-key 1
|
||||
empty objects 0; empty arrays 0; nulls 66 (doi 27, vertebral_region_code 22, url 13, license, view_box, svg_asset_id, name); booleans true 55, false 0
|
||||
max depth 10 at $.finding_zones[0].categories[0].children[0].children[0].name.fr
|
||||
== legacy-import/document.json bytes 224109 sha256 91e459f4…14c2 same top-level keys, LF, no BOM
|
||||
numbers int 159, float 36 (max 17 significant digits, 1424.2662353515625); no negative/exponent/-0; escapes 0; astral 0
|
||||
non-ASCII 1164 code points, 11 distinct (adds — U+2014 ×4, É ×2); strings 2889 (650 non-ASCII); max length 253
|
||||
objects 1808 (586 sorted, 1158 not, 64 single-key); empty arrays 1 ($.references); nulls 30; max depth 10
|
||||
```
|
||||
|
||||
Node checks:
|
||||
|
||||
```
|
||||
draft: file === JSON.stringify(v,null,2)+LF: false; first diff line 17 '"authors": ["Faunosteo"],' (Prettier collapses short arrays, e.g. "[1655, 545]"); 8284 vs 8695 lines
|
||||
draft: number tokens 208, not in ECMAScript Number#toString form 0, unsafe integers 0 (legacy: 195 / 0 / 0)
|
||||
draft: compact UTF-8 bytes 174,974; JSON.stringify(raw) 173,136 UTF-16 units vs JSON.stringify(Zod.parse(raw)) 245,636 (first diff: count_range:null default added)
|
||||
legacy: same pattern, 110,854 vs 190,448
|
||||
```
|
||||
|
||||
What these checks show:
|
||||
|
||||
- The hashed bytes are the input form: defaults are omitted (no empty arrays, no `count_range`). The parsed form adds those defaults, so its bytes differ.
|
||||
- Key order is construction or schema order, not sorted.
|
||||
- Numbers already print as ECMAScript does, and no string needs escaping.
|
||||
- The layout comes from Prettier's line fitting (printWidth 80), which is not `JSON.stringify` layout.
|
||||
|
||||
## 6. Documentation statements
|
||||
|
||||
| file:line | Statement (quoted or condensed) |
|
||||
|---|---|
|
||||
| docs/development/RUST_ARCHITECTURE_REVIEW.md:55-60 | "`documentSha256` is today the sha256 of the Prettier-formatted document text. It is computed once by the pipeline and only compared at runtime; a different hash sends an old archive through the rebase path … The study therefore does not chain the pipeline to Prettier (D-R5)" |
|
||||
| …REVIEW.md:145 | "sha256 of the Prettier-formatted lean document text … (`rebase.ts:119`, `envelope.ts:100`, pdf `export.ts:88`)". "lean" is inaccurate (the full curated document is hashed), and `packages/pdf` was deleted at R4.8 |
|
||||
| …REVIEW.md:584 | `serde_jcs` listed under "Rejected" (Part V) |
|
||||
| …REVIEW.md:636, :655 | R5 row: "seed golden regenerated once (decided)". Compatibility: "regenerated seed (new `documentSha256`) … rebase tests both ways on the two seeds" |
|
||||
| …REVIEW.md:837 (Part X), :873 (Annex A) | D-R5 rev 2: "`documentSha256` = sha256 of the bytes the Rust pipeline emits; generated dirs leave Prettier's scope; the one-time hash change makes older archives … rebase … 0 unresolved items". Alternatives named: keep Prettier, or canonical JSON (RFC 8785, `serde_jcs`) |
|
||||
| …REVIEW.md:974 (Annex C, owner amendment) | "Preference for a deterministic fingerprint over a canonical JSON representation independent of the formatter … study RFC 8785 (JCS) or a strictly defined alternative … Explicit compatibility with the fingerprints and archives already produced. Existing fingerprints are not changed now: the decision is prepared for R5" |
|
||||
| docs/development/RUST.md:2092-2094 | "`documentSha256` will change once (D-R5) and every older archive will rebase on import" |
|
||||
| RUST.md:1902 | report-compare loads `sheet-ref.json` |
|
||||
| docs/PDF-FORMAT.md:66, :106 | snapshot `ref.documentSha256`; "sha256 of the full catalog document (provenance pointer)" |
|
||||
| PDF-FORMAT.md:121-122 | "Invariant: `report.sheet.documentSha256 === sheetSnapshot.ref.documentSha256` (and the sheet ids match). The export refuses to write a file that breaks it." |
|
||||
| PDF-FORMAT.md:175, :195-200 | closed reason `sheet_mismatch`; "When the archive's sheet differs from the loaded one, the report is rebased …" |
|
||||
| docs/development/ANATOMICAL_SHEET_MODEL.md:317 | `sheet-ref.json` "(sheet id, null version id, sha256 of the document)" |
|
||||
| docs/development/STEP5_MIGRATION_ANALYSIS.md:74, :94-95, :137 | "`documentSha256` the hash of `draft/document.json`" |
|
||||
| docs/development/STEP6_PDF_ARCHIVE_ANALYSIS.md:88-89 | pre-Step-6 state: "Nothing compares `report.sheet` with the loaded sheet … Every `pnpm import:equine` run changes the draft's `documentSha256`" (the second sentence conflicts with the deterministic golden) |
|
||||
| STEP6:124, :160-161, :290-293, :775-776 | ref = "full catalog document's hash, kept as a pointer, not as a hash of the snapshot"; invariant; rebase trigger; export `sheet_mismatch` |
|
||||
| .claude/memory/decisions.md:327 (BDR-038) | Report v3 `sheet {sheetId, versionId\|null, documentSha256}` |
|
||||
| decisions.md:336 (BDR-039) | the snapshot hash check was rejected as "self-referential; sha copied … as provenance pointer" (about assets) |
|
||||
| decisions.md:361 (BDR-043) | alternative rejected: "Prettier-chained sheet identity (hash only compared at runtime → one-time rebase accepted)" |
|
||||
| decisions.md:371 (BDR-045) | "D-R5 fingerprint → canonical JSON (RFC 8785 JCS or strictly defined alternative) independent of formatter, explicit compat with old fingerprints/archives, existing fingerprints untouched, decision prepared for R5" |
|
||||
| .claude/memory/learnings.md:363 (LRN-048) | "`documentSha256` = sha256 of Prettier-formatted text (only compared at runtime → mismatch = rebase, not loss)" |
|
||||
| .claude/tasks/TODO.md:307 | "seed golden regenerated once (new `documentSha256`, D-R5)" |
|
||||
| CLAUDE.md:277 | `draft/` holds `sheet-ref` |
|
||||
|
||||
## 7. Risks and open questions (analyzer)
|
||||
|
||||
Risks:
|
||||
|
||||
- Any change to the emitted bytes with identical content (Rust emitter, Prettier bump, printWidth) changes the value. Every archive and draft carrying 27c7… then rebases with a visible message, normally with 0 unresolved items. That covers 99 committed PDFs, 8 JSON fixtures and 6 Rust oracle files, which would need decisions.
|
||||
- map.sheet and sheet-ref.json are two copies with no runtime equality check. If they diverge, every v2 import is stamped with one and rebased onto the other (low: one pipeline value plus the golden).
|
||||
- The bundle fingerprint is not format-checked at load. A malformed value surfaces later as `rebase_failed`, generic `invalid` or a refused `report.new` (low).
|
||||
- versionId is never compared, and create_sheet_draft copies content verbatim. Once sheets come from the database (Step 13), versions with identical content would share a fingerprint and skip the rebase (medium).
|
||||
- If a fingerprint has to be computed from database `content`, the jsonb text differs from the file (key order, numeric scale). The current value cannot be reproduced from jsonb (medium at Step 13).
|
||||
- Stale or contradictory docs (REVIEW:145 "lean" and deleted citation, RUST.md:2092 and TODO:307 vs BDR-045, STEP6:89) could steer the R5 plan.
|
||||
|
||||
Open questions:
|
||||
|
||||
- Which statement is in force for R5? BDR-045 / Annex C ("existing fingerprints untouched, explicit compatibility") or RUST.md:2092 / TODO:307 ("will change once")?
|
||||
- What does "compatibility with fingerprints already produced" mean concretely: keep 27c7… for the current draft, an old→new equivalence, or a one-time rebase with a message?
|
||||
- Which form gets canonicalised: the input file form (defaults omitted) or the parsed form (defaults filled)? They differ by about 72k UTF-16 units.
|
||||
- For database-served sheets, who computes and stores the fingerprint (pipeline or operator, Rust service, SQL)? No column exists.
|
||||
- Should sheetId or versionId join the rebase decision (today it uses the fingerprint only), given that the export and the envelope check sheetId too?
|
||||
@@ -0,0 +1,297 @@
|
||||
# R5 inventory: the TypeScript anatomy pipeline (`packages/anatomy`)
|
||||
|
||||
Root: `/home/bchanot/Documents/faunosteo` (paths repo-relative). Branch `feature/rust-r5-pipeline-core` cut at 9c0117a. Read-only analysis by an analyzer agent (one in-memory probe of the pipeline functions for §4.3), saved by the orchestrator. Runtime: Node 24.21.0, ICU 78.3, Unicode 17.0.
|
||||
|
||||
## Summary
|
||||
|
||||
1. `packages/anatomy` has 43 source files and 4,897 lines: legacy 1,842, curation 2,329, render 553, cli 119, root 54. Tests are 2,553 lines; the upload script is 99.
|
||||
2. The runtime libraries are Prettier 3.8.3, @xmldom/xmldom 0.9.12 and Zod 3.25.76, plus node:crypto, node:fs and node:util. Prettier and xmldom are declared as devDependencies but used at runtime.
|
||||
3. There are 18 generated files. Ten go through Prettier: 7 JSON and 3 Markdown. Five go through the xmldom XMLSerializer (horse.svg and 4 review SVGs). One is a raw SQL template, and two are byte copies of the legacy SVGs.
|
||||
4. `documentSha256` is the sha256 of the UTF-8 bytes of the Prettier-formatted `draft/document.json`, trailing newline included (`pipeline.ts:159-165,185`, `ids.ts:15-17`). Checked equal to the sha256 of the committed file. `equine_draft.sql` does not carry it; it embeds the document text instead.
|
||||
5. That hash is pinned in 18 tracked files (seed, crate oracles, app goldens, R4 PDFs). Any byte change in the draft document cascades to all of them.
|
||||
6. Measured on today's data, Prettier changes only Markdown table padding (134, 863 and 52 lines in the three reports). For JSON it changes layout only: short arrays collapsed onto one line and a trailing newline. The data is identical after re-parsing.
|
||||
7. The JSON key order is JS insertion order of the working document, which is never re-parsed by Zod. For example, `review_status` is appended last on annotations by the B/C review rule.
|
||||
8. UUIDs are built from the first 16 bytes of sha256(name), with the version nibble set to 5 and variant bits 10. Names are `faunosteo:asset:<sha>` and `faunosteo:sheet:equine:v1` (gives 503dc301…). Both checked.
|
||||
9. JS-specific hazards: number printing, `Math.round`, `Number('') = 0`; Unicode `\s`, `split(':', 2)`, NFD plus `toLowerCase`; UTF-16 sorting and slicing; xmldom's `/>`, escaping and namespace rules; Prettier's printNumber and its string-width function. None uses dates or randomness, and nothing calls localeCompare.
|
||||
10. Tests: 111 declarations, 131 cases once `it.each` rows are expanded. The golden test compares the exact text of every generated file; no comparison works on parsed trees.
|
||||
11. Five tests depend on the TS `render/` code. Three use it directly (render.layer, render.compose, r3.layer-oracles). Two use it through `renderDraftArtifacts` → `app-review` → `composeViewSvg` (seed.golden, curation.pipeline).
|
||||
12. 22 source files import `@faunosteo/domain`. Only 5 runtime symbols are used: the document schema, `validateSheetDocument`, the schema version, the stable-id pattern and `ReferenceSchema`. All already have Rust counterparts from R2.
|
||||
13. `@faunosteo/anatomy` has no code consumer outside itself, only the root `import:equine` script. `@faunosteo/domain` is imported only by `packages/anatomy`.
|
||||
14. Deleting the packages breaks things beyond imports: Rust tests read `packages/domain/tests/fixtures/report-v2.{full,minimal}.json`, and `apps/app/tests/unit/r4.census.test.ts:80-90` asserts xmldom sits under `packages/anatomy`.
|
||||
15. Open decision: the D-R5 hash rule as written (hash the bytes Rust emits) versus the owner's amendment in BDR-045 (a canonical-JSON fingerprint, decision "prepared for R5").
|
||||
|
||||
## 1. Package and tooling
|
||||
|
||||
- `packages/anatomy/package.json`: exports `.` → `src/index.ts` and `./render` → `src/render/index.ts` (:10-13); script `import:equine` = `tsx src/cli/import-equine.ts` (:20); dependencies `@faunosteo/domain` workspace and `zod` ^3.24.1 (:22-25); `@xmldom/xmldom` ^0.9.8 and `prettier` ^3.4.2 are devDependencies (:28, :31), yet `src/format.ts` and `src/legacy/svg.ts` import them at runtime, as do `curation/{assets,overlay,app-review}.ts`.
|
||||
- Lockfile (importer `packages/anatomy`): zod 3.25.76, @xmldom/xmldom 0.9.12, prettier 3.8.3, tsx 4.23.15, typescript 5.9.3, vitest 3.2.7, @types/node 20.19.41. Each of prettier and xmldom has a single version in the whole lockfile.
|
||||
- `vitest.config.ts`: environment node, globals, include `tests/**/*.test.ts`. `tsconfig.json`: ES2022, bundler resolution, strict, noEmit, covers src and tests. `eslint.config.js`: typescript-eslint recommended.
|
||||
- Root wiring: `package.json:28` `import:equine` = `pnpm --filter @faunosteo/anatomy import:equine`; `package.json:36` `assets:upload` → `packages/supabase/package.json:21` `node scripts/upload-sheet-assets.ts` (Node's native type stripping, not tsx).
|
||||
- Prettier options (`src/format.ts:9-19`) are identical to the root `.prettierrc.json`. Root `format` = `prettier --write .` (`package.json:24`). `.prettierignore` does not exclude `supabase/seed/**`, so a root format run also rewrites the generated `.json` and `.md` files.
|
||||
|
||||
## 2. Lines of code (wc -l)
|
||||
|
||||
| Area | Files | Lines |
|
||||
|---|---|---|
|
||||
| src/legacy | 15 | 1,842 |
|
||||
| src/curation | 19 | 2,329 |
|
||||
| src/render | 5 | 553 |
|
||||
| src/{index,format,markdown}.ts | 3 | 54 |
|
||||
| src/cli/import-equine.ts | 1 | 119 |
|
||||
| src total | 43 | 4,897 |
|
||||
| tests (16 suites + helpers.ts + fixtures/mini-sheet.ts) | 18 | 2,553 |
|
||||
| packages/supabase/scripts/upload-sheet-assets.ts | 1 | 99 |
|
||||
|
||||
## 3. Modules (path (lines): exports with line, inputs → outputs, external deps)
|
||||
|
||||
### 3.1 Shared and entry
|
||||
|
||||
| Module | Exports | In → Out | Deps |
|
||||
|---|---|---|---|
|
||||
| index.ts (3) | re-exports legacy, curation, render | | |
|
||||
| format.ts (30) | formatJson:21, formatMarkdown:28 | value → Prettier(`JSON.stringify(v,null,2)`, parser json); text → Prettier markdown | prettier |
|
||||
| markdown.ts (21) | cell:6, table:11, code:19 | null → `-`; escapes `\|` and LF only; tables unpadded (Prettier pads them) | |
|
||||
| cli/import-equine.ts (119) | none (main) | repo → writes legacy-import/ and draft/; prints 2 summary lines | node:fs/promises, node:path, node:url |
|
||||
|
||||
CLI behaviour: REPO_ROOT is 4 levels up (:32). Strings are written with `writeFile(…,'utf8')`, otherwise `copyFile` (:37-44). `removeStale` deletes every file in the two folders that the run did not produce (:47-55). The legacy folder gets `renderArtifacts` output plus the two SVG copies (:104-107). The draft folder gets `renderDraftArtifacts` output plus `assets/<file>` = cleanedSvg (:108-111).
|
||||
|
||||
### 3.2 legacy/ (importer, Step 4b)
|
||||
|
||||
| Module | Exports | In → Out | Deps |
|
||||
|---|---|---|---|
|
||||
| sources.ts (54) | EQUINE_SOURCE_PATHS:10, EQUINE_{LEGACY_IMPORT,CURATION,DRAFT}_DIR:20,23,26, readEquineSources:32 | root → SVG bytes, catalog text, vertebrae text | node:fs, node:path |
|
||||
| types.ts (115) | Anomaly, AssetRecord, LegacyIdMap, ImportResult, … | types | domain (types) |
|
||||
| ids.ts (54) | sha256Hex:15, deterministicUuid:19, assetIdFromSha256:33, sheetIdFor:37, slugify:42, isStableId:52 | | node:crypto, domain STABLE_ID_PATTERN |
|
||||
| svg.ts (116) | parseAnnotatedSvg:94 | SVG text → viewBox, one record per data-anno-id (first element wins, elementCount), anomalies | xmldom DOMParser |
|
||||
| vertebrae.ts (51) | parseVertebralFormula:16, vertebralRegions:45 | vertebrae.ts text, parsed with regexes (not imported) → counts, "varies A–B" | |
|
||||
| regions.ts (46) | inferRegionCode:36 | FR then EN label → code/root; root table :8-14 | |
|
||||
| categories.ts (98) | normaliseCategoryId:37, mapCategories:82 | annotations → categories in first-seen order; 2 slug fixes :14-17; authored labels :20-29 | |
|
||||
| catalog.ts (249) | parseLegacyCatalog:73, categoryKind:81, countTree:169, assertNothingDropped:206, mapCatalog:233 | pathologies.json → lean finding zones (optional keys only when present); throws if output count ≠ input | |
|
||||
| catalog-checks.ts (102) | catalogAnomalies:96 | foreign prefix, anamnesis variants, cross-zone duplicates | |
|
||||
| annotations.ts (207) | zoneNameKey:37, mapAnnotation:184 | annotation + map `fr\nen` → zone → input annotation, row, region, anomalies | |
|
||||
| authored-labels.ts (41) | authoredLabelAnomalies:30 | document → label.authored anomalies | |
|
||||
| importer.ts (343) | importEquineLegacy:313, MAIN/HEAD_VIEW_ID:50-51 | EquineLegacyInput → ImportResult | domain schema and validate |
|
||||
| report.ts (325) | renderImportReport:310 | ImportResult → French markdown, 9 sections, trailing `\n` | |
|
||||
| artifacts.ts (22) | renderArtifacts:13 | ImportResult → 4 formatted texts | format.ts |
|
||||
|
||||
Facts about the importer: the document written is the unparsed input form (`importer.ts:322`); the Zod parse (:317) only feeds `validateSheetDocument` (:341). Two guards throw: annotations read ≠ written (:223-228), and inline sources present (:234-239). The trigger id is `${opensDetail}-trigger` (`annotations.ts:117`); an observable annotation's id is the zone id, or `slugify(name.en)` as a fallback (:171). The slugify fallbacks never fire on today's data: `legacy-id-map.json` has `finding_nodes: {}`, and all 26 observables resolve to a zone.
|
||||
|
||||
### 3.3 curation/ (Steps 4c and 5)
|
||||
|
||||
| Module | Exports | In → Out | Deps |
|
||||
|---|---|---|---|
|
||||
| schema.ts (172) | OpSchema:101, ChangeSchema:110, ChangesetSchema:132, ReferencesFileSchema:141, needsConfirmation:164, changeSummary:168, ASSET_CHANGE_ID:29 | strict Zod; priority default `medium` (:119); level C needs a question (:123-126); duplicate ids refused (:128-148) | zod, domain ReferenceSchema |
|
||||
| sources.ts (38) | readEquineCuration:26 | curation/{references,changes}.json → parsed | node:fs |
|
||||
| targets.ts (195) | parseTarget:58, formatTarget:73, locate:131, zoneFindings:151, allFindings:158, findingScope:167, childList:185, childPrefix:192 | target grammar (:2-5) over plain JSON | |
|
||||
| fields.ts (80) | SETTABLE table (:9-43), splitField:45, readField:62, writeField:72 | dot paths; missing intermediate objects created | |
|
||||
| ops.ts (225) | applyOp:209, assertDeclared:184 | set, rename, add, remove, move, cite → in-place edits plus records | node:util isDeepStrictEqual |
|
||||
| rename.ts (57) | propagateRename:54 | link table (:14-23) and citations | |
|
||||
| types.ts (49) | ChangeRecord, record:35 | `structuredClone` of before/after; undefined → null | |
|
||||
| apply.ts (134) | applyChangeset:122, curatedIssues:103 | baseline + references + changeset → document + records; B/C changes → `to_confirm` (:46-73) | domain schema and validate |
|
||||
| assets.ts (59) | stripLegacyAnnotations:24, cleanedAsset:38, parseViewBox:49 | removes every `[data-anno-id]` element, and the root `style` when it equals the legacy value exactly (:17, :31-33) | xmldom parse and serialise |
|
||||
| migration-map.ts (178) | buildReportMigrationMap:127 | baseline, final, legacyIdMap, records, sheet ref → map + stats | |
|
||||
| stats.ts (133) | documentStats:77, statRows:113 | document → counts and French rows | |
|
||||
| changelog.ts (132) | renderChangelog:119, compactValue:29, recordSubject:38 | records + stats → markdown | |
|
||||
| review-report.ts (206) | renderReviewReport:201 | 10 fixed sections | |
|
||||
| labels.ts (29) | KIND/LEVEL/PRIORITY_LABELS, LEVELS | French labels | |
|
||||
| overlay.ts (214) | overlayMarkers:49, renderReviewOverlay:185 | cleaned SVG + before/after documents + view → review SVG | xmldom, node:util |
|
||||
| app-review.ts (26) | renderAppReviewDrawing:20 | cleaned SVG + document + view → `composeViewSvg` → text | xmldom, render/compose |
|
||||
| seed.ts (125) | renderDraftSeedSql:114, DOCUMENT_DOLLAR_TAG:13, DRAFT_CHANGE_NOTE:19 | sheetId, assets, documentJson → SQL | |
|
||||
| pipeline.ts (224) | buildEquineDraft:112, renderDraftArtifacts:181, file-name constants (:41, :155-156) | see §4 | |
|
||||
| index.ts (53) | re-exports | | |
|
||||
|
||||
Facts about the curation pipeline: CHG-000 (the asset step: 3 `set` ops, `pipeline.ts:68-104`) is prepended, and an authored CHG-000 is refused (:106-110). The references file replaces the document registry before any change (`apply.ts:128`). `curatedIssues` runs twice (`apply.ts:132`, `pipeline.ts:135`). The real changeset has 45 changes, CHG-001 to CHG-045: 14 A, 22 B, 9 C; 11 carry a question; 34 set an explicit priority. It holds 687 ops: 587 set, 45 cite, 33 rename, 18 add, 4 remove, 0 move. It cites 48 references. `move` is exercised only by tests.
|
||||
|
||||
### 3.4 render/ (frozen TS renderer; R3 counterpart in `crates/core/src/anatomy/{layer,compose}.rs`)
|
||||
|
||||
| Module | Exports | Role |
|
||||
|---|---|---|
|
||||
| layer.ts (261) | renderAnnotationLayer:232, styleForView:75, viewBoxOf:258, MAIN/DETAIL_VIEW_STYLE:49,53, FALLBACK_COLOR:59 | document + view (+ states) → LayerElement tree |
|
||||
| dom.ts (38) | createLayerElement:24, SVG_NS:9 | tree → DOM via createElementNS/setAttribute |
|
||||
| serialize.ts (25) | serializeLayer:13 | escapes `& < > "` only; called only by tests |
|
||||
| compose.ts (196) | composeViewSvg:54, highlightLayer:82, HALO_FILTER_ID:20 | sets viewBox, removes width/height, appends the layer last; the halo `<defs>` is created as the first child |
|
||||
| index.ts (33) | re-exports | |
|
||||
|
||||
The pipeline's only consumer of `render/` is `curation/app-review.ts:3`.
|
||||
|
||||
## 4. Generated artifacts
|
||||
|
||||
### 4.1 Producers
|
||||
|
||||
Text is written by `import-equine.ts:41` (utf8).
|
||||
|
||||
| File | Bytes | Producer | Inputs | Formatting |
|
||||
|---|---|---|---|---|
|
||||
| legacy-import/document.json | 224,109 | artifacts.ts:17 | result.document | JSON.stringify(v,null,2) → Prettier json |
|
||||
| legacy-import/legacy-id-map.json | 1,323 | artifacts.ts:18 | result.legacyIdMap | same |
|
||||
| legacy-import/assets.json | 408 | artifacts.ts:19 | result.assets | same |
|
||||
| legacy-import/IMPORT-REPORT.md | 22,402 | artifacts.ts:20 → report.ts:310 | ImportResult | Prettier markdown |
|
||||
| legacy-import/assets/Horse.svg, Horse_head.svg | 145,770 / 139,079 | import-equine.ts:63-74 | legacy-sources | copyFile (byte copy) |
|
||||
| draft/document.json | 302,794 | pipeline.ts:185, :203 | draft.document | formatJson |
|
||||
| draft/assets.json | 203 | pipeline.ts:204 | draft.assets | formatJson |
|
||||
| draft/sheet-ref.json | 217 | pipeline.ts:205-211 | ref, schema_version, species_id | formatJson |
|
||||
| draft/report-migration-map.json | 85,627 | pipeline.ts:187-194, :212 → migration-map.ts:127 | baseline, final, legacyIdMap, records, ref, `main` | formatJson |
|
||||
| draft/CHANGELOG.md | 1,114,533 | pipeline.ts:195-200, :215 → changelog.ts:119 | records, statsBefore/After, migration stats | Prettier markdown |
|
||||
| draft/REVIEW-REPORT.md | 108,015 | pipeline.ts:201, :216 → review-report.ts:201 | changes, records, document, statsBefore, legacy anomalies | Prettier markdown |
|
||||
| draft/review/annotations-{main,head}.svg | 141,278 / 137,275 | pipeline.ts:141-153 → overlay.ts:185 | cleanedSvg, baseline, draft, view | xmldom XMLSerializer, raw |
|
||||
| draft/review/app-{main,head}.svg | 146,232 / 139,462 | pipeline.ts:168-178 → app-review.ts:20 → compose.ts:54 | cleanedSvg, draft, view | xmldom XMLSerializer, raw |
|
||||
| draft/equine_draft.sql | 607,446 | pipeline.ts:217-222 → seed.ts:114 | sheetId, assets, documentJson, change note | string template, raw, ends `\n` |
|
||||
| draft/assets/horse.svg | 132,997 | import-equine.ts:110 ← assets.ts:24 | Horse.svg, decoded as fatal UTF-8 | xmldom XMLSerializer, raw |
|
||||
|
||||
Authored inputs (never generated): `legacy-sources/`: Horse.svg 145,770, Horse_head.svg 139,079, pathologies.json 174,156, vertebrae.ts 303; `curation/`: changes.json 226,769, references.json 17,766.
|
||||
|
||||
### 4.2 JSON shapes
|
||||
|
||||
- pathologies.json: `{schema_version:"2.0", scope_disclaimer, species{3}, zones{26}}`.
|
||||
- changes.json: `{changes[45]}`. Each change has id, level, kind, title, rationale, reference_ids, [question], [priority], ops.
|
||||
- references.json: `{references[48]}`; keys id, type, citation, url, doi.
|
||||
- document.json: `{schema_version, species_id, name, description, scope_disclaimer, attribution, vertebral_config, references, categories[5], views[2], annotations, finding_zones}`; legacy-import: 27 annotations, 26 finding zones; draft: 28 annotations, 27 finding zones.
|
||||
- legacy-id-map.json: `{views{1}, annotations{27}, categories{5}, finding_nodes{0}, assets{2}}`.
|
||||
- assets.json: `[{file, sha256, byte_size, svg_asset_id}]` (2 legacy, 1 draft).
|
||||
- sheet-ref.json: `{sheet_id, version_id:null, document_sha256, schema_version:2, species_id}`, snake_case.
|
||||
- report-migration-map.json: `{schema_version:1, sheet{sheetId, versionId, documentSha256}, main_view_id, views{head:"head"}, annotations{27×{id, name{fr,en}}}, finding_paths{792, 34 null}}`; `sheet` is camelCase.
|
||||
|
||||
### 4.3 What Prettier changes (measured in memory by the probe)
|
||||
|
||||
Markdown: only table cell padding and separator widths change: IMPORT-REPORT 134 lines, CHANGELOG 863, REVIEW-REPORT 52. No other line changes, and the line counts are identical. Column widths come from Prettier's string-width function (prettier 3.8.3 `plugins/markdown.js`): an all-ASCII string counts its length; otherwise, per code point, C0/C1 controls, U+0300–036F and U+FE00–FE0F count 0, East Asian Wide/Fullwidth characters and emoji count 2, everything else counts 1.
|
||||
|
||||
JSON: the data is identical after re-parsing; only layout changes: arrays of primitives collapse onto one line when they fit in 80 columns (`"authors": ["Faunosteo"]`, `[1655, 545]`); objects stay expanded, because JSON.stringify put a newline after `{`; a trailing `\n` is added.
|
||||
|
||||
Numbers: Prettier prints JSON numbers through printNumber (lowercase, `e+21` becomes `e21`, leading zeros of the exponent and trailing zeros stripped). There are 0 exponent-form numbers in the JSON today, 29 fractional values in draft/document.json.
|
||||
|
||||
### 4.4 xmldom 0.9.12 serialisation facts
|
||||
|
||||
- The XML declaration and whitespace text nodes round-trip: horse.svg keeps 703 lines and starts with `<?xml version="1.0" encoding="UTF-8"?>`.
|
||||
- Attributes keep document order. Values are escaped on `<>&"\t\n\r` (`< > & " 	 `). Text nodes are escaped on `<&>`.
|
||||
- A childless element is self-closed as `/>` with no space. Observed: `<g id="osteo-overlays">` became `<g id="osteo-overlays"/>` once its children were stripped.
|
||||
- A namespace declaration is written only when not already in scope, so layer children carry no xmlns.
|
||||
- Nothing follows the root: the files end with `</svg>`. Nodes added for the layer and overlay are appended with no whitespace between them.
|
||||
- The inputs contain no NBSP, tab, CR or BOM, and no comment in horse.svg.
|
||||
|
||||
## 5. Identities and hashes
|
||||
|
||||
- documentSha256: `documentJson = await formatJson(draft.document)` (`pipeline.ts:185`), then `sha256Hex(documentJson)` (`pipeline.ts:159-165`); `sha256Hex` hashes the string as UTF-8 (`ids.ts:15-17`); the same string is written as draft/document.json (`pipeline.ts:203`). Verified: 27c7383c…2660 matches both the committed file and the probe's recomputation.
|
||||
- Carried in sheet-ref.json `document_sha256` (`pipeline.ts:208`) and report-migration-map.json `sheet.documentSha256` (`pipeline.ts:192` → `migration-map.ts:162`).
|
||||
- equine_draft.sql does not carry it. It embeds `$equine_doc$` + `documentJson.trimEnd()` + `$equine_doc$::jsonb` (`seed.ts:79`) twice, in the insert (:82-95) and the converge statement (:101-112). It also embeds the cleaned asset sha256 (`seed.ts:61`).
|
||||
- The hash is pinned in 18 tracked files: draft/{sheet-ref,report-migration-map}.json; crates/core/tests/oracles/{editor,migration-map.draft,report-v3.demo}.json; crates/core/tests/oracles/r4/{compare/compare,text,tolerance}.json; apps/app/tests/fixtures/report-v3.demo.json; apps/app/tests/fixtures/goldens/r2/envelope.{demo,v2full,v2unresolved}.fr.nologos.json; apps/app/tests/fixtures/pdf/r3/store-path.report.json; apps/app/tests/fixtures/pdf/r4/read/scan.{crlf-unspaced-type-params,indirect-length,wrong-direct-length}.pdf; apps/app/tests/fixtures/r4/reports/{CAP,L}.json. (The fingerprint census counts 99 PDFs once the compressed attachments are inflated.)
|
||||
- UUIDs (`ids.ts:19-31`): first 16 bytes of sha256(UTF-8 name); byte 6 = (b & 0x0f) | 0x50; byte 8 = (b & 0x3f) | 0x80; lowercase 8-4-4-4-12. Names `faunosteo:asset:<sha256 hex>` (:33-35) and `faunosteo:sheet:<species>:v<n>` (:37-39). Verified: sheet 503dc301-a8b1-50d4-9e44-0919b2fc2916, horse.svg 7f767f74-3b36-51a7-8465-52dd7e2ead4a.
|
||||
- Other hashes: legacy asset sha = raw file bytes (`importer.ts:67-75`); cleaned asset sha = `TextEncoder().encode(cleanedSvg)` (`pipeline.ts:118`); source digests (pathologies.json, vertebrae.ts) = sha256 of the decoded then re-encoded text (`importer.ts:77-80`).
|
||||
|
||||
## 6. JavaScript-specific behaviours (a Rust port could reproduce these differently)
|
||||
|
||||
Object key insertion order drives every JSON artifact. The draft document is the working form and is never re-parsed. An edited key stays in place and a new key is appended (`fields.ts:76-79`). `review_status` is appended last by the B/C rule (`apply.ts:57`): the draft's annotation keys end `…, vertebral_region_code, review_status`. Added items keep the key order authored in changes.json (`ops.ts:150`). References follow the Zod schema order, with url/doi defaulting to null (`packages/domain/src/anatomical-sheet/schema.ts:200-207`). ChangeSummary follows Zod output order: `priority` sits at its schema position, and `question` is omitted when absent.
|
||||
|
||||
JSON.parse rules: integer-like keys iterate first (`catalog.ts:240`; `catalog-checks.ts:14-16,42,72`; `migration-map.ts:137-138,150`); today there are 0 integer-like zone keys and 0 integer-like legacy ids. A duplicate key keeps the last value.
|
||||
|
||||
Map/Set insertion order: `svg.ts:97` first element per id, in document order; `categories.ts:71-79` first seen, the name filled later without moving the entry; also `catalog-checks.ts:41,71`, `overlay.ts:54,72`, `migration-map.ts:152`.
|
||||
|
||||
Object identity: `apply.ts:36-43,53` key a Map/Set by item object; `ops.ts:38` (`!== self`) and `ops.ts:159,176` (`indexOf(item)`) compare identity.
|
||||
|
||||
Sorting: `catalog-checks.ts:78` count descending, then `<` on strings (UTF-16 code-unit order; the helper at :64-67 is misnamed byCodePoint); `review-report.ts:153-164` priority rank, then id compared with `<`. The sort is stable; there is no localeCompare in src.
|
||||
|
||||
Number to string (ECMAScript rules): `format.ts:22` and `changelog.ts:33` (JSON.stringify); `annotations.ts:61` and the report.ts tables; `layer.ts:99-107,171-175,260`; `overlay.ts:82-84,91,105,113,131,139-143,168,193`; `seed.ts:62`. Rust has `crates/core/src/json.rs` `js_number:18`, `to_js_pretty:171`, `to_js_compact:200`; serde_json is built with `preserve_order` and `float_roundtrip`. There is no equivalent of Prettier's layout.
|
||||
|
||||
Floating-point arithmetic: `layer.ts:171` max(2, r − 12); `overlay.ts:105` cy − r − 4; `:109-113` centroid = sum ÷ n; `:131` fontSize/8; `:168` y + (i+1)·fontSize·1.3 left to right; `:201` Math.round(width/90), where a −x.5 value rounds toward +∞, unlike Rust's f64::round.
|
||||
|
||||
Number() parsing: `svg.ts:54-56` `Number(null)` and `Number('')` give 0, whitespace trimmed, hex and exponents accepted; `svg.ts:37-47` and `assets.ts:50-53` `.trim().split(/[\s,]+/).map(Number)` (a leading comma yields a 0 entry, an odd point count drops the last number); `overlay.ts:197-199` splits without trim.
|
||||
|
||||
Regex classes: `\s` is Unicode whitespace, NBSP and U+FEFF included (`svg.ts:40`, `assets.ts:52`, `overlay.ts:198`, `vertebrae.ts:17,20,27`); `\w` and `\d` are ASCII-only, no `u` flag (`vertebrae.ts:20`; `schema.ts:112` `^CHG-\d{3,}$`); `STABLE_ID_PATTERN` is `/^[a-z0-9][a-z0-9_-]{0,63}$/`; the combining range is written as literal characters U+0300–U+036F (`ids.ts:45`, `regions.ts:23`).
|
||||
|
||||
Normalisation and case: `normalize('NFD')`, strip U+0300–036F, `toLowerCase()` (`ids.ts:42-50`, `regions.ts:22-24`), ICU 78.3 tables; slugify ends with `.slice(0,64)`; no toUpperCase.
|
||||
|
||||
UTF-16 lengths: `compactValue` caps text at 120 units and cuts with `slice(0,119) + '…'` (`changelog.ts:34-35`), which can split a surrogate pair. Zod `.min(1)` also counts UTF-16 units.
|
||||
|
||||
Splitting and trimming: `trim` and `trimEnd` use the JS whitespace set (`svg.ts:39`, `assets.ts:51`, `seed.ts:79`); `split(':', 2)` discards the remainder (`migration-map.ts:82`), whereas Rust's splitn keeps it.
|
||||
|
||||
Decoding: TextDecoder fatal mode for the SVGs (`importer.ts:83`, `pipeline.ts:114`), which strips a leading BOM; `Buffer.toString('utf8')` is non-fatal (U+FFFD, BOM kept) for pathologies.json and vertebrae.ts (`sources.ts:51-52`); `readFile(…,'utf8')` reads the curation files (`curation/sources.ts:23`).
|
||||
|
||||
Deep equality and copies: `isDeepStrictEqual` ignores key order, NaN equals NaN, +0 differs from −0 (`ops.ts:91`, `overlay.ts:63`); `structuredClone` deep copies (`types.ts:46-47`, `fields.ts:79`, `ops.ts:109,150`, `apply.ts:127-128`).
|
||||
|
||||
Zod semantics: curation schemas are strict objects, discriminated by `op`; `.int()` accepts 2.0; defaults apply. Reference url = `z.string().url()` + `/^https?:\/\//i` (domain `schema.ts:189-192`); Rust has `domain/sheet/schema.rs:100-107` with `js::is_url`.
|
||||
|
||||
Markdown escaping: `cell()` escapes `|` and LF only, not CR (`markdown.ts:6-9`). No clock, no randomness: no Date, no Math.random, no randomUUID in src.
|
||||
|
||||
## 7. Dependency on packages/domain
|
||||
|
||||
22 source files import it, 17 of them type-only.
|
||||
|
||||
| Runtime symbol | Used in | Domain source |
|
||||
|---|---|---|
|
||||
| AnatomicalSheetDocumentSchema | legacy/importer.ts:2, curation/apply.ts:2 | anatomical-sheet/schema.ts:295 |
|
||||
| validateSheetDocument | importer.ts:4, apply.ts:3 | anatomical-sheet/validate.ts:465 |
|
||||
| SHEET_DOCUMENT_SCHEMA_VERSION (=2) | importer.ts:3, legacy/report.ts:1 | schema.ts:18 |
|
||||
| STABLE_ID_PATTERN | legacy/ids.ts:2 | schema.ts:21 |
|
||||
| ReferenceSchema | curation/schema.ts:1 | schema.ts:200 |
|
||||
|
||||
Type-only symbols (schema.ts unless noted): AnatomicalSheetDocumentInput (:315) in 13 files; Geometry (:322); SheetView (:321); SheetIssue (validate.ts:20); Reference (:210); ViewBox (:320); Laterality (:323); FindingCategoryKind (:325), FindingNodeInput (:227), LocalizedOptionalText (:213); AnnotationCategory (:319); ReportMigrationMap (report/migration-map.ts:33) and SheetRef (report/v3.ts:38).
|
||||
|
||||
Test runtime symbols: AnatomicalSheetDocumentSchema, validateSheetDocument, ReportMigrationMapSchema (report/migration-map.ts:15); curation.migration-leaves only: ReportV2Schema (report/v2.ts:59), findingPathKey (report/migration-map.ts:35), migrateReportV2ToV3 (report/migrate.ts:106), readReportDocument (report/migrate.ts:156).
|
||||
|
||||
Rust counterparts from R2: `crates/core/src/domain/sheet/{schema,validate}.rs` and `domain/report/{migration_map,v2,v3,read}.rs`.
|
||||
|
||||
## 8. Tests
|
||||
|
||||
Declarations counted by grep; `it.each` rows expanded by hand. (The orchestrator ran the suites: anatomy 131/131 in 16 files, domain 262/262.)
|
||||
|
||||
| Suite (lines) | Decl | Cases | What it asserts | Files read | Render dep |
|
||||
|---|---|---|---|---|---|
|
||||
| curation.apply (529) | 29 | 41 | each op and its records; error messages carry `CHG-001 op 0`; determinism (:425); purity (:429); B/C rule | fixtures/mini-sheet.ts | no |
|
||||
| curation.assets (79) | 5 | 5 | 30 annotated elements + titles removed, 270 paths kept, declaration kept, style removed, deterministic + XML re-parse, sha, parseViewBox | legacy-sources/Horse.svg | no |
|
||||
| curation.migration-leaves (271) | 8 | 8 | 612 leaves: 442 identical, 141 moved (134 + 7), 29 removed; 26 observable annotations; spine regions | committed legacy-import/document.json, draft/document.json, draft/report-migration-map.json + domain migrate | no |
|
||||
| curation.migration-map (183) | 5 | 5 | schema-valid map, fates, every target exists; synthetic rename/move/remove/re-home | runDraft, mini-sheet | no |
|
||||
| curation.overlay (83) | 4 | 4 | moved/removed markers, legend, detail framing | mini-sheet | no |
|
||||
| curation.pipeline (110) | 6 | 6 | 0 integrity error, 1 asset, head view_box 60 100 500 380, Cd 15–21, carpus, 811 nodes, CHANGELOG rows, 10 headings, SQL never publishes, CHG-000 refused | runDraft + renderDraftArtifacts | indirect |
|
||||
| curation.schema (85) | 5 | 9 | committed curation files parse; rejections; duplicates | curation/*.json | no |
|
||||
| document (23) | 2 | 2 | 0 issue; determinism | real sources | no |
|
||||
| importer.counts (103) | 8 | 8 | 27/26, 2 views 1 trigger, 5 spine codes, 9 bilateral, 792/612/41 on both sides | sources + regex over the SVGs | no |
|
||||
| importer.guards (47) | 3 | 3 | guard messages; dollar-quote tag | inline SVG, real import | no |
|
||||
| importer.special-cases (128) | 10 | 10 | region, laterality, trigger, ssb, anamnesis, lean form, authored labels | real import, pathologies.json | no |
|
||||
| legacy.regions (43) | 4 | 4 | inferRegionCode | none | no |
|
||||
| r3.layer-oracles (162) | 1 | 1 | 8 layer trees equal the frozen JSON | crates/core/tests/oracles/sheet.draft.json and r3/layer.json | direct |
|
||||
| render.compose (230) | 10 | 11 | highlight, halo `<defs>`, framing; app-{main,head}.svg string-equal | runDraft, draft/assets/horse.svg, draft/review/app-*.svg | direct + xmldom |
|
||||
| render.layer (158) | 5 | 5 | pair/zone/trigger, styles, states, DOM + serializeLayer | runDraft, mini-sheet | direct |
|
||||
| seed.golden (87) | 6 | 9 | see below | both generated folders | indirect |
|
||||
| Total | 111 | 131 | | | |
|
||||
|
||||
Golden test (`seed.golden.test.ts`): `beforeAll` re-runs the import and the draft in memory (:29-34); legacy-import: 4 files compared as `readFile(…,'utf8')` string === fresh render (:37-44); no SQL file in legacy-import (:46-50); the SVG copies compared with Buffer.equals against the sources, plus sha and size (:52-62); draft: one `it` loops over every file `renderDraftArtifacts` returns, comparing strings (:66-70); the draft folder must hold exactly the produced files plus `assets/<file>` (:72-78); horse.svg text === cleanedSvg, plus sha and size (:80-86). No comparison is done on parsed data.
|
||||
|
||||
Determinism tests: `document.test.ts:16-22` (two imports, toEqual, plus `renderArtifacts`), `curation.apply.test.ts:425`, `curation.assets.test.ts:46`. No test rendered the draft folder twice before R5.1 (the golden pins it against the committed bytes); `r5.pipeline-oracles.test.ts` now does.
|
||||
|
||||
## 9. Consumers
|
||||
|
||||
`@faunosteo/anatomy` outside the package: `package.json:28` (script filter) and `packages/README.md:12` only. No import in apps/, packages/*, scripts/, crates/, docker/, the Dockerfile, the vite configs or the eslint configs. Rust cites it in comments only (`crates/core/src/anatomy/{layer,compose}.rs`, `tests/anatomy_layer.rs`).
|
||||
|
||||
`@faunosteo/domain` outside its package: imported by `packages/anatomy` only (package.json, 22 source files, 8 test files). `apps/app/eslint.config.js:68-72` bans it in the app. `apps/app/tests/unit/r4.census.test.ts` :41-57 and :59-78 assert that apps/app and packages/core-wasm neither import nor depend on it; :80-90 asserts the lockfile lists `@xmldom/xmldom` under exactly `['packages/anatomy']`. `packages/core-wasm/tests/n-1` is a verbatim copy, not an import.
|
||||
|
||||
Path-level dependents: `crates/core/tests/report_rules.rs:141-143` reads report-v2.full.json (5 times) and report-v2.minimal.json (4 times); `archive_rules.rs:204,603,801,820,911` reads `packages/domain/tests/fixtures`; `archive_support/mod.rs:81` documents horse-slice.ts as its source.
|
||||
|
||||
Consumers of the generated artifacts: the app (`apps/app/vite.config.ts:13` + `vite/sheet-bundle.ts:46-63` bundle document, sheet-ref, migration map, assets.json and the assets; `apps/app/tests/helpers/legacyLeaves.ts:17` reads legacy-import/document.json); the core-wasm test suites (runtime, anatomy, measure, serialisation, pdf-import, pdf-import.budget, pdf-export, pdf-export.session); `scripts/browser/lib/core-module.mjs:25`; the crate tests (anatomy_canvas_op, anatomy_compose, sheet_index, sheet_rules, sheet_contract, editor_rules, fixtures_round_trip, the pdf_* suites, archive_support); the database (`supabase/config.toml:44` lists equine_draft.sql in sql_paths; `supabase/tests/007_equine_seed.sql` plans 19 pgTAP checks; the PGlite harness `packages/supabase/tests/db/harness.ts:21`, `seeds.test.ts`); the crate oracles `sheet.{draft,legacy-import}.json` and `migration-map.draft.json` carry the seed path as `source`.
|
||||
|
||||
## 10. Upload tool
|
||||
|
||||
`packages/supabase/scripts/upload-sheet-assets.ts` (99 lines): reads `<dir>/assets.json` (:48-53); validates each entry (plain file name, sha `/^[0-9a-f]{64}$/`, id `/^[0-9a-f-]{36}$/`, :28-40); re-hashes each file and compares sha and size (:56-63); uploads `<sha256>.svg` to bucket `anatomical-sheet-assets` as image/svg+xml with upsert false, an "already exists" error tolerated (:77-84); calls RPC `confirm_sheet_asset_upload(asset_id)` (:86-89). Needs `SUPABASE_URL` and `SUPABASE_SERVICE_ROLE_KEY` (:42-46, :71-75), resolves the directory from INIT_CWD (:67-70), imports @supabase/supabase-js plus node:crypto, node:fs and node:path. It imports nothing from packages/anatomy; it depends only on the assets.json contract.
|
||||
|
||||
## 11. ANATOMICAL_SHEET_MODEL.md (docs/development)
|
||||
|
||||
§6 starts at :304; the folder table at :313-317; changeset fields, ops and target grammar :319-331; asset cleaning and CHG-000 :333-337; review drawings :339-344; Step 4c content (:346) says "44 changes + CHG-000, 48 references" while the file now holds 45 authored changes (CHG-045, colours, documented at :401-404); head view :368-375; status semantics :377-384; migration map (:393-399): 792 = 571 identical + 180 renamed or moved + 7 re-homed + 34 removed, matching CHANGELOG.md:35-43 and the committed map; seed SQL :406-410; import rules :412-429; validation tiers §2.1 at :108.
|
||||
|
||||
## 12. Risks (analyzer)
|
||||
|
||||
- Any byte change in draft/document.json changes documentSha256: formatter, key order, number text or the trailing newline. That rewrites 18 pinned files and makes every older archive rebase once (certain for a Rust emitter under D-R5 as written).
|
||||
- Typed serde round-trips emit schema key order, not the working form's insertion order (e.g. `review_status` last), which produces a byte diff (high).
|
||||
- Prettier's JSON layout and its Markdown padding (string-width function) are not reproduced by serde_json or a plain padder (high if bytes must be preserved, none if the files are regenerated by decision).
|
||||
- Re-cleaning Horse.svg with the Rust reader (roxmltree plus its own serialiser, BDR-053) instead of xmldom changes the horse.svg bytes. That cascades into a new asset sha and svg_asset_id, a document change, new SQL, a new Storage object to upload, and changes to the anatomy_compose and pdf oracles that read horse.svg (medium to high).
|
||||
- Deleting packages/domain breaks the crate tests that read its fixtures (certain unless the fixtures move).
|
||||
- `r4.census.test.ts:80-90` fails once xmldom leaves the lockfile (certain).
|
||||
- Edge-input differences (Unicode `\s`, `Number('')`, `split` limit, `Math.round` sign, UTF-16 slice) are not triggered by today's inputs (no NBSP, tab, CR, BOM, astral character or exponent found) (low).
|
||||
- NFD and lowercase table differences between ICU 78.3 and a Rust crate: slugify never fires on today's data (low).
|
||||
- Prettier Markdown escaping could alter more than padding on future content (today: padding only) (low to medium).
|
||||
|
||||
## 13. Open questions (analyzer)
|
||||
|
||||
- Which fingerprint rule binds R5: D-R5 as written, or the BDR-045 amendment?
|
||||
- Byte-preserving port or one-time regeneration? Either the 18 files are reproduced byte for byte (Prettier JSON layout, Markdown padding, xmldom serialiser, insertion order), or they are regenerated once and checked by parsed-tree comparison.
|
||||
- horse.svg: keep the xmldom-cleaned bytes (asset 7f767f74…) as a frozen input, or re-derive them through the Rust reader (new sha, operator upload)?
|
||||
- Where do `packages/domain/tests/fixtures/report-v2.*.json` live after `packages/domain` is deleted?
|
||||
- Upload tool HTTP crate: the review (:300) names reqwest, the worker uses ureq. Which one?
|
||||
- Prettier on the seed: should `supabase/seed/**` join `.prettierignore` (today a root `pnpm format` rewrites the generated JSON and Markdown)?
|
||||
- Stale comment: `vertebrae.ts:3` points to `apps/app/src/domain/species/equine/vertebrae.ts`; the source is `legacy-sources/vertebrae.ts`.
|
||||
@@ -509,3 +509,18 @@ Decision: J1 + J2 declared PASS as a whole on iPhone Safari iOS (no per-check sp
|
||||
Alternatives: wait for per-check iPhone measures (rejected by owner: device-level declaration suffices for merge, measures = follow-up); regenerate goldens on any gap (forbidden).
|
||||
Recorded: RUST.md §10.15 (risk table + checks), PDF_IMPORT_HARDENING.md §7, R4_J1/J2_RESULTS + MINI_RECETTE, CHANGELOG headers, TODO R4.13.
|
||||
Status: active; limits stay: no release 0.2.0, no deploy, no DNS/server change, no R5 before next owner signal, D2/D4/D5 deferred.
|
||||
|
||||
## BDR-068 | 2026-10-22 | R5 owner decisions: D-R5 = RFC 8785 of validated doc, aliases, F1 fix, byte-exact port, frozen SVG bytes
|
||||
|
||||
Context: owner validated R5.1 (« Validation R5.1, décision D-R5 et lancement du portage Rust »).
|
||||
Decision: D-R5 = option B, convention `sha256-rfc8785-validated-v1` = sha256 of RFC 8785 canonical JSON of the validated document (defaults filled, canonical model), real RFC 8785 (UTF-16 key sort, ES number print, UTF-8, no Unicode normalisation, I-JSON refusals, Rust/JS deterministic) proven by official vectors + edge cases + independent differential; `v1` freezes the construction and validation rules (later defaults/schema never change v1 identity retroactively). Alias mechanism approved: Report v3 `sheet.documentSha256` stays 64 hex; new reports stamp current convention; old reports open with no rebase on identical document; aliases only from trusted frozen historical documents, never from an archive or bundle; unknown fingerprint ≠ official sheet; real anatomical change still migrates; no archive/fixture/oracle rewritten. F1 fix authorised first. Port byte-exact by default, differences declared one by one; fingerprint-bearing artifacts regenerated once at R5.6 only. Historical SVG bytes (horse.svg) kept, never re-serialised. `.prettierignore` on Rust-written generated paths, narrow, with a reproducible check. Upload tool reuses `ureq` (no reqwest by default) at R5.5. Slicing R5.1b to R5.7 validated; stop after R5.2.
|
||||
Alternatives: A (Rust emitted bytes, re-ties identity to a serialiser), C (Rust reproduces Prettier bytes, keeps formatter chain); one-time visible rebase of every old archive (rejected: transparent alias possible).
|
||||
Status: active. R5.1b + R5.2 done 2026-10-22; documentSha256, seed, report contracts untouched until R5.6; no merge, no package deletion, no 0.2.0, no deploy.
|
||||
|
||||
## BDR-069 | 2026-10-24 | Refuse three non-canonical v3 report shapes at reader entry
|
||||
|
||||
Decision (owner go): `check_v3_shape` on raw value before `Report::deserialize`: `sheet`, `practitioner` must be objects, `severity` string. At `read_report_document` v3 + `parse_envelope`. Rationale: serde derives accepted positional arrays / one-key enum maps Zod refused → crafted file could swap same-typed fields silently. Alternatives: generic shape guard (draft/shape.rs style) on whole contract (wider, not authorised); `deserialize_with` in contract structs (touches contract). Status: applied d502dfa; v2 + other contracts keep serde leniency (not in go). Owner confirmed fingerprint convention `sha256-rfc8785-validated-v1`, one alias `27c7383c…2660`; R5.6 scope files approved, switch not authorised.
|
||||
|
||||
## BDR-070 | 2026-10-25 | D-R5.6 active: fingerprint `sha256-rfc8785-validated-v1`, registry alias, M1 option 1
|
||||
|
||||
Decision (owner, activation 9e87016): sheet fingerprint = SHA-256(JCS RFC 8785 of R2 typed reading, no integrity error, warnings allowed, depth 128). Equine canonical 12d78872…74e2; old 27c7383c…2660 = sole TRUSTED_ALIASES entry, bound to sheet 503dc301 + content. Alias honoured only via LoadedSheet::recognises_historical_stamp (loader-verified v1 private state + document recomputed at decision + one-way registry pair + same sheet id); equivalence pub(crate). M1 option 1: snapshot of old report keeps its stamp (build_sheet_snapshot); export, archive reader strict. v0 refs (no block) load unverified, no privilege; null/malformed block refused. Rust = sole seed writer, TS legacy until R5.7. Alternatives rejected: A file-bytes (Prettier-bound), C Prettier reproduction; re-stamp at rebase (rewrites reports); export-side alias (M1 defect, no proof). Status: active on feature branch; rollback not transparent for post-switch reports (rebased, nothing lost) → settle before user release.
|
||||
@@ -192,3 +192,52 @@ Method: memory bench 3 engines × demo/L × 6 exports + Chromium L 16 + raster c
|
||||
Result: verifier pass 1 ECARTS(1): core with no feature did not compile (`restore_copy` used gated `pdf::read` const; workspace feature unification via crates/wasm hid it in rust-check/check:all) → gated + no-feature clippy in rust-check; pass 2 ECARTS(1): §6 lacked J2 merge-gate row → added. Security pass 1 BLOCK(1): `.gitleaks.toml` allowlist (R4.1, owner-approved in journal) lacks owner `[gated]` marker → owner action; MEDIUM serve.mjs `--host ""` bound every interface.
|
||||
Anomalies: my first host fix = exact-string denylist → bypassed by `0`, `0::`, `::ffff:0.0.0.0`, scoped `::` (re-gate 1); rebuilt resolve-then-BlockList; re-gate 2 found post-bind check dead for IPv6 (`family` 'IPv6' vs 6) and my fstat-first CLI read opened FIFOs before the type check (regression). Each fix of a gate finding introduced the next finding: 3 re-gates for 2 small fixes. Contract CHECKs used root vitest path (absent): oracle defect.
|
||||
Action: owner checkpoint; floor 21 DELETED_TEST (R4.8 deletions, approved in brief §1) need owner `[gated]` clarification line to clear formally.
|
||||
|
||||
## EVAL-026 | 2026-10-22 | R5.1 freeze + D-R5 study — analyzers ×2, oracle generator, cross-stack digests
|
||||
|
||||
Method: 2 analyzer agents (pipeline inventory 43 files, fingerprint census: producers, consumers, 99 PDFs, docs), both refused to write files (role rule) → orchestrator saved reports; generator test (`MAKE_R5_ORACLES=1`) + freeze re-run green 6/6, anatomy 137, domain 262; Node scripts (matrix, PGlite jsonb) + scratchpad Rust binary on core crate: digests compared value by value; tie census 200k values.
|
||||
Anomalies: R2 oracle test self-referential on number printing (LRN-070) → F1 undetected since R2; docs contradict (BDR-045 vs RUST.md §10.10 / TODO "will change once") → cross-reference added; analyzer inventory counted "10 anomalies" wrong in orchestrator draft (actual 27, records 825 not 690, 16 ledger artifacts) → corrected from oracle counts before commit.
|
||||
Action: F1 fix proposed, owner go; D-R5 recommendation B + aliases; checkpoint delivered, STOP.
|
||||
|
||||
## EVAL-027 | 2026-10-22 | R5.1b + R5.2 gates: check:all, security, verifier
|
||||
|
||||
Method: check:all green 21:16:39 (app 434, core-wasm 58, domain 262, anatomy 141, supabase 40, curation suites, worker + images); security-auditor semgrep 129 rules / 36 files → PASS, 0 HIGH, 5 LOW; fresh verifier on contract `2026-10-22-rust-r5-f1-importer.md` → CONFORME 12/12 (ran cargo workspace + curation suites + import-equine --check itself, red-before proven by rustc probe).
|
||||
Anomalies: LOW report.rs `Map` index panics on hand-built ImportResult → fixed (get chain) after verdict, importer suite + --check rerun green; LOW quadratic vertebrae scan, symlink-following `listed()` in --check, lint allows (accepted, operator inputs). check:all ran on pre-commit tree (code identical to 03bdb5e). Verifier floor note: plan B.3.5 "(proposal, not implemented)" heading from R5.1, intended.
|
||||
Action: checkpoint R5.2 delivered, STOP before R5.3.
|
||||
|
||||
|
||||
## EVAL-028 | 2026-10-23 | R5.3 gates: fresh verifier ×4 + security; successive passes found real defects
|
||||
|
||||
Method: check:all after each fix round; security-auditor (semgrep 725 rules, checklist) → PASS, 6 LOW (3 fixed: Map index, destination claimed + canonicalised, INIT_CWD absolute only); fresh verifier each round, no history. Pass 1 ECARTS(1) (floor on a plan heading) + 2 unreachable divergences aligned; pass 2 null-on-optional (LRN-072); pass 3 (first attempt died on API credit error, rerun on Opus) repeated keys refused by serde derive, NUL touch-mark silent drop (LRN-073), `--out symlink/` trailing slash followed the link; pass 4 C1-C12 met, C13 docs gaps (lossy UTF-8 read, 2 contrived splits undeclared, stale counts).
|
||||
Anomalies: green byte parity on real data hid every finding: all were hostile-input or file-protection defects. A verdict reported after post-pass edits was not re-verified → owner required a fresh pass on the final state.
|
||||
Action: refusal tests now compare reasons by stage; Object.prototype characterised + tested; migration-map debt characterised; final fresh verifier + security on the final R5.3 state before R5.4. Next ports: budget ≥3 verifier passes, probe hostile inputs (null, duplicates, reserved names, symlinks, encodings) in the oracle from the start.
|
||||
|
||||
## EVAL-029 | 2026-10-23 | R5.4 gates: 14 rounds (verifier + security + check:all) to converge
|
||||
|
||||
Method: each round fresh verifier (opus, contract from disk, own TS-vs-Rust differential) + fresh security-auditor (semgrep + checklist + memory probes) + `pnpm check:all`; superseded runs stopped. Rounds 1-5: Markdown guard families (denylist → allowlist at round 3); rounds 6-11: resource bounds (table amplification, records per op, string headers, drawings per annotation, fixed parts); 12-14: doc precision only. Final: c4e717d security PASS (LOW only), check:all green 16:41:34; eb667ad (docs/comments only over c4e717d) verifier all 9 clauses MET, ECARTS(13) = floor-guard hits on audit prose quoting suppression keywords (product paths clean).
|
||||
Anomalies: docs repeatedly claimed bounds "before any row is built" / "tens of MB" that the next audit disproved; counts drifted after each fix; FLOOR family needs a contract Clarification (human gate), never an audit edit.
|
||||
Action: LRN-074, LRN-075; checkpoint R5.4 to owner with decisions (FLOOR clarification, curation input cap vs 65× parse, bare URLs autolinked, 2 declared quadratic scans), STOP before R5.5.
|
||||
|
||||
## EVAL-030 | 2026-10-23/24 | R5.5 gates: 2 rounds + owner real-stack recette
|
||||
|
||||
Method: fresh verifier + security-auditor (opus; pinned Fable agents died on usage credits, re-dispatched), check:all per round; owner ran real-stack e2e by hand.
|
||||
Anomalies: round 1 verifier ECARTS(4) all real (oversize body → Network+retry, "retried 3×" = 3 attempts, runbook exit codes, FLOOR script never OK); security M1 key on argv in e2e script. Round 2 security 5 LOW (Protocol retried, broadcast DEV addr accepted). Census doc claim "earlier revisions" wrong → corrected before checkpoint.
|
||||
Action: fixes + negative tests; owner closed R5.5 2026-10-24 (e2e PASS 11/11, owner's run). LRN-076.
|
||||
|
||||
## EVAL-031 | 2026-10-24 | R5.6 prep: claimed rehearsal green after extending the patch
|
||||
|
||||
Method: fresh verifier applied patch in scratch copy, ran every feature set + switched-seed simulation; security read patch.
|
||||
Anomalies: docs/TODO said "rehearsed green"; green run covered 5-consumer wiring only, then generator + loaded.rs added without rerun → report-compare E0063, 4 curation tests failing unreported; switched seed never simulated (+3 failures incl. frozen R4 O7). Security M1: alias-path export writes report 27c7 + snapshot 12d7 → pre-switch/N-1 readers drop snapshot.
|
||||
Action: rerun gate after every patch extension, claim only measured state; patch fixed (build, strict block, NotCanonical); docs corrected; M1 to owner.
|
||||
|
||||
## EVAL-032 | 2026-10-25 | R5.6 rehearsal: docs claimed export guarded by fingerprint_verified
|
||||
|
||||
Method: isolated worktree (patch + switched seed), rust:check + check:all green, fresh security + verifier (opus), verifier probe on core export API.
|
||||
Anomalies: RUST.md said M1 "needs fingerprint_verified" for every consumer; export path (input.rs) never sees LoadedSheet → v0 sheet snapshot gets alias at export (app unreachable, API reachable). Fix needs snapshot_rules.rs, outside owner's 8-file scope → stopped. Also `pnpm format` rewrote append-only audits in worktree (reverted).
|
||||
Action: claim a guard per call site, not per design; owner decision on snapshot_rules.rs; never run format over .claude/.
|
||||
|
||||
## EVAL-033 | 2026-10-25 | R5.7 closure: deletion gates + pre-merge rerun
|
||||
|
||||
Method: ledger 407 TS cases case by case before `git rm`; fresh verifier + security (opus) twice; check:all on R5.7 state, then rerun on 27bf59c after post-check edits; verifier ran 14 scratch reintroduction probes against census.
|
||||
Anomalies: round 1 verifier ECARTS(2): contract C6 "tsx gone" false vs pnpm optional peer (owner Clarification), digest typo `…dfe7` in docs. Two edits landed after first check:all → owner required full rerun. Census regex accepted `#[ignore]` targets (fixed) and still accepts ignore-before-test (LOW). Docs pointed to premerge record before it existed; `.bin/tsx` wording root-only; SUPABASE.md layout stale since R2. `pnpm audit` never run before this round (cargo advisories only): 57 pre-existing.
|
||||
Action: LRN-078, LRN-079; rerun gate after any post-check edit; run `pnpm audit` in every pre-merge round; checkpoint to owner, STOP before merge.
|
||||
@@ -318,3 +318,65 @@ Owner verdict: J1.7 PASS w/ reserve, M3 complementary, J2.3 option 1 final, logo
|
||||
## 2026-10-21 (R4 closed and merged)
|
||||
|
||||
Owner: J1 + J2 PASS on computer, Android, real iPhone Safari iOS (declared as a whole; model/versions/timings/memory not given → marked missing, not invented); R4.12 validated; merge authorized → BDR-067. Consolidation 3a7937e: R4_J1/J2_RESULTS + MINI_RECETTE, RUST.md §10.15 (risk table, deadlines, final checks), PDF_IMPORT_HARDENING §7 (items 3/19 unblocked, 5/6/7/9/20 open), CHANGELOG headers, TODO R4.13. Final checks on 22bbfb4 (code identical through 3a7937e): check:all green 22:15:50 (app 434, domain 262, core-wasm 58, anatomy 131, supabase 40, worker TLS + debian13 digest), audit --prod clean, gitleaks 111 commits clean (tree: ignored `.env.worker` only), no local path/personal id, merge-tree clean, goldens 8b86b92 / oracles 4439fd2 / digests d492bf7 untouched, 51 deletions all 0dc1ba2. `gitflow finish feature rust-r4-pdf-core` → merge 96ae75e (parents 66909b6 + 3a7937e, 113 commits), develop pushed = origin, branch deleted local + origin, main 605ee7c untouched, release/0.2.0 untouched. Not done: release 0.2.0, deploy, DNS, R5, D2/D4/D5. Next: owner signal for R5 (entry: plan D-R5, `packages/anatomy` pipeline → `crates/core`, seed golden regenerated once).
|
||||
|
||||
## 2026-10-22 (R5.1 freeze + D-R5 study)
|
||||
|
||||
Owner: R4 bilan validated, R5.1 authorised (freeze TS pipeline, study D-R5, stop before Rust pipeline + fingerprint change). Branch `feature/rust-r5-pipeline-core` from 9c0117a. Two analyzers (inventory, documentSha256 census) → `.claude/audits/R5_*.md`. Oracles `oracles/r5/{pipeline,ids,fingerprints}.json` via `r5.pipeline-oracles.test.ts` (MAKE_R5_ORACLES=1; 6 cases, determinism two runs, ledger vs committed). Study scripts `scripts/r5-fingerprints.mjs`, `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`; throwaway Rust experiment (scratchpad) reproduced every draft digest byte-for-byte, JCS 40 lines no crate, RFC vectors pass; jsonb round trip keeps B only. Found F1: `js_number` tie rounding (Grisu up vs ES6 even; `840.1692504882812`, 7.4 % of float32 values; `zmij` matches) hidden by self-referential R2 oracle test → LRN-070, EVAL-026, not fixed (owner go). Plan `2026-10-22-rust-r5-pipeline-core.md`, RUST.md §11, TODO R5.1 ticked. Recommendation: B on validated form + alias mechanism (old `27c7…` stays valid, no visible migration). STOP at checkpoint. No Rust pipeline, no fingerprint change, no merge.
|
||||
|
||||
## 2026-10-22 (R5.1b F1 fix, R5.2 legacy importer in Rust)
|
||||
|
||||
Owner validated R5.1, decided D-R5 = B `sha256-rfc8785-validated-v1` + aliases, F1 go, byte-exact port, frozen SVG bytes, ureq at R5.5 → BDR-068. R5.1b aed6b1e: `js_number` digits from `zmij` (edge only, no new crate), V8 oracle 44,322 rows (1,552 mismatches before → 0), `zod_oracle` now checks oracle bytes (red on old printer), rust:check + wasm + check green, no oracle/golden/digest moved. R5.2 03bdb5e: `crates/core/src/curation` (feature `curation`, 3,187 lines: Prettier string width table transcribed, JSON layout rules, table alignment, slugify via icu_normalizer NFD, uuids, legacy importer, IMPORT-REPORT) + `import-equine --check|--out`; legacy-import/ byte-exact (6 files), pipeline.json legacy parts equal, 10 synthetic TS cases, 968 Prettier probes; declared refusals: emoji width, malformed SVG, wrong-typed catalog nodes. Docs 82f1664 (RUST.md §11.9-11.11, CHANGELOG), contract `2026-10-22-rust-r5-f1-importer.md`. Gates: check:all green 21:16:39, security PASS (5 LOW, report Map-index panic fixed), verifier CONFORME 12/12 → EVAL-027. STOP after R5.2 checkpoint.
|
||||
|
||||
## 2026-10-22/23 (R5.3 curated draft in Rust)
|
||||
|
||||
Owner validated R5.1b/R5.2, go R5.3, named `pnpm r5:import-equine`. e0f149b: `crates/core/src/curation/draft` (2,718 lines: schema, targets, fields, records, ops, rename, apply, migration_map, stats, seed, pipeline) + `import-equine` both folders + `scripts/import-equine.sh`; draft 5 texts + frozen horse.svg byte-exact (`--check` 12/0), 825 records, 46 changes, stats, migration stats 792/571/180/7/34, 612 leaves 442/141/29, 58 synthetic TS cases (`oracles/r5/curation-synthetic.json`, 9 byte-exact, 49 refused alike). Security PASS (6 LOW, 3 fixed). Verifier 4 passes: found null-on-optional, repeated keys, NUL touch-mark drop, symlink trailing slash, lossy UTF-8; all fixed or declared (RUST.md §11.12); pass 4 C1-C12 met, C13 doc gaps fixed after, not re-verified. check:all green 00:10:03. documentSha256 unchanged. STOP at R5.3 checkpoint; R5.4 on owner go.
|
||||
|
||||
## 2026-10-23 (R5.3 closed after 6 final gate rounds)
|
||||
|
||||
Owner validated R5.3 functionally, asked a fresh verifier on the final state, Object.prototype characterisation, migration-map debt kept + characterised, LRN/EVAL. 9901106/8654905: Object.prototype member = plain key, safe, test; debt test from TS-frozen cases; refusal reasons compared by stage; LRN-072, LRN-073, EVAL-028. Then 6 rounds (verifier + security + check:all each) on the final state, each finding real edge families via TS-vs-Rust fuzzing (up to 150,000 changesets): NUL field segment silent drop, stack overflow (8 segments, 64 levels, 200,000 values), non-exclusive --out, serde struct-as-array / enum-as-map (draft/shape.rs; R2 derives raised to owner, gated readers), non-list/non-object children, JSON.parse key order + big ints, __proto__ segments, seed identity guard, reads confined to seed. Sixth round c6e0d85: CONFORME 13/13, security PASS LOW only, check:all green. LOW leftovers carried into R5.4. Next: R5.4 (owner go conditional on these checks: met).
|
||||
|
||||
## 2026-10-23 (R5.4 review files in Rust, 14 gate rounds)
|
||||
|
||||
R5.4: `draft/{labels,changelog,review_report,drawing,overlay,app_review,svg_compare}.rs` + `curation/report_guard.rs` (allowlist guard); CHANGELOG 1,114,533 B, REVIEW-REPORT, 3 drawings byte-exact, app-main.svg = R3 trigger-name rule only; `--check` 6/0 + 12/0, 1 declared. Gates: rounds 1-5 Markdown families → allowlist + 798-probe TS sweep; rounds 6-11 bounds (records 16 Mi weight charged per record, report tables 32 Mi summed, drawings 64 MiB projected per annotation + fixed parts); 12-14 docs. Final c4e717d security PASS, check:all green; eb667ad verifier 9/9 MET, only audit-prose FLOOR. LRN-074, LRN-075, EVAL-029. STOP at R5.4 checkpoint; R5.5 on owner go.
|
||||
|
||||
## 2026-10-23 (R5.5)
|
||||
- R5.4 adjustments: FLOOR clarification + negative-proof script (21 audit prose requalified, evals.md:205 STUB declared), 4 MiB cap changes/references (+39 MB parse at 3.98 MB), serde tolerance characterised, no reader changed.
|
||||
- R5.5: `upload-sheet-assets` Rust (core rules pure, bin ureq), local-only destination, plan before write, no overwrite, confirm after verify; 12 stub tests + 7 unit.
|
||||
- Blocked: own run vs real local stack refused by permission layer, not retried; owner runs `pnpm e2e:sheet-assets`. Fable gate agents out of credits → re-dispatched on opus.
|
||||
- Gates: verifier ECARTS(4)→fixed→C1-C13 MET; security PASS ×2; check:all green 21:47:11. STOP at checkpoint.
|
||||
|
||||
## 2026-10-24 (R5.5 close)
|
||||
- Owner closed R5.5: real-stack e2e PASS 11/11 (owner's manual run). LOW fixes: Protocol final, broadcast DEV addr refused.
|
||||
- BDR-069 serde: 3 v3 shapes refused at reader entry; all R2-R4/PDF/N-1 suites green unchanged.
|
||||
- Found: DEV stack on CLI default JWT secret + 54321 LAN-open → LAN service_role (ENVIRONMENTS §3.6, owner decision). SQL confirm characterised, unchanged.
|
||||
- R5.6 plan written, 4 decisions open. STOP.
|
||||
|
||||
## 2026-10-24 (R5.6 prep)
|
||||
- Owner: R5.5 validated, D-R5.6 decided (compiled registry authority, 5 consumers, Rust sole generator at switch). DEV key risk accepted temporarily, separate hardening task.
|
||||
- Fingerprint module inert + 9 tests; Rust draft value = frozen 12d7…74e2 first try. Switch as rehearsed patch (green), tree restored, --check 0 diff.
|
||||
- Rehearsal found: fingerprint before drawings bound (order), loaded.rs/facade.rs outside approved files. STOP before switch.
|
||||
- Correction: "patch rehearsed green" was false (green run predated generator/load parts; patch broke report-compare build). Verifier caught it; patch fixed, docs corrected. Lesson → EVAL-031.
|
||||
|
||||
## 2026-10-25 (R5.6 rehearsal)
|
||||
- Owner: M1 option 1, 8-file scope, v0 kept without alias privilege. Rehearsed in detached worktree, seed switched there only (copy approved by owner at prompt).
|
||||
- M1 in input.rs; equivalence one-way + fingerprint_verified (v1 recomputed at load, sha2 in wasm). Goldens reproduced byte-exact on switched sheet.
|
||||
- Original contexts via pre-switch fixtures (O7, O10, R2, ledger, TS + app goldens); check:all 00:56:22 green switched.
|
||||
- Slip: `pnpm format` rewrote append-only audits in worktree → reverted. Never format over .claude/.
|
||||
|
||||
## 2026-10-25 (R5.6 M1 fix)
|
||||
- Cause M1: export sees only snapshot. Fix in build_sheet_snapshot via recognises_historical_stamp (private verified + recompute at use); input.rs strict, unchanged.
|
||||
- null block refused, warnings allowed (is_publishable), canonical depth 128. Fresh worktree final: all gates green, security PASS, verifier CONFORME 8/8. STOP before activation.
|
||||
|
||||
## 2026-10-25 (R5.6 active)
|
||||
- Owner authorised switch. Patch applied (sha = verified worktree), equivalence pub(crate), comments + trust boundary. Seed: fresh --out, 2 files copied (03e7f6ab, 0e991bc2), --check 0 diff.
|
||||
- Switch commit 9e87016: rust:check + check:all green; security PASS LOW; verifier 13/14, ECARTS(1)=S7 pre-activation wording → Clarification.
|
||||
- Rollback measured: new v1 report on pre-switch build → rebased, 0 unresolved, notice shown. BDR-070, LRN-077. STOP, no merge, no R5.7.
|
||||
|
||||
## 2026-10-25 (R5.7)
|
||||
- Owner validated R5.6, authorised R5.7. Census: anatomy 144 + domain 263 TS cases mapped; 15 anatomy contracts + 1 domain oracle only TS checked → ported to Rust first (+19 tests, curation_leaves.rs).
|
||||
- Deleted anatomy, domain, TS upload script, legacy scripts; LEDGER.tsv 407 (256/149/2), FILES.tsv 112; fixtures moved sha kept; xmldom out; tsx left as vite optional peer.
|
||||
- TS replay rehearsed at 59521dc + 756b7ef. check:all green 03:58:13; security PASS 2 LOW; verifier ECARTS(2): tsx residue (owner), digest typo fixed. STOP before merge.
|
||||
|
||||
## 2026-10-25 (R5.7 pre-merge)
|
||||
- Owner validated R5.7, accepted tsx as vite optional peer → C6 Clarification. Census now refuses `#[ignore]` ledger targets (cd69f0e). 149 replaced rows checked active, 2 deletions checked.
|
||||
- Rerun on 27bf59c: frozen install, dedupe, check:all green 13:41:41, --check 0 diff, report:compare ok; security PASS (pnpm audit 57 pre-existing, MEDIUM); verifier CONFORME 10/10. LRN-078, LRN-079, EVAL-033. STOP before merge.
|
||||
@@ -519,3 +519,57 @@ Apply: when an implementation is swapped under a test that spied the old one, re
|
||||
|
||||
- Context7 said nginx `listen ... http2` deprecated → `http2 on;`, but live VPS sends `Server: nginx/1.24.0` and 1.24 rejects `http2 on` (`unknown directive`). Apply: before changing a runbook directive, read the target's version (Server header, `nginx -v`) and `nginx -t` both forms in throwaway containers of each version; document both when the fleet straddles.
|
||||
- WHATWG URL normalisation can create the escape it was meant to catch (`/..//host` → pathname `//host`, same origin for the probe). Apply: sanitiser re-validates the value it returns (not only the input), plus an idempotence test; fuzz with dot segments, `%2e`, tabs inside `..`.
|
||||
|
||||
## LRN-070 | 2026-10-22 | Rust `{:e}` shortest digits ≠ ECMAScript on exact decimal ties; oracle that re-prints expected through the port cannot see it
|
||||
|
||||
Found by R5.1 study. `js_number` (core json.rs) takes shortest digits from `format!("{:e}")` (Grisu): exact tie between two shortest candidates → Grisu picks upper, ECMAScript/Python/`zmij` pick even. `840.1692504882812` (= 840.16925048828125, float32-exact, Horse.svg legacy cx) → Rust `…813`. Rate: 7,401 / 100,000 float32-exact values in 0..2000, 769 / 100,000 random doubles, `zmij` (serde_json 1.0.154 formatter, locked) 0. Draft seed has no tie → every R2/R3/R4 suite green, but `sheet_contract.rs` `zod_oracle` computes expected = `to_js_pretty(oracle.output)` → same function both sides → blind. 134 JSON fixtures Node vs Rust pretty: 2 differ (legacy baseline, that value).
|
||||
Apply: shortest-float port → take digits from `zmij`/ryu-class crate, never `{:e}`; tie vectors (float32-exact values) in unit tests; oracle expected must be the recorded JS text, never re-printed by the port under test. Any R5 importer emitting legacy coordinates needs the fix first.
|
||||
|
||||
## LRN-071 | 2026-10-22 | Prettier byte parity for generated JSON/Markdown is cheap: four layout rules + one width table, oracle-probed
|
||||
|
||||
R5.2 reproduced Prettier 3.8.3 `parser: json` over `JSON.stringify(v, null, 2)` and markdown table alignment in ~350 lines Rust, byte-exact on 968 synthetic probes + 7 seed files + 3 reports. Rules that mattered: objects always expanded (stringify newline after `{`), array breaks when >1 element and all elements arrays >1, or child breaks, or flat form + trailing comma > 80 cols; broken all-number arrays use `fill`, others one per line; `printNumber` drops `e+`; width = length if all bytes 0x20..0x7F else per code point (controls/combining/VS16 = 0, East Asian wide/fullwidth = 2) after RGI emoji replaced by spaces. Emoji table not reproduced → refuse (`UnsupportedWidth`), declared. Probe generator in TS (vitest, `MAKE_R5_ORACLES=1`) sweeping depth × key width × count × digits around the 80-col boundary found both bugs (`e+21` form, width fast path) before any real file did.
|
||||
Apply: any port that must match a formatter's bytes: sweep the boundary with synthetic probes written by the formatter itself; transcribe tables from the installed package (same table lives in `index.mjs` and plugin), never from memory; refuse unreproduced classes loudly instead of approximating.
|
||||
|
||||
## LRN-072 | 2026-10-23 | serde `Option<T>` reads JSON `null` as absent; Zod `.optional()` refuses it
|
||||
|
||||
Found by R5.3 verifier pass 2: changeset `scope` / `position` / `field` declared `Option` + `#[serde(default)]` → `"scope": null` parsed as absent, Zod refused (`expected 'self' | 'subtree', received null`). Committed data had none → every parity test green, only a hostile input shows it. Core already had `js::present` (R2) for exactly this; port forgot it. Complement of LRN-048 (serializer side: `.optional()` absent vs nullable `null`).
|
||||
Apply: porting a Zod schema to serde: `.optional()` → `default, deserialize_with = "js::present"`; `.nullable()` → `js::nullable`; add one `null` refusal case per optional key to the differential oracle.
|
||||
|
||||
## LRN-073 | 2026-10-23 | In-band markers replacing object identity must be reserved keys refused at input
|
||||
|
||||
R5.3 applier replaced TS object identity (touched items) with a transient key `"\0touched"`, stripped after each change. Verifier pass 3: an authored value holding that key was taken for a mark and the cleanup deleted it silently, where TS refused it at the strict gate → silent author-data loss. Fix: refuse any NUL-prefixed key in the changeset at parse (`reserved key`), test it.
|
||||
Apply: any in-band sentinel (key, id prefix, magic value) gets a reserved namespace, a refusal at the input boundary and a test; never rely on "no author would write that". Prefer out-of-band state (side table by path) when the cost is similar.
|
||||
|
||||
## LRN-074 | 2026-10-23 | Formatter-parity guard: allowlist + generator sweep, never denylist
|
||||
|
||||
R5.4 ported CHANGELOG/REVIEW-REPORT without a Markdown printer: reproduce Prettier's table alignment only, refuse text Prettier would rewrite. Denylist guard leaked every gate round (emphasis, links, autolinks, entities, strong pairs, URL underscores, image refs, nested padded wiki links, underscore paired across code spans, wiki opened in one field closed in another, field segment closing author `[` from a code span). Allowlist (letters, digits, single spaces, fixed punctuation, few ruled forms; code spans only the generator alphabet `[A-Za-z0-9_:/. -]`; list-item fields refuse `[[ ]] {% %}` whole) + TS-written sweep (798 probes, 400 seeded random, cross-field pairs) converged: rounds 11-14 found 0 silent over ~400,000 TS-vs-Rust cases.
|
||||
Apply: byte parity with a formatter you do not port → allowlist of what the real data uses, refuse rest loudly, sweep frozen by the formatter itself incl. random strings and constructs split across fields/lines; state guarantee as empirical; add each gate-found family to sweep.
|
||||
|
||||
## LRN-075 | 2026-10-23 | Resource bounds: charge at the allocation site, projection must be a tested upper bound
|
||||
|
||||
R5.4 security rounds 6-11 each broke a bound that looked done: table bound checked after raw rows built; records bound checked after a whole op (one op records once per item, each record cloning change texts); weight counted 1/byte while each cloned String costs header + alloc (~56 B for 1-char id); drawings projected from cleaned SVG only, unnamed triggers copy opened view names ×4; projection 477 B under written for a draft with no annotations (fixed parts uncharged). Each time docs claimed "refused before built".
|
||||
Apply: bound = check before the clone/allocation, at every producer (record push, marking, subtree loop); weight ≈ real bytes (64 per node/String + bytes); output projection = upper bound per element + fixed part per file, with a test asserting projected ≥ written on real and degenerate inputs (empty, max-copy); measure hostile inputs at 1/10 of the read cap before claiming a bound in docs.
|
||||
|
||||
## LRN-076 | 2026-10-24 | Stub suite ≠ real-service recette; record both apart
|
||||
|
||||
Context: R5.5 upload tool. 12 loopback-stub tests green; Claude's run vs real local stack refused by permission layer (transfer tool + service key). Owner ran `pnpm e2e:sheet-assets` by hand → PASS 11/11.
|
||||
Pattern: stub proves tool logic (plan, refusals, retries, no overwrite); only real service proves endpoints, status codes, grants (anon key refused at Storage, not stub). Ship a hand-run e2e script beside stub suite; record each result with its author (simulated by Claude / real by owner), never merge into one "tested".
|
||||
Apply: any tool touching a real service with a privileged key: stub tests in `cargo test`, owner-run e2e script, both results in RUST.md + TODO separately. Owner validated 2026-10-24.
|
||||
|
||||
## LRN-077 | 2026-10-25 | Privilege decision needs proof at the call site holding the evidence
|
||||
|
||||
Context: R5.6 M1. Alias granted at export from snapshot's claimed value; export never saw the loaded sheet. Load-time flag alone also insufficient (pub mutable document/reference).
|
||||
Pattern: put the decision where the evidence object lives (build_sheet_snapshot holds LoadedSheet), keep private verified state set only by the trusted constructor, and recompute the proof at decision time; every other site stays strict equality. Make the raw comparison crate-private so no caller can skip the proof.
|
||||
Apply: any equivalence/allowlist privilege (aliases, migrations, trust marks): one decision method on the verified object, recompute, strict elsewhere, negative tests on the direct API not just app paths.
|
||||
|
||||
## LRN-078 | 2026-10-25 | Before deleting old engine, census contracts only its test code checks
|
||||
|
||||
Context: R5.7. Oracles + Rust parity looked full; case-by-case map of 407 TS cases found 15 anatomy + 1 domain contracts no Rust test checked (612-leaf fates, 26 legacy ids, region inference, R5.1 ledger rows, overlay removal, halo, CHG-000, dollar tag, viewBox, cite of added ref, move via renamed path, folder file list, map vs oracle per entry). Oracle generators' own asserts carried them.
|
||||
Pattern: ledger row per old case (contract, disposition, live target, oracle); census test checks target is non-ignored `#[test]` fn and oracle file exists; port gaps before `git rm`.
|
||||
Apply: any engine deletion (R-phases): map every case first, "oracle covers it" is a claim to prove per case. Owner validated 2026-10-25.
|
||||
|
||||
## LRN-079 | 2026-10-25 | Optional peer in lockfile ≠ live use of deleted engine
|
||||
|
||||
Context: R5.7 C6. After removing anatomy, `tsx` stayed in pnpm-lock.yaml as vite's optional peer (`optional: true`, no importer, no bin); `pnpm install` keeps it, `dedupe --check` clean; only hand edit/override would drop name. Contract said "tsx gone" → false NOT-MET.
|
||||
Pattern: prove use by `pnpm why <pkg> -r` path + importer blocks + manifest/script/source census, not lockfile name grep. Never hand-edit lockfile or add override just to drop a name.
|
||||
Apply: dependency-removal clauses state objective (no declared dep, no script, no import), allow transitive/optional peers; census guards declared deps and `optional: true`. Owner validated 2026-10-25.
|
||||
+59
-1
@@ -304,7 +304,65 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo
|
||||
- [x] owner actions: J1 + J2 on the OD-6 devices (done 2026-10-21, iPhone included); still owner-side: delete the UFW 4173 rule after the recette sessions (`sudo ufw status numbered`, `sudo ufw delete <n>`)
|
||||
- [ ] owner action (non-blocking): `packages/pdf/node_modules` leftover (13 pnpm symlinks, 6 bin shims, 1 vitest cache file, 0 tracked or useful files) → `rm -rf packages/pdf/node_modules && rmdir packages/pdf` (the recursive delete was refused to Claude by the permission layer)
|
||||
- [ ] outside R4 (owner 2026-10-17, non-blocking): worker container ignores SIGTERM (Docker kill after 10 s; chore, e.g. `init: true`); README test counter stale (367 vs 1,015)
|
||||
- [ ] R5 curation pipeline + importer + upload tool: seed golden regenerated once (new `documentSha256`, D-R5), `packages/anatomy` + `packages/domain` deleted, docs map; all before Step 13
|
||||
- [ ] R5 curation pipeline + importer + upload tool (plan `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md`, branch `feature/rust-r5-pipeline-core` from 9c0117a; owner 2026-10-22: R5.1 authorised, stop before any Rust pipeline code and before any `documentSha256` change; no merge without validation, no deletion of packages/anatomy or domain, no golden or historical change, no 0.2.0, no deploy, no SaaS feature): `packages/anatomy` + `packages/domain` deleted at the end, docs map; all before Step 13
|
||||
- [x] R5.1 freeze + D-R5 study (2026-10-22): inventory `.claude/audits/R5_PIPELINE_INVENTORY.md`, census `.claude/audits/R5_DOCUMENT_SHA256_CENSUS.md`, oracles `crates/core/tests/oracles/r5/{pipeline,ids,fingerprints}.json` (generator `packages/anatomy/tests/r5.pipeline-oracles.test.ts`, `MAKE_R5_ORACLES=1`), study scripts `scripts/r5-fingerprints.mjs` + `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`, RUST.md §11, anatomy 137/137, domain 262; F1 found: core `js_number` rounds exact ties upward (JS: to even; `zmij` matches), 1 legacy value, invisible on the draft, R2 contract test self-referential → checkpoint, STOP
|
||||
- [x] owner decisions (2026-10-22, BDR-068): D-R5 = B `sha256-rfc8785-validated-v1` (applied at R5.6 only), aliases from trusted frozen documents only, F1 go, byte-exact by default + declared differences, historical SVG bytes kept, `.prettierignore` on Rust-written paths + reproducible check, `ureq` preferred at R5.5; stop after R5.2
|
||||
- [x] R5.1b F1 fix (aed6b1e): `zmij` digits; V8 oracle `oracles/r5/numbers.json` 44,322 rows (1,552 mismatches before, 0 after); 840.1692504882812 test; `zod_oracle` checks oracle bytes vs core compact print (red on old printer); consumers audited R2-R4: rust:check + wasm + check green, no oracle/golden/digest moved
|
||||
- [x] R5.2 legacy importer in Rust (feature `curation`, `crates/core/src/curation/**`, 3,187 lines) + `import-equine --check | --out`: legacy-import/ 4 texts + 2 SVG byte-exact, pipeline.json legacy parts equal, raw report = pre-Prettier digest, determinism, 10 synthetic TS cases (2 byte-exact, 8 refused alike), format probes (968 JSON, 10 tables, slugs, uuids, Number, Math.round, key order); declared differences RUST.md §11.10; contract `.claude/tasks/contracts/2026-10-22-rust-r5-f1-importer.md` → checkpoint, STOP before R5.3
|
||||
- [x] R5.3 changeset applier, migration map, stats, seed SQL (CLOSED 2026-10-23 on c6e0d85: fresh verifier CONFORME 13/13 after 6 final rounds, security PASS LOW only, check:all green; owner go 2026-10-22 « Validation R5.1b / R5.2 et autorisation R5.3 »: `pnpm r5:import-equine --check | --out <dest>` names the Rust binary, `pnpm import:equine` kept; no fingerprint switch; `horse.svg` frozen bytes read, never re-cleaned; R5.4 files (review SVGs, CHANGELOG, REVIEW-REPORT) out of scope; stop at the checkpoint)
|
||||
- [x] R5.3a `crates/core/src/curation/draft/{schema,targets,fields,ops,rename,apply}.rs`: changeset + references files parsed strictly (Zod shape order on re-emit, `priority` default, level C question, duplicate ids), target grammar, settable fields, the six ops on the plain JSON document (`serde_json::Value`, insertion order), rename propagation, review rule (touched items → `to_confirm`, identity through a transient marker key), shape + integrity gate through the R2 core (`AnatomicalSheetDocument` + `validate_sheet_document`)
|
||||
- [x] R5.3b `draft/{migration_map,stats,seed,pipeline}.rs`: replay of rename/remove/move records over the 792 legacy paths, re-homing, stats; `documentStats` (17 fields, JS key order); `renderDraftSeedSql` byte-exact; `build_equine_draft` (CHG-000 from the frozen `horse.svg` bytes and the head view box) + `render_draft_artifacts` (document, assets, sheet-ref with sha256 of the document bytes, migration map, seed SQL)
|
||||
- [x] R5.3c `import-equine`: both folders (`<dest>/legacy-import`, `<dest>/draft`), `--check` compares the produced files and lists the R5.4 files as pending (not counted), `--out` refuses a non-empty folder, a symlink, a file, an empty path; `scripts/import-equine.sh` + root script `r5:import-equine`
|
||||
- [x] R5.3d proofs (draft 5 files byte-exact, 12/12 `--check`, 46 changes + 825 records + stats equal, 612 leaves 442/141/29, 58 synthetic cases: 9 byte-exact + 49 refused alike + 5 references files, rust:check green): `tests/curation_draft.rs` (draft files byte-exact vs committed, `pipeline.json` draft section: 46 changes, 825 records, stats, migration stats, assets, issues; determinism; 612-leaf walk through the core's migration on the produced map); TS generator `r5.curation-synthetic-oracles.test.ts` → `oracles/r5/curation-synthetic.json` (mini sheet: every op kind, positions, subtree scopes, review rule B/C, renames with cascades, re-homing, refusals) run by the Rust suite; `pnpm rust:check`, `pnpm check:all`
|
||||
- [x] R5.3e (security PASS 6 LOW / 3 fixed; verifier 4 passes, last C1-C12 met + C13 docs fixed after; check:all green 00:10:03) gates: security auditor (diff), independent verifier on the contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`; docs RUST.md §11.12-11.13, CHANGELOG, CLAUDE.md commands; memory → checkpoint report, STOP before R5.4
|
||||
- [x] R5.3g final rounds 1-6 (verifier + security + check:all each): NUL field segments, depth/segment/value bounds, exclusive --out writes, shape family, JSON.parse normalisation, non-list/non-object children, __proto__ segments, seed identity guard, reads confined to the seed; all fixed or declared (RUST.md §11.12-11.13)
|
||||
- [x] R5.3f owner 2026-10-23: Object.prototype member characterised (safe, accepted, test), migration-map debt kept + characterisation test, refusal reasons compared by stage; final fresh verifier + security on the R5.3 state
|
||||
- [ ] debt (owner 2026-10-23, after R5): migration map replays cascade records by their op (`applyRecord` / `apply_record`): a view, zone, region, category or cited-reference rename corrupts the map; fix both sides as a declared change before any changeset uses such a rename (RUST.md §11.12)
|
||||
- [ ] OWNER DECISION (found 2026-10-23 by the R5.3 final verifier): serde derives of the R2 contracts read a struct from an array and a unit enum from a one-key map, where Zod refused; likely reachable by the gated readers of imported PDFs (`archive/read.rs`, `report/read.rs`, `pdf/import.rs`); probe the readers, then a per-file go for a strict deserialisation (RUST.md §11.13)
|
||||
- [ ] chore (pre-existing, security gates R5.3): `clippy.toml` disallowed list misses `fs::create_dir*`, `OpenOptions::open`, `read_dir`, `canonicalize`, `remove_*`, `env::current_dir`, `env::args_os`, `process::Command`; extend so the core's no-I/O rule is enforced, not only observed
|
||||
- [ ] R5.4 review overlays + app drawings, CHANGELOG + REVIEW-REPORT, --check over the whole draft (owner go 2026-10-23, conditional on the R5.3 final checks: met on c6e0d85; contract `.claude/tasks/contracts/2026-10-23-rust-r5-review.md`; stop at the checkpoint, no R5.5)
|
||||
- [x] R5.4a `draft/{labels,changelog,review_report}.rs` + `stats::stat_rows`: CHANGELOG and REVIEW-REPORT raw text = ledger pre-Prettier digests, formatted text byte-exact (`report_guard::format_report`: allowlist guard + `markdown::format_markdown` alignment); compactValue on UTF-16 units
|
||||
- [x] R5.4b `draft/{overlay,app_review}.rs` on the core tree (`read_svg`, frozen horse.svg, prolog kept verbatim): review overlays and app drawings; bytes vs xmldom where reproducible, parsed trees otherwise; the R3 trigger-name decision declared for app-main
|
||||
- [x] R5.4c `render_draft_artifacts` complete (TS order); `import-equine --check` covers the 12 draft files with explicit criteria (bytes; declared tree rule), PENDING list and `shortcut:` removed; fails on any missing or unauthorised difference
|
||||
- [x] R5.4d proofs: `tests/curation_review.rs` (the 6 files, ledger digests, determinism); TS generator `r5.review-synthetic-oracles.test.ts` → `oracles/r5/review-synthetic.json` (changelog raw + formatted, review report, overlays, app drawings on mini-sheet cases: Unicode, pipes, newlines, long values, emoji refused, every status); R5.3 LOW carry-over (destination identity, cite membership, unreadable-seed message, stale comment, doc nits)
|
||||
- [x] R5.4e gates: 14 rounds; c4e717d security PASS (LOW only) + check:all green 16:41:34; eb667ad (docs only over it) fresh verifier 9/9 clauses MET, ECARTS = 13 FLOOR hits on audit prose (owner Clarification); docs RUST.md §11.14-11.15, CHANGELOG, CLAUDE.md; memory LRN-074, LRN-075, EVAL-029 → checkpoint delivered, STOP before R5.5 (R5.4 closes on owner validation)
|
||||
- [x] R5.5 `upload-sheet-assets` bin + runbook (CLOSED 2026-10-24 by the owner: real local stack `pnpm e2e:sheet-assets` PASS 11/11, run by the owner by hand; owner go 2026-10-23 « Validation R5.4 et autorisation conditionnelle R5.5 »; local stack only; no production, no deploy; stop at the checkpoint, no R5.6)
|
||||
- [x] R5.5-0 R5.4 closing adjustments: FLOOR clarification (contract + `scripts/floor-clarification-check.sh`, 19 audit-prose hits requalified, 1 STUB in evals.md:205 outside scope, reported), 4 MiB cap on `changes.json` / `references.json` (core `MAX_CURATION_FILE_BYTES`, bin bounded read; 222 KB + 17 KB committed, parse of a 3.98 MB changeset peaks +39 MB), `tests/curation_caps.rs` (cap, margin, stable ids of any species), serde R2 tolerance probe (characterisation only, nothing changed)
|
||||
- [x] R5.5-1 core module `curation::upload` (pure, no I/O): manifest parse + entry rules, destination rule, key derivation, per-entry plan from observed states, closed error kinds
|
||||
- [x] R5.5-2 bin `upload-sheet-assets` (feature `upload`, ureq): `--check` read-only and write mode, destination verified (loopback or the DEV host, http, explicit port) then handshake (private bucket) before any transfer, existing objects verified by sha256 never overwritten, confirm only after the object is verified, bounded retries, secrets never printed
|
||||
- [x] R5.5-3 scripts: `assets:upload` points at the Rust bin (`scripts/upload-sheet-assets.sh`), TS script kept until R5.7 census; runbook SUPABASE.md §3.1, RUST.md §11.16
|
||||
- [x] R5.5-4 tests (real-stack e2e NOT run by Claude: permission layer refused; owner runs `pnpm e2e:sheet-assets`): unit (core), bin tests against a stub HTTP server (refusals, partial failure, resume, network errors), local Supabase run (upload, rerun, anon refused, wrong key, tamper); `pnpm rust:check`, `pnpm check:all`, `r5:import-equine --check`
|
||||
- [x] R5.5-5 gates (round 1: verifier ECARTS(4) + security PASS 1 MEDIUM 9 LOW, fixed; round 2 fad85c0: verifier C1-C13 MET, check:all green 21:47:11, security PASS 5 LOW declared RUST.md §11.16): fresh security audit, fresh verifier on the contract `.claude/tasks/contracts/2026-10-23-rust-r5-upload.md`; memory; checkpoint report (9 points), STOP
|
||||
- [ ] R5.5 closing corrections (owner 2026-10-24 « Clôture R5.5, corrections ciblées et préparation R5.6 »; contract `.claude/tasks/contracts/2026-10-24-rust-r5-5-closing.md`)
|
||||
- [x] LOW: `Protocol` final, DEV address own unicast (broadcast refused), tests
|
||||
- [x] serde: `check_v3_shape` at `read_report_document` v3 + `parse_envelope` (sheet, practitioner, severity), test turned around
|
||||
- [x] `confirm_sheet_asset_upload` characterised (no SQL change), DEV keys + 54321 documented (ENVIRONMENTS.md §3.6)
|
||||
- [x] R5.6 plan `.claude/tasks/plans/2026-10-24-rust-r5-6-fingerprint-switch.md` (4 decisions open)
|
||||
- [x] gates: round 1 check:all 03:39:24, security PASS (2 MEDIUM, 6 LOW), verifier ECARTS(2) → positional-parent bypass fixed (1ec9e1b); round 2 check:all 03:51:55, verifier K3/K4 behaviour MET, 2 formal gaps → contract Clarification (owner to confirm) → checkpoint, STOP (no R5.6 without signal)
|
||||
- [ ] R5.6 preparation (owner 2026-10-24 « arbitrage D-R5.6 »; nothing active)
|
||||
- [x] `domain/sheet/fingerprint.rs` + `sheet_fingerprint` tests (RFC, V8, Node differential, draft 12d7…, registry, census 123/124 alias)
|
||||
- [x] switch patch `.claude/tasks/plans/r5-6-switch.patch` (5-consumer wiring rehearsed green rust:check + check:all 14:32:05; whole patch: 4 curation tests fail on current seed, +3 on switched seed, RUST.md §11.19), files restored
|
||||
- [x] gates on 3b9a395: check:all 14:41:00, security PASS (M1 snapshot stamp → owner, M2 block strictness partly fixed), verifier ECARTS(2) (report-compare build in patch fixed, false "rehearsed green" corrected) → checkpoint, STOP (switch on signal)
|
||||
- [ ] R5.6 rehearsal (owner 2026-10-24 « corrections préalables à la bascule »; contract `.claude/tasks/contracts/2026-10-25-rust-r5-6-rehearsal.md`; isolated worktree, branch inactive)
|
||||
- [x] M1 option 1 (input.rs), directional equivalence + fingerprint_verified, v0/v1 load rules, report-compare status, generator order
|
||||
- [x] original contexts (O7, O10, R2 Zod, R5.1 ledger, TS golden, app goldens) + separate post-switch tests; rust:check + check:all 00:56:22 green on switched state
|
||||
- [x] gates: security PASS (1 MEDIUM, 6 LOW), verifier ECARTS(3) = export grants alias without fingerprint_verified (fix needs snapshot_rules.rs, outside scope) → checkpoint, STOP, owner decision
|
||||
- [x] R5.6 M1 correction + full rehearsal (owner 2026-10-25; snapshot_rules.rs authorised): recognises_historical_stamp, null block refused, warnings allowed, depth 128; fresh worktree r56final: rust:check + check:all 01:53:59 green; security PASS (LOW); verifier CONFORME 8/8 → checkpoint, STOP (activation on signal)
|
||||
- [x] R5.6 D-R5 switch ACTIVE (owner go 2026-10-25, commit 9e87016): seed sheet-ref 03e7f6ab + migration map 0e991bc2 only, --check 0 diff, rust:check + check:all green, security PASS LOW, verifier ECARTS(1)=S7 wording → Clarification; BDR-070; rollback not transparent for post-switch reports (settle before user release) → checkpoint, STOP, no merge, no R5.7
|
||||
- [ ] R5.7 consumer census, domain fixtures moved, packages deleted, xmldom + prettier out of the lockfile, docs, ledger; `pnpm check:all`; merge only on the owner's signal (owner go 2026-10-25 « Validation R5.6 et autorisation R5.7 »: rollback limit accepted on the branch, no intermediate release; plan `.claude/tasks/plans/2026-10-25-rust-r5-7-cleanup.md`, contract `.claude/tasks/contracts/2026-10-25-rust-r5-7-cleanup.md`; stop at the checkpoint, no merge)
|
||||
- [x] R5.7a census + matrices (LEDGER 407: 256 ported, 149 replaced, 2 deleted, 0 gap; FILES 112), 15 anatomy + 1 domain uncovered contracts ported first (+19 Rust tests, curation_leaves.rs new)
|
||||
- [x] R5.7b `report-v2.*.json` → `crates/core/tests/fixtures/` (sha256 kept), Rust readers repointed
|
||||
- [x] R5.7c deletion (anatomy, domain, TS upload script, legacy scripts, Dockerfile lines, lockfile by `pnpm install`)
|
||||
- [x] R5.7d census tests (r4 xmldom inverted + non-vacuity, R5 census, ESLint ban)
|
||||
- [x] R5.7e historical reproducibility documented and rehearsed (59521dc: git status clean, 144+262; 756b7ef: 2 files, 144+263) in a detached worktree
|
||||
- [x] R5.7f docs (CLAUDE.md, WORKSPACE, packages/README, README, RUST.md §11.23, CHANGELOG, sheet model, SUPABASE, TEST_STRATEGY, PDF-FORMAT)
|
||||
- [x] R5.7g checks (frozen install, rust:check, check:all green 03:58:13, --check 0 diff, TS 943→546, Rust 364→383) + fresh security + fresh verifier → checkpoint, STOP
|
||||
- [x] R5.7h owner validated R5.7 2026-10-25 (« dernière vérification pré-fusion »); C6 Clarification (tsx = vite optional peer, accepted); census refuses `#[ignore]` targets (cd69f0e); pre-merge rerun on 27bf59c: frozen install + dedupe ok, check:all green 13:41:41, --check 0 diff, report:compare ok, security PASS, verifier CONFORME 10/10 (`.claude/audits/2026-10-25-rust-r5-7-premerge.md`) → STOP, merge `--no-ff` into develop only on the owner's signal
|
||||
- [ ] later maintenance (owner accepted temporarily 2026-10-25, LOW, security final): census regex still accepts `#[ignore]` before `#[test]` or `#[cfg_attr(…, ignore)]` (no mandatory target affected); fix = inspect the whole attribute block above `fn <test>(`
|
||||
- [ ] before any release or public exposure (owner 2026-10-25, not blocking develop): `pnpm audit` 57 advisories, all pre-existing on develop; treat or qualify each one, keeping its published severity apart from Faunosteo's actual exposure. Priority: zod 3.25.76 GHSA-mrhx-c7xc-m3w5 (oversized constrained collections; fixed 4.6.6 = major): first map every path that feeds untrusted data to Zod and its size guards, then plan the zod 4 migration as its own chore. Others: tinypool/vitest, vite 7.3.6 dev server (LAN), happy-dom, pnpm 10.13.1, brace-expansion, js-yaml, nanoid, postcss, source-map-js
|
||||
- [ ] if a new anatomical changeset lands: owner decision on a new baseline for the Rust tests pinning the real draft (46 changes, 825 records, 612 fates, fingerprints); oracles never regenerated to hide a divergence
|
||||
- [ ] owner action (non-blocking): `packages/{anatomy,domain}/node_modules` leftovers (git-ignored pnpm symlinks; recursive delete refused to Claude) → `rm -r packages/anatomy/node_modules packages/domain/node_modules && rmdir packages/anatomy packages/domain`
|
||||
- [ ] DEV hardening (owner 2026-10-24, risk accepted temporarily, not scheduled): Supabase on CLI demo keys + 54321 LAN-open (rootless, UFW allow incoming, no 54321 rule) → own JWT secret/signing keys via env() or narrower 54321 rule before any untrusted device or user (ENVIRONMENTS.md §3.6); never these keys in production; `confirm_sheet_asset_upload` object check before public exposure
|
||||
- [ ] separate chore (D-R7): form constraints module + PGlite `pg_constraint`/config.toml parity test
|
||||
- [ ] Release 0.2.0 (branch release/0.2.0 = Steps 3–6, version.txt + package versions 0.2.0, CHANGELOG [0.2.0] 2026-10-11): deploy-trigger check done 2026-10-12 (CI = branches main/develop only, no tag trigger, no deploy job, docker job builds `faunosteo:ci` locally; hooks push only; DEPLOY.md deploy = manual on the VPS). Default 0.2.0 build carries NO sheet (SHEET_BUNDLE=none) → the live local tool must not be redeployed from main without `--build-arg SHEET_BUNDLE=draft`. Finish via /release-candidate (finish + tag span, tag push on explicit go); CHANGELOG back-merge conflict expected (Unreleased Step 7 above [0.2.0]). Awaiting the user's go.
|
||||
---
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# CONTRACT: Rust R5.3, the draft side of the anatomical pipeline (changeset applier, migration map, stats, seed SQL)
|
||||
|
||||
- date: 2026-10-22 | branch `feature/rust-r5-pipeline-core` | plan `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md` §B.5 row R5.3
|
||||
- brief: owner 2026-10-22 « Validation R5.1b / R5.2 et autorisation R5.3 » (applied as written; its §1 to §9 are the clauses below)
|
||||
- predecessor: R5.2 contract `.claude/tasks/contracts/2026-10-22-rust-r5-f1-importer.md` (the legacy importer, `import-equine --check | --out`)
|
||||
|
||||
## Scope
|
||||
|
||||
Port `packages/anatomy/src/curation/{schema,targets,fields,ops,rename,apply,types,migration-map,stats,seed,pipeline}.ts`
|
||||
(the draft side without the review drawings and the two Markdown reports, which are R5.4) into
|
||||
`crates/core/src/curation/draft/**` behind the existing `curation` feature, and extend the
|
||||
`import-equine` binary to the `draft/` folder. Out of scope: `overlay.ts`, `app-review.ts`,
|
||||
`changelog.ts`, `review-report.ts`, `assets.ts` (`stripLegacyAnnotations`: the cleaned `horse.svg`
|
||||
is frozen, read as bytes), the fingerprint convention (R5.6), the upload tool (R5.5), any deletion
|
||||
under `packages/`.
|
||||
|
||||
## Clauses (each one checkable)
|
||||
|
||||
C1. **Inputs.** The core reads the legacy `ImportResult` (R5.2), the bytes of the frozen
|
||||
`draft/assets/horse.svg`, the head view box text, `curation/references.json` and
|
||||
`curation/changes.json` as texts. No `std::fs`, no clock in `crates/core/src/curation/**`
|
||||
(the binary `crates/core/src/bin/import_equine.rs` is the only file allowed the file system).
|
||||
|
||||
C2. **Strict schemas.** `changes.json` and `references.json` are refused on: an unknown key in any
|
||||
object, a wrong type, an `id` not matching `^CHG-\d{3,}$`, an empty `title.fr` / `title.en` /
|
||||
`rationale` / target / field / rename id / cited reference id / `question`, an empty `ops`, an
|
||||
empty `cite.reference_ids`, a level C change without a `question`, a duplicate change id, a
|
||||
duplicate reference id, a position that is not a non-negative integer or a strict
|
||||
`{after}` / `{before}` object, a reference failing the R2 `Reference` contract. `priority`
|
||||
defaults to `medium`. Declared difference: a `set` without `value` is refused (TypeScript
|
||||
wrote `undefined`, i.e. dropped the key).
|
||||
|
||||
C3. **Applier semantics** (`apply.rs`, `ops.rs`, `rename.rs`, `targets.rs`, `fields.rs`): the
|
||||
six ops behave as `ops.ts` on the plain JSON document (insertion order of keys, `review_status`
|
||||
appended last when created), with the settable table of `fields.ts`, the target grammar of
|
||||
`targets.ts`, the rename cascade of `rename.ts` (one record per rewrite), `assertDeclared`,
|
||||
`assertNotReviewed`, and the review rule of `apply.ts` (every touched finding or annotation of a
|
||||
B or C change set `to_confirm` with a record, first-touch order, items removed from the document
|
||||
skipped). Any op that cannot apply exactly refuses with the change id and op index; the result
|
||||
then passes the R2 shape (`AnatomicalSheetDocument`) and integrity (`validate_sheet_document`)
|
||||
gate, errors refuse, warnings are returned.
|
||||
|
||||
C4. **Records.** Every record is `{change, op, target, field, before, after}` with `change` in
|
||||
the Zod shape order (`id, level, kind, title, rationale, reference_ids, question?, priority`),
|
||||
`op` in the Zod shape order of its kind, absent values as `null`.
|
||||
|
||||
C5. **Migration map** (`migration_map.rs`): `buildReportMigrationMap` as `migration-map.ts`
|
||||
(replay of rename / remove / move records, zone renames, annotation and view renames, the
|
||||
finalPaths check, unique re-homing, stats `nodes / identical / remapped / rehomed / removed`).
|
||||
|
||||
C6. **Stats** (`stats.rs`): `documentStats` with its 17 fields in the TypeScript key order,
|
||||
the fixed-key counters (`point, zone, detail_trigger`; `inherited, reviewed, to_confirm`)
|
||||
extended by unlisted values as JavaScript does.
|
||||
|
||||
C7. **Seed SQL** (`seed.rs`): `renderDraftSeedSql` byte for byte (header, sheet insert, asset
|
||||
inserts, `insert_sheet_draft`, converging update, the dollar tag refusal, `trimEnd`).
|
||||
|
||||
C8. **Draft artifacts** (`pipeline.rs`): `build_equine_draft` runs CHG-000 (asset record of the
|
||||
frozen bytes: sha256, byte size, uuid; `view:main.svg_asset_id`, `view:head.svg_asset_id = null`,
|
||||
`view:head.view_box` from the head SVG's viewBox) then the authored changes, refuses a changeset
|
||||
reusing `CHG-000`; `render_draft_artifacts` returns `document.json`, `assets.json`,
|
||||
`sheet-ref.json` (`document_sha256` = sha256 of the `document.json` bytes, the historical rule),
|
||||
`report-migration-map.json`, `equine_draft.sql`, in the TypeScript order, plus the byte copy
|
||||
`assets/horse.svg`.
|
||||
|
||||
C9. **Parity proofs** (`crates/core/tests/curation_draft.rs`): the six R5.3 files of the
|
||||
committed `draft/` byte for byte; the `pipeline.json` draft section (sheetId, 46 changes, 1
|
||||
asset, statsBefore, statsAfter, issues, migrationStats, 825 records) equal under the JS compact
|
||||
print; two runs byte-equal; the 612 legacy leaves migrated through the core's `read_report_document`
|
||||
on the produced map: 442 identical, 141 moved, 29 removed, every resolved leaf a leaf of the
|
||||
produced document, the 29 flagged paths exactly those the map sends to null.
|
||||
|
||||
C10. **Synthetic differential** (`oracles/r5/curation-synthetic.json`, generator
|
||||
`packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`, `MAKE_R5_ORACLES=1`): changesets
|
||||
on the mini sheet covering every op kind, numeric and anchored positions, subtree scopes, the
|
||||
review rule on levels A / B / C, the five rename cascades, a reference rename, re-homing, and the
|
||||
refusal classes of `curation.apply.test.ts` and `curation.schema.test.ts`; each accepted case
|
||||
freezes the formatted document, the records, the stats, the migration map and the seed SQL, each
|
||||
refused case the refusal; the Rust suite reproduces every accepted case byte for byte and refuses
|
||||
every refused case.
|
||||
|
||||
C11. **Binary and script.** `import-equine --check` compares `legacy-import/` and `draft/` with
|
||||
the committed folders and writes nothing; the R5.4 files (`review/*.svg`, `CHANGELOG.md`,
|
||||
`REVIEW-REPORT.md`) are listed as pending and not counted (marked `shortcut:` for R5.4).
|
||||
`import-equine --out <dest>` writes `<dest>/legacy-import/**` and `<dest>/draft/**`, refuses an
|
||||
empty path, a path that exists and is not a directory, a symlink, a folder holding any file;
|
||||
never deletes. `scripts/import-equine.sh` + root script `r5:import-equine` forward the
|
||||
arguments; `pnpm import:equine` (TypeScript) unchanged.
|
||||
|
||||
C12. **Frozen references untouched.** No file under `supabase/seed/**`, `crates/core/tests/oracles/**`
|
||||
(except the new `curation-synthetic.json`), `apps/app/tests/fixtures/**`,
|
||||
`packages/core-wasm/tests/n-1/**` changes; `documentSha256` stays `27c7383c…2660`; no gated
|
||||
reader (`domain/{archive/read,report/rebase,report/read,session}.rs`, `pdf/{read,json_js,import}.rs`)
|
||||
changes; `packages/anatomy` and `packages/domain` keep every test (anatomy gains the generator).
|
||||
|
||||
C13. **Gates.** `pnpm rust:check` green, `pnpm check:all` green, security auditor PASS on the
|
||||
diff, this contract verified CONFORME by a fresh verifier; RUST.md §11.12 (R5.3) and §11.13
|
||||
(checkpoint) written, CHANGELOG Unreleased, CLAUDE.md command list; differences declared one by
|
||||
one. No merge, no R5.4, no fingerprint switch, no release, no deploy.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Contract: R5.1b (F1 fix) and R5.2 (legacy importer in Rust, `import-equine`)
|
||||
|
||||
Base: 64d4595 on `feature/rust-r5-pipeline-core` (« chore(memory): R5 plan, pipeline inventory and documentSha256 census audits, TODO R5.1, journal, LRN-070, EVAL-026 »). Range: `64d4595..HEAD`; `<base>` below is 64d4595. Date: 2026-10-22.
|
||||
|
||||
## REQUEST (verbatim excerpts, owner, « Validation R5.1, décision D-R5 et lancement du portage Rust »)
|
||||
|
||||
« Je valide la correction du défaut d'arrondi de `js_number` dans le cœur Rust. La nouvelle implémentation doit reproduire les règles ECMAScript, notamment les égalités décimales exactes et les cas limites identifiés. Utiliser une implémentation éprouvée déjà présente dans l'écosystème du projet lorsque c'est approprié […]. Ajouter : un test caractérisant précisément le cas 840.1692504882812 ; des vecteurs couvrant les égalités d'arrondi ; des tests différentiels Rust / ECMAScript et des cas limites ; une référence indépendante pour éviter qu'une implémentation calcule elle-même son attendu. Auditer les consommateurs de `js_number` dans R2, R3 et R4. Conserver les oracles et goldens gelés. Si une divergence apparaît sur les corpus historiques, […] arrêter le portage avant toute modification de golden ou de contrat. »
|
||||
|
||||
« Par défaut, chaque fichier produit par Rust doit correspondre à l'ancien pipeline octet pour octet. Les différences éventuellement nécessaires doivent être déclarées individuellement […]. Ne pas modifier silencieusement les données métier pour satisfaire un oracle. »
|
||||
|
||||
« Je décide de conserver les octets de `horse.svg` et des autres assets historiques […]. » « Ne pas ajouter `reqwest` par défaut […]. Ce choix sera implémenté à R5.5, pas pendant R5.2. »
|
||||
|
||||
« Vérifier les cas limites de deterministicUuid, slugify, UTF-16, Math.round et Number(''). Ne pas se contenter de tests sur les données équines actuelles lorsque le contrat est générique et multi-espèces. »
|
||||
|
||||
« Conserver les tests TypeScript et les oracles gelés durant le portage. Ne pas modifier `documentSha256`, le seed officiel ou les contrats des bilans avant R5.6 […]. Ne supprime pas encore packages/anatomy ou packages/domain. »
|
||||
|
||||
## FILE SCOPE
|
||||
|
||||
- R5.1b: `crates/core/src/json.rs`, `Cargo.toml` and `crates/core/Cargo.toml` (`zmij`), `Cargo.lock` (one dependency edge), `crates/core/tests/json_numbers.rs`, `crates/core/tests/common/mod.rs` (`zod_oracle` raw-text check, `repo_path`), `crates/core/tests/oracles/r5/numbers.json`, `scripts/r5-number-oracle.mjs`
|
||||
- R5.2: `crates/core/src/curation/**` (new), `crates/core/src/lib.rs` (module behind `curation`), `crates/core/src/bin/import_equine.rs` (new), `Cargo.toml` and `crates/core/Cargo.toml` (feature `curation`, `icu_normalizer`, optional `sha2`, bin, two test targets), `Cargo.lock` (one dependency edge), `crates/core/tests/{curation_format,curation_legacy}.rs`, `crates/core/tests/oracles/r5/{prettier-json,markdown-tables,js-semantics,legacy-synthetic}.json`, `packages/anatomy/tests/{r5.format-oracles,r5.legacy-synthetic-oracles}.test.ts`, `scripts/rust-check.sh` (the `curation` lines)
|
||||
- Docs and memory: `docs/development/RUST.md` §11.9 to §11.11, `CHANGELOG.md` (Unreleased), this contract, `.claude/tasks/TODO.md`, `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md` (owner decisions), `.claude/memory/{decisions,journal,learnings,evals}.md`
|
||||
|
||||
Out of scope: `supabase/seed/**` (no generated or authored file changes), `apps/**`, `packages/**/src`, `packages/core-wasm/**`, the existing oracles (`crates/core/tests/oracles/*.json`, `oracles/r3/**`, `oracles/r4/**`, and the R5.1 files `oracles/r5/{pipeline,ids,fingerprints}.json`), `apps/app/tests/fixtures/**` (goldens, R4 digests), the gated readers (`crates/core/src/domain/{archive/read,report/rebase,report/read,session}.rs`, `crates/core/src/pdf/{read,json_js,import}.rs`), `crates/worker/**`, `pnpm-lock.yaml`; any deletion.
|
||||
|
||||
## ACCEPTANCE CRITERIA
|
||||
|
||||
1. F1 fixed with an existing implementation: `js_number`'s digits come from `zmij` (already in `Cargo.lock` through serde_json 1.0.154 before the base: `git show <base>:Cargo.lock | grep -A2 'name = "zmij"'`); `git diff <base> -- Cargo.lock` adds dependency edges only (`"zmij"` and `"icu_normalizer"` under `faunosteo-core`), no new `[[package]]`.
|
||||
2. Independent reference: `crates/core/tests/oracles/r5/numbers.json` was written by Node (`scripts/r5-number-oracle.mjs`, `String(value)`, values as IEEE bits) and holds ≥ 40,000 rows including the bits `408a415aa0000000` → `840.1692504882812`; `crates/core/tests/json_numbers.rs` compares `js_number(f64::from_bits(..))` to every row and has a dedicated test for 840.1692504882812 (both signs) and the two other tie values of the study.
|
||||
3. Red before, green after: on the base's `json.rs` the number suite fails (orchestrator run: « 1552 mismatches, first: 408a415aa0000000: 840.1692504882813 vs V8 840.1692504882812 ») and the strengthened `zod_oracle` fails `the_legacy_import_baseline_prints_as_zod_outputs_it`; after the fix both pass. The verifier checks by reading that `zod_oracle` compares the oracle's raw bytes with `to_js_compact(oracle) + "\n"` before using `to_js_pretty`.
|
||||
4. R2 to R4 preserved: `git diff --stat <base> -- crates/core/tests/oracles/*.json crates/core/tests/oracles/r3 crates/core/tests/oracles/r4 apps/app/tests/fixtures crates/core/tests/conformance crates/core/tests/fixtures` is empty; the check:all log (path in the brief) shows the cargo suites (pdf_serialize, pdf_layout, anatomy_*, sheet_*, report_*, archive_*) and the app 434 / core-wasm 58 / domain 262 cases green.
|
||||
5. Formatting parity, generic: `crates/core/tests/curation_format.rs` passes: the 968 Prettier JSON probes match except case 19 (emoji, refused with `UnsupportedWidth`), the 10 table probes match except case 5 (emoji, refused), the 7 generated JSON files of the seed print byte for byte, slugs (ids.json 801 rows + 22 edges) and uuids (6 + 7) match, `Number()` (39 texts), points (12), `Math.round` (18, bits kept), `trim` (8) and `Object.keys` order (3) match. The probes were written by Prettier and Node (`r5.format-oracles.test.ts`), not by Rust.
|
||||
6. Importer parity: `crates/core/tests/curation_legacy.rs` passes: the 4 text files of `legacy-import/` byte for byte and the 2 SVG copies; every key of `pipeline.json`'s `legacy` section (sheetId, viewBoxes, stats, sourceDigests, assets, anomalies, issues, regionInferences, zoneStats, annotationRows) equal in the JavaScript compact form; the raw report's sha256 equals the ledger's `before_prettier_sha256`; two runs byte-equal; the 10 synthetic cases of `legacy-synthetic.json` (2 accepted: files byte for byte and every result part; 8 refused by the TypeScript: refused by Rust too).
|
||||
7. Binary: `cargo run -p faunosteo-core --features curation --bin import-equine -- --check` prints `legacy-import: 6 files, 0 differences` and exits 0; `--out` refuses a non-empty folder (exit 2, nothing written); the binary never deletes a file (read `crates/core/src/bin/import_equine.rs`: no `remove_*` call).
|
||||
8. Declared differences only: every behaviour where the port refuses what JavaScript accepted is listed in RUST.md §11.10 (emoji widths; malformed SVG, DTD or caps; non-string ids, non-array `children` / `categories` / `sources`; prototype-named keys) and none of them is reachable by the committed sources (criterion 6 proves the equine folder).
|
||||
9. Gating: the curation code compiles only with feature `curation` (`#[cfg(feature = "curation")]` on the module; the bin and the two suites `required-features`); `crates/wasm/Cargo.toml` and `crates/worker/Cargo.toml` do not enable it; `scripts/rust-check.sh` runs clippy and tests with `--features curation`.
|
||||
10. Scope held: `git diff --stat <base> -- supabase apps packages/core-wasm packages/domain packages/anatomy/src packages/supabase crates/worker pnpm-lock.yaml` is empty; `packages/anatomy` and `packages/domain` still exist; no `reqwest` anywhere in `Cargo.lock`; no deletion (`git diff --diff-filter=D --name-only <base>` empty); `documentSha256` and `sheet-ref.json` unchanged.
|
||||
11. Suites: `pnpm check:all` exit 0 on the final code state (log ending `== all checks passed`); `pnpm rust:check` exit 0 with the curation lines; `git status --short` empty at the end.
|
||||
12. Docs: RUST.md §11.9 (F1: fix, oracle, consumers audited with the result on R2 to R4), §11.10 (R5.2: modules, parity proofs, declared differences, binary), §11.11 (checkpoint R5.2, conditions for R5.3); CHANGELOG Unreleased lines for F1 and the importer; the owner's decisions recorded (BDR) with D-R5 = `sha256-rfc8785-validated-v1` not yet applied.
|
||||
|
||||
## CLARIFICATIONS
|
||||
|
||||
Q: emoji in a table cell or a JSON string: reproduce Prettier's RGI emoji table? / A: no: the table is not reproduced; a text holding an emoji candidate is refused (`UnsupportedWidth`), declared, and absent from every committed source (the only emoji of the sources, in `Horse.svg`, never reaches a generated text). [owner brief « Les différences éventuellement nécessaires doivent être déclarées individuellement »]
|
||||
Q: the binary's command name and a pnpm script? / A: no new pnpm script at R5.2 (a public name is the owner's to choose); the bin is reached through cargo. [CLAUDE.md « a name that becomes public → ask »]
|
||||
@@ -0,0 +1,70 @@
|
||||
# CONTRACT: Rust R5.4, the review files of the draft (review drawings, CHANGELOG, REVIEW-REPORT) and the whole-draft `--check`
|
||||
|
||||
- date: 2026-10-23 | branch `feature/rust-r5-pipeline-core` | plan `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md` §B.5 row R5.4
|
||||
- brief: owner 2026-10-23 « Validation R5.3 et autorisation conditionnelle R5.4 » §4 to §9 (conditions §1 met: R5.3 verifier CONFORME 13/13 on c6e0d85)
|
||||
- predecessor: `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md` (R5.3)
|
||||
|
||||
## Scope
|
||||
|
||||
Port `packages/anatomy/src/curation/{labels,changelog,review-report,overlay,app-review}.ts` and `stats.ts` `statRows` into
|
||||
`crates/core/src/curation/draft/**` (feature `curation`), complete `render_draft_artifacts`, and make `import-equine --check`
|
||||
compare the whole generated draft. Out of scope: the fingerprint convention (R5.6), the upload tool (R5.5), any change to
|
||||
the seed, to `horse.svg`, to an oracle or golden, to a gated reader, to `packages/anatomy` or `packages/domain` sources.
|
||||
|
||||
## Clauses
|
||||
|
||||
C1. **No I/O in the core.** The new modules take values and texts; only `bin/import_equine.rs` touches the file system.
|
||||
|
||||
C2. **Markdown byte parity.** `CHANGELOG.md` and `REVIEW-REPORT.md` produced by the core equal the committed files byte for
|
||||
byte; the raw texts (before the table alignment) equal the ledger's pre-Prettier digests of `oracles/r5/pipeline.json`.
|
||||
Changes, categories, references, statistics, line order, Unicode, escapes (`\|`, newlines in cells) and the 120-unit
|
||||
`compactValue` cut (UTF-16 units, `…`) as the TypeScript.
|
||||
|
||||
C3. **SVG drawings.** `review/annotations-{main,head}.svg` and `review/app-{main,head}.svg` are built on the core's tree from
|
||||
the frozen `draft/assets/horse.svg` read by the core's one SVG reader (nothing removed), never re-cleaned; the prolog of
|
||||
the source is kept verbatim. Each file is compared with the committed xmldom file byte for byte; where bytes differ, the
|
||||
parsed trees are compared (element names, namespaces, attributes and values, text, element order) and every difference is
|
||||
declared. The R3 owner decision (2026-10-15, a trigger without its own name carries the opened view's name per language)
|
||||
is the one expected difference, on `app-main.svg`; any graphic difference is listed for the owner's validation.
|
||||
|
||||
C4. **Whole-draft `--check`.** `import-equine --check` compares every file of `draft/` (12) and `legacy-import/` (6) with
|
||||
explicit criteria: byte identity; for a file with a declared tree rule, tree equality after that rule; it exits non-zero
|
||||
when a file is missing, not produced, or differs without an authorised rule. The `PENDING_R5_4` list and its `shortcut:`
|
||||
comment are gone. `--out` writes the 18 files (6 + 12) with the R5.3 protections.
|
||||
|
||||
C5. **Differential.** A TypeScript generator (`packages/anatomy/tests/r5.review-synthetic-oracles.test.ts`,
|
||||
`MAKE_R5_ORACLES=1` → `crates/core/tests/oracles/r5/review-synthetic.json`) freezes, on mini-sheet cases, the raw and
|
||||
formatted CHANGELOG, the raw and formatted REVIEW-REPORT, the review overlays and the app drawings; the Rust suite
|
||||
reproduces each one (Markdown byte for byte; SVG as C3) or refuses as the declared refusal (emoji width).
|
||||
|
||||
C6. **Determinism.** Two runs give the same bytes for every file.
|
||||
|
||||
C7. **Frozen references untouched.** No change under `supabase/seed/**`, `apps/app/tests/fixtures/**`,
|
||||
`packages/core-wasm/tests/n-1/**`, `crates/core/src/domain/**`, `crates/core/src/pdf/**`, `crates/core/tests/oracles/**`
|
||||
(except the new `review-synthetic.json`); `documentSha256` stays `27c7383c…2660`; `horse.svg` bytes unchanged.
|
||||
|
||||
C8. **R5.3 protections kept and LOW carry-over.** Strict changesets, path and symlink protections, no overwrite or delete,
|
||||
depth and resource bounds, R2 integrity gate unchanged; carried from the R5.3 final gate: the destination folder itself
|
||||
identity-checked, cite membership hashed, a failed seed walk named as such, the stale test comment, the R5.3 doc nits.
|
||||
|
||||
C9. **Gates.** `pnpm rust:check` and `pnpm check:all` green; security auditor PASS on the diff; this contract CONFORME by a
|
||||
fresh verifier; RUST.md §11.14 (R5.4) and §11.15 (checkpoint), CHANGELOG, CLAUDE.md. No merge, no R5.5, no fingerprint
|
||||
switch, no release, no deploy.
|
||||
|
||||
## Clarifications
|
||||
|
||||
- 2026-10-23 (gate round 2, C7): `crates/core/tests/oracles/r5/markdown-sweep.json` is a second new oracle of this slice,
|
||||
written by the TypeScript generator `packages/anatomy/tests/r5.markdown-sweep-oracles.test.ts` (`MAKE_R5_ORACLES=1`). It
|
||||
freezes what Prettier does to authored Markdown, from which the port's `UnsupportedMarkdown` guard is derived (C2 read as
|
||||
"byte for byte with the TypeScript, or a loud refusal declared in RUST.md §11.14"). It is added, never a frozen file
|
||||
changed; C7 excepts it with `review-synthetic.json`.
|
||||
|
||||
- 2026-10-23 (owner « Validation R5.4 et autorisation conditionnelle R5.5 » §2, FLOOR): the `floor-guard.sh` hits on a
|
||||
Markdown line under `.claude/audits/**` that only quote one of the guard's suppression keywords in prose (the audits
|
||||
state that none was added) are requalified as descriptive mentions.
|
||||
The clarification does not reach executable code, active contracts, production scripts, generated artefacts or a real
|
||||
removal: those findings keep failing. Past audit reports are not edited. Proof: `scripts/floor-clarification-check.sh`
|
||||
(requalifies by path and kind, then runs a throwaway repo where a real keyword in `src/*.ts`, `src/*.rs` and in an
|
||||
active contract is still reported). Result on this branch vs develop: 20 findings, 19 requalified (audit prose), 1 kept:
|
||||
a STUB marker in `.claude/memory/evals.md:205` (a quoted stub-marker phrase in a heading, append-only registry), outside the
|
||||
owner's clarification, reported to the owner and not requalified.
|
||||
@@ -0,0 +1,71 @@
|
||||
# CONTRACT: Rust R5.5, the sheet-asset upload tool (`upload-sheet-assets`) and the R5.4 closing adjustments
|
||||
|
||||
- date: 2026-10-23 | branch `feature/rust-r5-pipeline-core` | plan `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md` §B.5 row R5.5
|
||||
- brief: owner 2026-10-23 « Validation R5.4 et autorisation conditionnelle R5.5 » §1 to §10 and §12
|
||||
- predecessor: `.claude/tasks/contracts/2026-10-23-rust-r5-review.md` (R5.4)
|
||||
|
||||
## Scope
|
||||
|
||||
Port `packages/supabase/scripts/upload-sheet-assets.ts` into the core crate (`crates/core`, feature `upload`): rules in
|
||||
`src/curation/upload.rs`, binary in `src/bin/upload_sheet_assets/{main,storage}.rs`; scripts `scripts/upload-sheet-assets.sh`,
|
||||
`scripts/sheet-assets-e2e.sh`; pnpm `assets:upload` (Rust), `e2e:sheet-assets`, TypeScript kept as `assets:upload:legacy`
|
||||
until the R5.7 census. Plus the R5.4 closing adjustments (owner §2, §3, §6, §7). Out of scope: the fingerprint convention
|
||||
(R5.6), any seed or `documentSha256` change, a gated reader change, deleting `packages/anatomy` or `packages/domain`, any
|
||||
deploy, any production or managed-Supabase access.
|
||||
|
||||
## Clauses
|
||||
|
||||
C1. **Core free of I/O.** `curation::upload` takes values and returns values or `UploadError`; no `std::fs`, `std::env`,
|
||||
network or clock. `ureq` is an optional dependency enabled only by feature `upload` (never by the wasm facade or the
|
||||
worker); only the binary reads files, the environment and the network.
|
||||
|
||||
C2. **Local destination only, verified first.** There is no default destination. `SUPABASE_URL` must be `http`, carry an
|
||||
explicit port, no credentials, path, query or fragment, and a host that is `localhost`, a loopback address, or the private
|
||||
IPv4 address equal to `FAUNOSTEO_DEV_HOST`; anything else is refused (kind `Destination`) before any request. The key is
|
||||
never sent to a refused destination.
|
||||
|
||||
C3. **Inputs verified before the first request.** The folder lies under `supabase/seed/anatomical-sheets` (canonical path,
|
||||
symlinks resolved). `assets.json` is size-capped before parsing, strict on keys, plain `.svg` file names, lowercase sha256
|
||||
and uuid, size range, no repeated file, hash or asset id, 1 to 64 entries. Each drawing is read under a cap and equals its
|
||||
entry (size, sha256) and is accepted by the core's SVG reader with nothing removed. A failure stops the run with no request.
|
||||
|
||||
C4. **Handshake.** Before any object read or write: the bucket `anatomical-sheet-assets` exists, is private, and the key is
|
||||
accepted (401/403 are `Access`).
|
||||
|
||||
C5. **No overwrite, delete or visibility change.** Objects are created with `x-upsert: false`; an object under the key is
|
||||
read back and hashed: other bytes refuse the run (`State`), never replaced. A missing asset row, or a row recording another
|
||||
hash, refuses the run. The tool issues no DELETE, no PUT, no bucket or policy change. The plan of every entry is computed
|
||||
(read-only) before the first write; a run that cannot complete writes nothing.
|
||||
|
||||
C6. **Confirm after verify; safe resume.** The row is confirmed only after the object is read back and hashes to its key. After
|
||||
any failure a rerun resumes: a verified object is kept, a confirmed row is left alone. Transport errors and 5xx are retried
|
||||
three times, refusals are final.
|
||||
|
||||
C7. **Secrets.** The service key comes from the environment only, is held in a type whose `Debug` is redacted, travels in two
|
||||
request headers, never in a URL, a message or a log; the agent uses no proxy and no redirect; server error text is not echoed.
|
||||
|
||||
C8. **Closed errors and exit codes.** Failures fall in `Manifest | Destination | Content | Handshake | State | Access |
|
||||
Network`; exit 0 done, 1 refused, 2 usage (argument not UTF-8), 3 network. No failure silently drops an entry.
|
||||
|
||||
C9. **Identity and bytes preserved.** The object key is `<sha256>.svg` with the manifest hash, the row is addressed by the
|
||||
manifest `svg_asset_id`, the bytes uploaded are the file's bytes (no re-cleaning, no re-serialisation).
|
||||
|
||||
C10. **Tests.** `tests/curation_upload.rs` (manifest, bytes, destination, plan), `tests/upload_tool.rs` (the binary against a
|
||||
loopback stub: check writes nothing, run then rerun, different bytes, missing and foreign row, 401/403, resume after a failed
|
||||
confirm, unreachable port, refused destinations, folder outside the seed, usage), `tests/curation_caps.rs`;
|
||||
`scripts/sheet-assets-e2e.sh` for the real local stack (run by the owner; the assistant's own run was refused by the permission
|
||||
layer and is reported as such). `scripts/rust-check.sh` runs clippy and the tests with `--features upload`.
|
||||
|
||||
C11. **Runbook.** `docs/development/SUPABASE.md` §3.1, `docs/development/RUST.md` §11.16, CHANGELOG, CLAUDE.md commands.
|
||||
|
||||
C12. **Frozen references untouched.** No change under `supabase/seed/**`, `apps/app/tests/fixtures/**`,
|
||||
`packages/core-wasm/tests/n-1/**`, `crates/core/src/domain/**`, `crates/core/src/pdf/**`, `crates/core/tests/oracles/**`;
|
||||
`documentSha256` stays `27c7383c…2660`; `pnpm r5:import-equine --check` still reports 0 differences on 18 files.
|
||||
|
||||
C13. **R5.4 closing adjustments.** (a) `changes.json` and `references.json` are refused above 4 MiB before any parse allocation
|
||||
with an explicit message, in the core (`MAX_CURATION_FILE_BYTES`) and in the binary read; other budgets unchanged; (b) the FLOOR
|
||||
clarification of the R5.4 contract holds with its negative proof (`scripts/floor-clarification-check.sh`); (c) no gated reader
|
||||
and no serde tolerance is changed (characterisation only, RUST.md §11.16).
|
||||
|
||||
C14. **Gates and limits.** `pnpm rust:check` and `pnpm check:all` green; security auditor PASS on the diff; this contract
|
||||
CONFORME by a fresh verifier. No merge, no R5.6, no release, no deploy, nothing outside the local stack.
|
||||
@@ -0,0 +1,43 @@
|
||||
# CONTRACT: R5.5 closing corrections and R5.6 preparation
|
||||
|
||||
- date: 2026-10-24 | branch `feature/rust-r5-pipeline-core` | base dead546..13db553 (R5.5 checkpoint)
|
||||
- brief: owner 2026-10-24 « Clôture R5.5, corrections ciblées et préparation R5.6 » §1 to §9
|
||||
|
||||
## Clauses
|
||||
|
||||
K1. **Owner result recorded.** RUST.md §11.18, TODO, journal and registries record `pnpm e2e:sheet-assets` PASS (11 checks) as
|
||||
the owner's manual run, distinct from the simulated suite; no new measurement invented, no rerun with the service key, no secret
|
||||
in any file or commit.
|
||||
|
||||
K2. **Retry.** `Protocol` errors are final (no retry); only `Io`, `Timeout`, `HostNotFound`, `ConnectionFailed` and 5xx are
|
||||
retried; negative test proves one request on a malformed answer.
|
||||
|
||||
K3. **DEV address.** The DEV address counts only if it is an own unicast address; a subnet broadcast is refused; tests cover
|
||||
loopback, the host's unicast and broadcast addresses, a public and a malformed value.
|
||||
|
||||
K4. **Serde hardening, three shapes only.** `sheet` and `practitioner` as non-objects and `severity` as a non-string are refused
|
||||
with `ReadFailure::Invalid` (field named) at the v3 entries `read_report_document` and `parse_envelope`; no other contract,
|
||||
version or field narrowed or widened; frozen N-1 copies, goldens, fixtures and oracles untouched; every R2-R4, PDF import,
|
||||
rebase and envelope suite green unchanged.
|
||||
|
||||
K5. **SQL characterised, not changed.** No file under `supabase/migrations` or `supabase/tests` changes; RUST.md §11.18 states the
|
||||
grants, the missing hash/existence check and why it is not exploitable from an API role.
|
||||
|
||||
K6. **DEV stack documented, not changed.** ENVIRONMENTS.md §3.6 lists the 54321 checks and the default-key finding; no firewall,
|
||||
network or Supabase configuration changed.
|
||||
|
||||
K7. **R5.6 plan only.** `.claude/tasks/plans/2026-10-24-rust-r5-6-fingerprint-switch.md` covers the owner's 12 points and lists the
|
||||
open decisions; no fingerprint code, no seed or `sheet-ref.json` change, `documentSha256` stays `27c7383c…2660`.
|
||||
|
||||
K8. **Gates.** `pnpm rust:check` and `pnpm check:all` green; security PASS on the diff; fresh verifier on this contract. No merge,
|
||||
no release, no deploy, no R5.6.
|
||||
|
||||
## Clarifications
|
||||
|
||||
- 2026-10-24 (closing gates round 2, K4, orchestrator, submitted to the owner at the checkpoint): at `parse_envelope` the
|
||||
refusal is a `serde_json::Error` naming the field (the function's existing signature), not `ReadFailure::Invalid`; its one
|
||||
caller (`session.rs`, gated) drops the archive copy as before. Changing the signature would touch `session.rs`, outside this
|
||||
go. Refusing a positional report, observation or envelope is part of K4: without it the three named shapes pass unseen
|
||||
under a positional parent (Zod refused those containers too).
|
||||
- 2026-10-24 (FLOOR): the R5.4 clarification (`.claude/audits/**` Markdown prose quoting a suppression keyword is a
|
||||
descriptive mention) applies to this contract's audit reports; nothing else is requalified.
|
||||
@@ -0,0 +1,58 @@
|
||||
# CONTRACT: R5.6 switch rehearsal (isolated), owner brief 2026-10-24 « Validation de la préparation R5.6 et corrections préalables à la bascule »
|
||||
|
||||
- branch `feature/rust-r5-pipeline-core` at 943a25a (official state, inactive); patch `.claude/tasks/plans/r5-6-switch.patch`
|
||||
- rehearsal: detached worktree `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/3571c861-cd6e-4751-a25e-c26da95beb85/scratchpad/r56wt` = 943a25a + the patch, seed switched there only
|
||||
|
||||
## Clauses
|
||||
|
||||
S1. **M1 option 1.** On export, an old report whose stamp is the registered historical value of the same sheet, against the
|
||||
sheet's verified canonical value, archives a snapshot carrying the report's own stamp; only for the registered pair of the
|
||||
same sheet id, with the loaded sheet's `v1` value recomputed from its document at load; never from an archive's alias list; a
|
||||
new report keeps the canonical value. Tests cover the owner's 9 scenarios (old PDF import, no rebase, re-export coherent,
|
||||
current reader, frozen N-1 reader, labels/views/annotations/branding preserved, rollback reader, forged/unknown refused, real
|
||||
change rebases), not only hash equality.
|
||||
|
||||
S2. **Scope.** Rust changes only in `report/rebase.rs`, `archive/read.rs` (unchanged allowed), `session.rs`,
|
||||
`pdf/write/input.rs`, `pdf/compare/lines.rs`, `sheet/loaded.rs`, `facade.rs`, `bin/report_compare.rs`, plus the
|
||||
`sheet/fingerprint.rs` module, the generator (`curation/draft/pipeline.rs`), Cargo/tests/scripts; archive and envelope checks not
|
||||
loosened; public errors kept where possible; N-1 readers untouched.
|
||||
|
||||
S3. **v0/v1 loading.** A reference without a block is historical (v0): loads as before, unverified, no alias privilege, never
|
||||
`v1`. A present block is strict (known keys, `v1` convention, 64 lowercase hex, not an alias value, aliases only from the
|
||||
registry for this sheet and value, value recomputed from the document); any contradiction refused.
|
||||
|
||||
S4. **Artifacts and oracles.** In the switched seed only `sheet-ref.json` and `report-migration-map.json` differ; their value
|
||||
and block are consistent with `v1`; `document.json`, `horse.svg`, `equine_draft.sql` byte-identical; no R2-R5 oracle or golden
|
||||
changed; O7, O10, the R2 Zod oracle, the R5.1 ledger and the app goldens run in their original (pre-switch) context, with a
|
||||
separate test of the post-switch behaviour; every declared difference targeted, tested, unable to hide an anatomical change.
|
||||
|
||||
S5. **report:compare.** The alias line keeps the draft/published status and prints both raw values; other differences still
|
||||
reported.
|
||||
|
||||
S6. **Generators.** `pnpm import:equine` → the Rust pipeline; `import:equine:legacy` → the TypeScript one; historical tests use
|
||||
the generator they refer to.
|
||||
|
||||
S7. **Isolation.** The official branch state stays inactive (seed `27c7…`, no new convention active), no DB/Supabase mutation,
|
||||
no oracle regenerated, nothing deleted.
|
||||
|
||||
S8. **Gates on the switched state.** `pnpm rust:check`, `pnpm check:all`, RFC 8785, Node differential, alias security, the five
|
||||
consumers + load checks, authentic archive parity, N-1 on re-exported archives, snapshot-bearing reports, real-change rebase,
|
||||
the two generated artifacts; security audit; fresh verifier.
|
||||
|
||||
## Clarifications
|
||||
|
||||
- 2026-10-25 (owner « autorisation de correction M1, nouvelle répétition obligatoire »): scope extended by
|
||||
`domain/archive/snapshot_rules.rs`; the M1 decision moves into `build_sheet_snapshot` and `assert_exportable`
|
||||
(`pdf/write/input.rs`) returns to strict equality (unchanged from 943a25a). Rules added: an alias is granted only by
|
||||
`LoadedSheet::recognises_historical_stamp` (loader-verified `v1` value, private state, recomputed from the current
|
||||
document at the decision, registry pair, same sheet id); `"fingerprint": null` and malformed blocks refused; warnings
|
||||
never block a `v1` fingerprint (`is_publishable`), errors do; `canonical_json` bounded at 128 levels with a controlled
|
||||
error; the direct export API is tested, not only the application paths. Final state: worktree
|
||||
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/3571c861-cd6e-4751-a25e-c26da95beb85/scratchpad/r56final`
|
||||
(detached at 8e77e37 + the patch, seed switched by the patch).
|
||||
|
||||
- 2026-10-25 (owner « Autorisation officielle de la bascule R5.6 »): S7 (isolation) is superseded for the activation: the
|
||||
owner authorised the official switch on `feature/rust-r5-pipeline-core` (commit 9e87016) with `equivalence` crate-private,
|
||||
stale comments fixed, the trust boundary documented, and the regeneration of exactly `draft/sheet-ref.json` and
|
||||
`draft/report-migration-map.json` (procedure: fresh temporary `--out`, comparison, the two files only, `--check`). The other
|
||||
S7 parts still hold: no database or Supabase mutation, no oracle regenerated, nothing deleted, no merge.
|
||||
@@ -0,0 +1,71 @@
|
||||
# CONTRACT: R5.7 consumer census and deletion of `packages/anatomy` and `packages/domain`, owner brief 2026-10-25 « Validation R5.6 et autorisation R5.7 »
|
||||
|
||||
- branch `feature/rust-r5-pipeline-core`, base 756b7ef (R5.6 active since 9e87016)
|
||||
- plan `.claude/tasks/plans/2026-10-25-rust-r5-7-cleanup.md`
|
||||
|
||||
## Clauses
|
||||
|
||||
C1. **Census.** Every consumer of `packages/anatomy` and `packages/domain` outside them is listed (runtime and type
|
||||
imports, fixtures read by Rust, TS tests, oracles and goldens, pnpm scripts, build dependencies, seed scripts, docs,
|
||||
dev and check tools, historical references). Nothing still required by an active feature is deleted.
|
||||
|
||||
C2. **Matrices.** `crates/core/tests/curation/FILES.tsv` names every deleted tracked file with its Rust counterpart or the
|
||||
reason it goes; `crates/core/tests/curation/LEDGER.tsv` has one row per TS case of both packages at 756b7ef (407), with the
|
||||
case's contract, old location, equivalent after the deletion, status ported / replaced / kept / deleted (with the
|
||||
justification), and the historical reference kept. No row is a gap: a contract still relevant and uncovered is ported first.
|
||||
|
||||
C3. **Fixtures.** `report-v2.full.json` and `report-v2.minimal.json` move under `crates/core/tests/fixtures/` with their bytes
|
||||
unchanged (sha256 before = after, recorded); the Rust readers use the new place. No oracle, golden, frozen digest, migration
|
||||
data, N-1 reader or reference PDF is changed, regenerated or removed; a moved reference file keeps its sha256.
|
||||
|
||||
C4. **Active checks preserved.** The contracts the brief §5 lists stay covered by tests that run in `pnpm rust:check` or
|
||||
`pnpm check:all` (sheet model validation, reference integrity, deterministic UUIDs, legacy catalog conversion, changeset
|
||||
application, migration map, statistics, seed SQL, SVG rendering, RFC 8785, historical aliases, PDF import and export, old
|
||||
archive compatibility); the application and interface suites keep passing; test counts before and after are measured and
|
||||
documented.
|
||||
|
||||
C5. **Historical reproducibility.** The documentation names the reference commits of the TypeScript pipeline, the versions
|
||||
(Node, pnpm, dependencies), the historical generation and comparison commands, the isolated-worktree procedure and the expected
|
||||
sha256 of the artifacts; the procedure is rehearsed once; the old engine is not reintroduced in the current tree.
|
||||
|
||||
C6. **Dependencies and scripts.** The two packages' own dependencies leave the lockfile through `pnpm install`; `@xmldom/xmldom`
|
||||
is gone; no workspace manifest declares `tsx`, no script runs it, and the lockfile may keep it only as a transitive optional peer
|
||||
the package manager resolves itself (Clarification C6 below); root `prettier` stays (used by `pnpm format` and `db:types`); `pnpm import:equine` stays the Rust pipeline;
|
||||
`import:equine:legacy` and `assets:upload:legacy` are removed; no other dependency changes major version; the lockfile installs
|
||||
with `--frozen-lockfile`.
|
||||
|
||||
C7. **Census tests.** `r4.census.test.ts` proves xmldom and the TS engines are no active dependency, and cannot pass on a
|
||||
lockfile or tree that reintroduces them (a non-vacuity check); a census proves no production path (app, glue, supabase package,
|
||||
Dockerfile, Vite/TS config) imports a deleted engine, and that the two matrices agree with the tree.
|
||||
|
||||
C8. **Docs.** CLAUDE.md, WORKSPACE.md, packages/README.md, RUST.md §11, CHANGELOG, TODO, the curation commands and the test
|
||||
registers are updated; the seed SQL header is kept (frozen bytes) and explained beside it; audits are append-only; the brief §10
|
||||
follow-ups stay tracked (rollback limit, DEV hardening, migration-map cascade debt, direct export API trust boundary, SQL checks
|
||||
before public exposure, accepted LOW items).
|
||||
|
||||
C9. **Checks.** `pnpm install --frozen-lockfile`, `pnpm rust:check`, `pnpm check:all`, `pnpm import:equine --check` (0
|
||||
difference), the RFC 8785, alias and archive suites, PDF and `report:compare`, the N-1 reader, the moved fixtures, the sheet and
|
||||
seed integrity suites, and the absence of any import of the deleted packages, all green on the final state.
|
||||
|
||||
C10. **Limits.** Branch only; no merge, no release, no deploy, no Supabase DEV data change, no migration, no new feature; no
|
||||
seed byte changed.
|
||||
|
||||
## Clarification C6 (owner, 2026-10-25, brief « Validation R5.7 et dernière vérification pré-fusion » §1)
|
||||
|
||||
The first wording read « `@xmldom/xmldom` and `tsx` are gone ». The owner accepts explicitly that `tsx` stays named in
|
||||
`pnpm-lock.yaml` when it is only an optional peer dependency of Vite. The objective C6 checks is: no old TypeScript pipeline
|
||||
active; no functional import from `packages/anatomy` or `packages/domain`; no script kept that requires them; no direct `tsx`
|
||||
dependency left without use; legitimate transitive or optional references of the package manager may stay. The lockfile is
|
||||
not edited by hand and no override or exclusion is added to drop the name.
|
||||
|
||||
Measured path (`pnpm why tsx -r`, 2026-10-25): `tsx` 4.23.15 is reached only as the `peer` of `vite` 7.3.6 (vite declares
|
||||
`tsx: ^4.8.1` under `peerDependencies` with `peerDependenciesMeta.tsx.optional: true`), through `vite`, `vitest`,
|
||||
`@vitest/mocker`, `vite-node`, `@tailwindcss/vite` and `@vitejs/plugin-react-swc` in the devDependencies of `apps/app`,
|
||||
`packages/core-wasm` and `packages/supabase`; its snapshot block carries `optional: true`; no importer lists it; no
|
||||
root `node_modules/.bin/tsx` is linked (pnpm links the peer's bin under the
|
||||
`node_modules/.bin` of `apps/app`, `packages/core-wasm` and `packages/supabase`; no script calls it; verifier precision 2026-10-25).
|
||||
|
||||
The exception does not hide a consumer: `r5.census.test.ts` still fails if a manifest declares `tsx` (any dependency field),
|
||||
if a manifest script names `tsx`, `@faunosteo/{anatomy,domain}` or `packages/{anatomy,domain}` or ends in `:legacy`, if an importer block lists `tsx`, if the
|
||||
`tsx` snapshot loses `optional: true`, or if any app, Vite, glue or Supabase-package source imports a deleted package; the
|
||||
verifier checks these probes, not the lockfile name.
|
||||
@@ -0,0 +1,147 @@
|
||||
# PLAN: Rust R5, the anatomical pipeline (legacy importer, curation, migration map, seed, upload tool), `packages/anatomy` and `packages/domain` deleted
|
||||
|
||||
- date: 2026-10-22 | rev 1 (R5.1 only; the later slices are a proposal for the R5.1 checkpoint) | branch: `feature/rust-r5-pipeline-core` (cut from develop 9c0117a, R4 merged)
|
||||
- spec: owner brief 2026-10-22 « Clôture R4 validée et autorisation R5.1 » (§A.0); `docs/development/RUST_ARCHITECTURE_REVIEW.md` II.2 rows "anatomy curation pipeline" and "upload-sheet-assets" (lines 299, 300), VI.1 row R5 (636), VI.3 (655), VI.4 (662 to 668), D-R5 (837, 873), Annex C D-R5 amendment (974); BDR-043, BDR-045; CLAUDE.md "Anatomical sheet model", "Schemas", archive-reader gate
|
||||
- facts: `.claude/audits/R5_PIPELINE_INVENTORY.md` (the TypeScript pipeline), `.claude/audits/R5_DOCUMENT_SHA256_CENSUS.md` (the fingerprint), the oracles under `crates/core/tests/oracles/r5/`, the study scripts `scripts/r5-fingerprints.mjs` and `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`
|
||||
- status: R5.1 validated 2026-10-22; D-R5 decided (A.5); R5.1b and R5.2 validated 2026-10-22; R5.3 done (contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`, RUST.md §11.12), checkpoint R5.3 pending; R5.4 on the owner's go.
|
||||
- not in scope of R5.1: any Rust implementation of the pipeline, any change to `documentSha256` or to the readers, the deletion of `packages/anatomy` or `packages/domain`, any golden or historical reference regeneration, release 0.2.0, deployment, SaaS features.
|
||||
|
||||
# Part A. For the owner
|
||||
|
||||
## A.0 Your brief (2026-10-22), applied as written
|
||||
|
||||
Branch from develop by gitflow (done). Existing plan and decisions read, no new architecture study. R5.1 = characterise the TypeScript pipeline and freeze its results (legacy import, draft, changesets, migration maps, CHANGELOG and REVIEW-REPORT, review drawings and overlays, SVG assets and upload, seed SQL, current fingerprints, determinism, compatibility with older versions and archives); reuse the existing oracles, fixtures and goldens where they suffice; keep the N-1 readers; never bend a historical reference to a new implementation. D-R5: a targeted, reproducible study of option A (sha256 of the bytes the Rust pipeline emits) against option B (sha256 of RFC 8785 canonical JSON) on the project's real data, a matrix of old and new fingerprints, an encoding change told apart from a content change, a versioned convention and a compatibility mechanism for the old fingerprints, no silent loss, no visible migration just to hide an algorithm change when a transparent and safe compatibility exists. Business invariants of §4 preserved by every later migration; no sheet declared validated by a technical migration. Stop before any Rust pipeline code and before any fingerprint change.
|
||||
|
||||
## A.1 What R5 delivers (the whole phase, for orientation)
|
||||
|
||||
The curation pipeline (legacy import, changeset application, migration map, CHANGELOG, REVIEW-REPORT, review drawings, cleaned asset, seed SQL) and the asset upload tool run in Rust (`crates/core` curation modules behind a feature, `cli` binaries `import-equine` and `upload-sheet-assets`), with the same inputs (legacy sources, `curation/changes.json`, `curation/references.json`) and the same semantics (ids, counts, 612-leaf migration, integrity rules). `packages/anatomy` and `packages/domain` leave the repository once nothing imports them; the N-1 Zod reader under `packages/core-wasm/tests/n-1` stays. The sheet fingerprint follows the D-R5 decision.
|
||||
|
||||
## A.2 R5.1 deliverables (this slice)
|
||||
|
||||
1. Inventory of the TypeScript pipeline: `.claude/audits/R5_PIPELINE_INVENTORY.md`.
|
||||
2. Oracles frozen with the TypeScript code (§B.2), generated by `MAKE_R5_ORACLES=1` from `packages/anatomy/tests/r5.pipeline-oracles.test.ts`, which otherwise checks the pipeline still matches its freeze.
|
||||
3. The D-R5 study (§B.3): facts, the fingerprint matrix, the compatibility mechanism, the versioned convention, costs, platform determinism, I-JSON and Unicode checks, the incidence on the archives already exported.
|
||||
4. The checkpoint report (the owner's §5 list), written in RUST.md §11 and delivered in the conversation.
|
||||
|
||||
## A.3 Findings that need your decision at the checkpoint
|
||||
|
||||
- D-R5 itself (§B.3.6): A, B or C (reproduce Prettier's bytes), the canonical form (input or validated), the compatibility mechanism, the convention's name.
|
||||
- F1, `js_number` tie rounding (§B.3.4): the core prints an exact decimal tie upward (`840.1692504882813`) where JavaScript, Python and `zmij` print to even (`…812`); 1 value of the legacy sources, 7.4 % of float32-exact coordinates, 0.8 % of random doubles. Invisible today (the curated draft holds no tie; the R2 contract test re-prints its expected text through the same function). The fix is small (digits from `zmij`, already locked through serde_json) and changes core bytes only for tie values; it precedes any Rust importer. Your go is needed because it touches a core serialiser (D-R11).
|
||||
- The documentation contradiction: BDR-045 and Annex C say "existing fingerprints untouched", RUST.md §10 and the TODO still say "will change once". The plan follows BDR-045 (the later amendment) until you say otherwise.
|
||||
- Byte-preserving port or one-time regeneration of the generated files (§B.5): the Rust pipeline can reproduce the JSON bytes only by emitting insertion order and Prettier's short-array rule, and the Markdown bytes by padding tables (the only Prettier effect measured); the review SVGs cannot match xmldom's serialiser and are compared as parsed trees. The recommendation is: JSON files byte-exact where cheap (`to_js_pretty` plus the one array rule) or regenerated once with a JCS-equality proof, Markdown byte-exact by padding, SVGs regenerated once with a tree proof, and `supabase/seed/**` added to `.prettierignore` at the switch (today a root `pnpm format` would reformat Rust-written files).
|
||||
- `draft/assets/horse.svg`: keep the xmldom-cleaned bytes as the frozen asset (sha `0ea5202f…`, id `7f767f74…`, already the Storage object) and have the Rust tool verify rather than regenerate it, or re-clean it with the R3 reader and accept a new asset sha, id, document, SQL and Storage upload. Recommendation: keep the frozen bytes (the cleaning was a one-time legacy transformation; the R3 reader already proves the file).
|
||||
|
||||
## A.4 Stop rules of R5.1
|
||||
|
||||
Stop and report when: a generated file does not reproduce from a fresh run (the golden test fails); an oracle generator would have to alter a committed generated file or golden; a study step needs a change to a gated reader; the branch would need a develop merge.
|
||||
|
||||
## A.5 Owner decisions (2026-10-22, BDR-068)
|
||||
|
||||
R5.1 validated. D-R5 = option B, convention `sha256-rfc8785-validated-v1`, applied at R5.6 only, proven by the RFC vectors, edge cases and an independent differential; `v1` freezes the construction and validation rules. Aliases of the historical fingerprints from trusted frozen documents only (never from an archive or a bundle); Report v3 `sheet.documentSha256` keeps 64 hex; an unknown fingerprint is never an official sheet; a real anatomical change still migrates. F1 fixed first (R5.1b), with an independent reference. Byte-exact reproduction by default, every difference declared. Historical SVG bytes kept (never re-serialised). `.prettierignore` on the Rust-written paths, narrow, with a reproducible check. `ureq` preferred for the upload tool (R5.5). Slicing B.5 validated; stop after R5.2 (checkpoint: F1 result and impact on R2-R4, importer parity, byte differences, tests and audits, residual risks, conditions for R5.3).
|
||||
|
||||
# Part B. The plan
|
||||
|
||||
## B.1 Inventory
|
||||
|
||||
See `.claude/audits/R5_PIPELINE_INVENTORY.md`: 43 source files, 4,897 lines (legacy 1,842, curation 2,329, render 553, cli 119), 2,553 lines of tests (131 cases, 16 files), runtime libraries Prettier 3.8.3, @xmldom/xmldom 0.9.12, Zod 3.25.76 (the first two declared as devDependencies). 18 generated files: 7 JSON and 3 Markdown through Prettier, 5 SVGs through xmldom's XMLSerializer, 1 raw SQL template, 2 byte copies. The real changeset: 45 changes (14 A, 22 B, 9 C), 687 ops (587 set, 45 cite, 33 rename, 18 add, 4 remove, 0 move), 48 references. Measured Prettier effect on today's data: Markdown table padding only (134, 863 and 52 lines), JSON short scalar arrays collapsed plus a trailing LF; the data is identical after re-parsing. The JSON key order is the working form's insertion order (`review_status` appended last), not the schema order a typed serde struct would emit. `@faunosteo/anatomy` has no consumer outside itself; `@faunosteo/domain` is imported by `packages/anatomy` only (5 runtime symbols, all with R2 counterparts); the crate tests read `packages/domain/tests/fixtures/report-v2.*.json` and `r4.census.test.ts` pins xmldom under `packages/anatomy`. Entry points: `importEquineLegacy` (legacy sources → ImportResult), `buildEquineDraft` (ImportResult + curation → DraftResult), `renderArtifacts` (4 legacy files), `renderDraftArtifacts` (document, assets, sheet-ref, migration map, 4 review SVGs, CHANGELOG, REVIEW-REPORT, seed SQL), `cli/import-equine.ts` (writes both folders, removes stale files), `packages/supabase/scripts/upload-sheet-assets.ts` (bucket `anatomical-sheet-assets`, key `<sha256>.svg`, `upsert: false`, bytes verified against `assets.json`, RPC `confirm_sheet_asset_upload`). JavaScript semantics to pin: `deterministicUuid` (sha256 of the name, version nibble 5, variant 10xx), `slugify` (NFD, combining marks U+0300 to U+036F stripped, `toLowerCase`, `[^a-z0-9]+` → `-`, trimmed, 64 chars), `sha256Hex` over UTF-8, `JSON.stringify(v, null, 2)` then Prettier json (short scalar arrays collapsed when they fit in 80 columns, objects always expanded), Prettier markdown, xmldom `XMLSerializer` for the review SVGs, `TextDecoder('utf-8', { fatal: true })`.
|
||||
|
||||
## B.2 Oracles frozen at R5.1 (`crates/core/tests/oracles/r5/`, generator `MAKE_R5_ORACLES=1`, never regenerated without the owner)
|
||||
|
||||
| Id | Content | Location | Used by |
|
||||
|---|---|---|---|
|
||||
| P1 | the committed generated folders `legacy-import/` (5 files) and `draft/` (14 files): the byte oracle, already pinned to a fresh run by `seed.golden.test.ts` | `supabase/seed/anatomical-sheets/equine/{legacy-import,draft}` | every R5 slice |
|
||||
| P2 | `pipeline.json`: legacy ImportResult without the document (sheetId, viewBoxes, stats, source digests, assets, 27 anomalies, issues, region inferences, zone stats, annotation rows); DraftResult without the documents (46 changes, assets, stats before and after, issues, migration stats, 825 change records with before and after); the ledger of the 16 text artifacts (folder, file, bytes, sha256; for the 3 markdown files the sha256 before Prettier and whether Prettier changed the text) | `oracles/r5/pipeline.json` (1.3 MB) | R5.2 to R5.5 |
|
||||
| P3 | `ids.json`: `deterministicUuid` over the 3 names the pipeline derives ids from plus 3 edges; `slugify` over 17 edge labels and every `name.fr` / `name.en` of both documents | `oracles/r5/ids.json` | R5.2 |
|
||||
| P4 | `fingerprints.json`: for the 7 JSON artifacts the fingerprint under each rule (today's file bytes, A pretty with and without LF, A compact, B RFC 8785; for the two documents also the validated form under A and B); the ES6 text of the 65 non-integer numbers of the two documents; the two RFC 8785 vectors | `oracles/r5/fingerprints.json` | D-R5, R5.2 |
|
||||
| P5 | migration semantics: `draft/report-migration-map.json` (P1) and the 612-leaf suite `curation.migration-leaves.test.ts`, plus the R2 oracle `oracles/migration-map.draft.json` and the Rust `archive_support` walk | existing | R5.3 |
|
||||
| P6 | the review drawings `draft/review/{annotations,app}-{main,head}.svg` (P1, xmldom bytes) and the R3 layer and highlight oracles | existing | R5.4 (parsed-tree comparison) |
|
||||
| P7 | the upload tool's contract (B.1) and the operator check SUPABASE.md §3.1 | documented | R5.5 |
|
||||
| P8 | determinism: two in-process runs byte-equal (test in the generator file); `pnpm import:equine` not run (it rewrites the folders; the golden proves the same bytes) | test | every slice |
|
||||
|
||||
## B.3 D-R5 study
|
||||
|
||||
### B.3.1 Today's rule
|
||||
|
||||
`documentSha256 = sha256Hex(formatJson(document))`: the Prettier-formatted JSON text of the curated document in its input form (defaults omitted), the exact bytes of `draft/document.json`, embedded verbatim in `equine_draft.sql` and copied into `sheet-ref.json` and `report-migration-map.json`. One value has ever existed: `27c7383c…2660`, unchanged since 0a17e39 (2026-10-10), inside 99 committed PDFs, 8 JSON fixtures and 6 Rust oracle files. Compared, never recomputed, by the rebase (fingerprint only), the envelope invariant and the export (sheetId and fingerprint). The database stores the document as `jsonb` and no fingerprint. Full census: `R5_DOCUMENT_SHA256_CENSUS.md`.
|
||||
|
||||
### B.3.2 The matrix (real data, `node scripts/r5-fingerprints.mjs`, cross-checked in Rust)
|
||||
|
||||
| Artifact | Today (file bytes) | A: `to_js_pretty` + LF | A without LF | B: RFC 8785 | B validated form |
|
||||
|---|---|---|---|---|---|
|
||||
| draft/document.json | 27c7383c…2660 | 90f668a3…aa1d | 79f143c0…eb51 | a56e4a24…88a1 | 12d78872…74e2 |
|
||||
| legacy-import/document.json | 91e459f4…14c2 | 8c94a464…adc2 (Node) | 6fb31371…7f66 (Node) | d8b5f2cf…81eb (Node) | 1217637e…c024 (Node) |
|
||||
| draft/report-migration-map.json | 4139c51d…fce9 | 4139c51d…fce9 (same) | 01fac788…3471 | f082b66f…2ca6 | n/a |
|
||||
| draft/assets.json, sheet-ref.json, legacy-id-map.json, legacy assets.json | file = A with LF (no short array to collapse) | | | | |
|
||||
|
||||
Rust reproduction (throwaway experiment on the core crate, `to_js_pretty`, `to_js_compact`, a 40-line JCS on `js_number`): every draft digest identical to Node, byte for byte; the legacy-import digests differ through the single tie value (F1, §B.3.4); the RFC vectors pass; JCS idempotent. Node vs Rust `JSON.stringify(v, null, 2)` over the 134 JSON fixtures of the repository: 2 differ, both the legacy baseline, both on that value; `oracles/characterisation.json` is not parseable by serde (lone surrogates, R2 L1 rule).
|
||||
|
||||
Encoding change vs content change: the today / A / B columns of one row are three encodings of one value (parse → deep equal; `validated_equals_input` is false only because the validated form fills defaults, not because content differs). A change of the draft content changes every column.
|
||||
|
||||
### B.3.3 Facts the brief asked for
|
||||
|
||||
- UTF-16 key order: every key of every artifact is ASCII; UTF-16 order equals code-point and byte order on this data. The Rust canonicaliser sorts by `encode_utf16()` anyway (the RFC vector with U+0080, ö, €, 😀 and U+FB33 passes).
|
||||
- Numbers: 208 (draft) and 195 (legacy) tokens, 29 and 36 non-integer, none negative, no exponent, no `-0`, no integer beyond 2^53; all already in ECMAScript `Number::toString` form. JCS mandates exactly that form; the core's `js_number` implements it with one tie defect (F1).
|
||||
- Unicode: 788 non-ASCII strings (precomposed French accents, « », –, °, —), all NFC, no control character, no escape, no astral character, no lone surrogate. JCS does not normalise; a lone surrogate (I-JSON forbids it) is refused by the core (L1) and never produced by the pipeline.
|
||||
- I-JSON: satisfied by both documents (no duplicate key, numbers within f64, valid Unicode).
|
||||
- Cost and dependencies: option A costs nothing (the core already has `to_js_pretty`, D-R11); option B costs about 40 lines in `crates/core/src/json.rs` over `js_number` and serde_json's escaper, no new crate (`serde_jcs`, listed as rejected in the review, is not needed; `zmij` 1.0.23 is already locked through serde_json 1.0.154); a 15-line Node twin for tests and tooling. Hashing: `sha2` becomes a normal dependency of the core (today dev-only) or the hash stays outside the library (the tool computes it); the wasm bundle grows by the sha256 code only if the browser must verify fingerprints, which no current path does.
|
||||
- Platform determinism: B is a pure function of the parsed value (sort, ES6 numbers, fixed escapes); Node 24 and Rust 1.99 agree on every draft artifact. A depends on the writer's layout rules; today's value depends on Prettier 3.8.3's line fitting.
|
||||
- Database: a PostgreSQL 17 `jsonb` round trip (PGlite, `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`) keeps B for both documents and breaks today's value and A (keys reordered, scale kept). B is the only candidate a service could recompute from `anatomical_sheet_versions.content` at Step 13.
|
||||
|
||||
### B.3.4 F1: `js_number` prints exact ties upward
|
||||
|
||||
`crates/core/src/json.rs` takes the shortest digits from `format!("{:e}")` (Grisu); on an exact decimal tie between two shortest candidates ECMAScript picks the even one, Grisu the upper one. Measured: `840.1692504882812` (legacy annotation `cx`, from Horse.svg) prints `…813`; 7,401 of 100,000 float32-exact values in 0..2000 and 769 of 100,000 random doubles differ; `zmij` (serde_json's formatter) differs on none. Reached by the canvas layer (R3), the PDF writer, the archive attachment, the R2 contract tests. Invisible today: the curated draft carries no tie value, and `sheet_contract.rs` re-prints its expected text through `to_js_pretty`, so the oracle cannot see it. Proposed fix (after your go, before R5.2): digits from `zmij` (or `serde_json::to_string(&f64)`), the two tie values and the 8 long decimals of the legacy SVG as unit tests, `zod_oracle` comparing against the JS text recorded in the oracle rather than a re-print.
|
||||
|
||||
### B.3.5 Compatibility mechanism (approved at A.5, applied at R5.6)
|
||||
|
||||
The sheet side carries the convention and the aliases; the report contract does not change.
|
||||
|
||||
- `sheet-ref.json` (and the bundle payload `ref`, and `anatomical_sheet_versions` when Step 13 stores a fingerprint): `document_sha256` holds the current convention's value; a new `fingerprint` block names the convention (`"algorithm": "rfc8785-sha256"` for B, `"js-pretty-sha256"` for A) and lists `aliases`: the fingerprints of the same content under earlier conventions, computed once at the switch from the frozen P4 oracle (`27c7383c…2660` for the current draft). The alias is bound to identical content by construction (the oracle records old and new over the same bytes), so a report stamped `27c7…` is the same sheet, not a rebase.
|
||||
- The rebase and the envelope invariant treat a fingerprint equal to the current value or to an alias as "same document". Gated readers: `report/rebase.rs`, `archive/read.rs`, `session.rs`, `pdf/write/input.rs` (your ask-before-touch rule; the change is one membership test).
|
||||
- A new report is stamped with the current value; an old archive opens with no message and no rebase; an archive on an older content revision rebases as today. No archive is rewritten; the exported envelope keeps what it holds.
|
||||
- Report v3 keeps `sheet.documentSha256` (64 hex) so the N-1 reader and every fixture stay valid; the convention's name lives with the sheet, which is the one place that knows which rule produced its value.
|
||||
- Fixtures: 99 PDFs, 8 JSON fixtures and 6 oracles keep `27c7…` untouched (they remain valid archives of the same content through the alias).
|
||||
|
||||
### B.3.6 Options compared
|
||||
|
||||
| | A: bytes the Rust pipeline emits (`to_js_pretty` + LF) | B: RFC 8785 of the validated document | C: Rust reproduces Prettier's bytes |
|
||||
|---|---|---|---|
|
||||
| Value at the switch | 90f668a3… (changes once) | 12d78872… (changes once) | 27c7383c… (unchanged) |
|
||||
| Needs the alias mechanism to avoid a visible migration | yes | yes | no |
|
||||
| Survives a reformat, a writer change, a jsonb round trip | no | yes | no |
|
||||
| Code | 0 lines (exists) | ~40 lines Rust + 15 Node, no crate | a Prettier-json subset printer (fill rule for scalar arrays, 80 columns), bound forever to a formatter's layout |
|
||||
| Risk | identity re-tied to a serialiser (the Prettier problem moved to Rust) | one more concept; the canonical form must be named (input or validated) | fragile: any Prettier rule the data exercises later changes the value |
|
||||
| Review's row | D-R5 rev 2 | Annex C preference, BDR-045 | rejected rev 1 (in Node form) |
|
||||
|
||||
Recommendation: B over the validated form (what every reader holds, recomputable from the database after validation), the alias mechanism of §B.3.5, the convention named in `sheet-ref.json`, F1 fixed first. The old value `27c7…` stays valid forever as an alias; no archive sees a message for the switch. If you prefer no new concept at all, C is possible on this data (the only Prettier effect measured is the collapse of short scalar arrays), but it keeps the fingerprint chained to a formatter's layout, which Annex C refused.
|
||||
|
||||
## B.4 Business invariants (brief §4) and where they are proven
|
||||
|
||||
| Invariant | Oracle or test |
|
||||
|---|---|
|
||||
| stable ids | P3 (uuid, slug), P1 documents, `legacy-id-map.json` |
|
||||
| zones and anatomical points, main and detail views | P1 documents (annotations, views), P2 annotation rows and records |
|
||||
| annotation ↔ observation relations (`finding_zone_id`, `opens_view_id`) | P1, integrity rules (R2 core, 21 codes) |
|
||||
| vertebral configurations | P1 `vertebral_config`, P2 region inferences |
|
||||
| bilingual data | P1, P3 slug over every `name.fr` / `name.en` |
|
||||
| historical references | `curation/references.json` (authored, never generated), P1 |
|
||||
| existing reports and bilans | P5 (migration map, 612 leaves), the R2 rebase oracles, the R4 PDF fixtures, the alias mechanism |
|
||||
| no sheet validated by a migration | the seed inserts a draft (`insert_sheet_draft`, status draft); publication stays a platform-admin action |
|
||||
|
||||
## B.5 Slices after R5.1 (proposal for the checkpoint; nothing started)
|
||||
|
||||
| Slice | Deliverable | Proof |
|
||||
|---|---|---|
|
||||
| R5.1 | this freeze and study | oracles committed, generator test green, checkpoint report |
|
||||
| R5.1b | F1 fix in `json.rs`; `zod_oracle` compares the recorded JS text | tie unit tests; every R2/R3/R4 suite unchanged (no tie in the draft) |
|
||||
| R5.2 | legacy importer in Rust (`crates/core/src/curation/legacy`): SVG sources through the R3 reader, catalog, regions, ids, anomalies, stats; `import-equine` emits `legacy-import/` | P1 bytes of document.json, legacy-id-map.json, assets.json reproduced exactly; IMPORT-REPORT compared as parsed tables (P2 pre-Prettier digest) or byte-exact if the Prettier markdown effect is nil |
|
||||
| R5.3 | changeset schema, targets, ops, applier, migration map, stats, seed SQL | P1 document.json, sheet-ref.json, report-migration-map.json, equine_draft.sql byte-exact under the decided fingerprint; P2 825 records equal; 612-leaf suite ported |
|
||||
| R5.4 | review overlays and app drawings on the R3 layer and compose; CHANGELOG and REVIEW-REPORT | parsed-tree comparison against the xmldom files (declaration, prefixes, whitespace declared); markdown as parsed tables |
|
||||
| R5.5 | `upload-sheet-assets` binary (reqwest or ureq, service key from the environment), operator runbook | local stack run, SUPABASE.md §3.1 |
|
||||
| R5.6 | D-R5 switch: fingerprint convention, aliases, the one membership test in the gated readers (your go per file), seed regenerated once by the Rust tool, CHANGELOG | old archives open with no message; a report on an older content revision still rebases; N-1 reader green |
|
||||
| R5.7 | consumer census, `packages/anatomy` and `packages/domain` deleted (`packages/domain/tests/fixtures/report-v2.*.json` moved under `crates/core/tests/fixtures` first, `r4.census.test.ts` xmldom assertion inverted), `@xmldom/xmldom` and `prettier` leave the anatomy lockfile, docs (CLAUDE.md map, WORKSPACE, packages/README, RUST.md §11), ledger of the deleted tests | grep census, `pnpm check:all`, test counts reconciled |
|
||||
|
||||
Order constraint: R5.6 needs the owner's D-R5 decision; R5.2 needs R5.1b.
|
||||
|
||||
## B.6 Test ledger
|
||||
|
||||
`crates/core/tests/curation/LEDGER.tsv` at R5.7: one row per TS case of `packages/anatomy/tests` and `packages/domain/tests` at the freeze commit (file, title, disposition ported / replaced / kept / deleted, target). Counts at the freeze: anatomy 131 cases in 16 files (the R5.1 generator adds 6), domain 262 in 13 files.
|
||||
@@ -0,0 +1,108 @@
|
||||
# PLAN: Rust R5.6, the sheet fingerprint switch to `sha256-rfc8785-validated-v1` (prepared, NOT started)
|
||||
|
||||
- date: 2026-10-24 | branch `feature/rust-r5-pipeline-core` | parent plan `2026-10-22-rust-r5-pipeline-core.md` §B.3.5, §B.5 row R5.6
|
||||
- brief: owner 2026-10-24 « Clôture R5.5, corrections ciblées et préparation R5.6 » §6 (convention confirmed, file scope
|
||||
approved: `report/rebase.rs`, `archive/read.rs`, `session.rs`, `pdf/write/input.rs`; switch NOT authorised yet)
|
||||
- decisions on record: BDR-068 (D-R5 = B), owner 2026-10-24 (one alias, `27c7383c…2660`; Report v3 unchanged)
|
||||
|
||||
## A. What changes, what does not
|
||||
|
||||
The value: `sha256(JCS(validated document))`, where the validated document is the R2 typed reading
|
||||
(`AnatomicalSheetDocument` + `validate_sheet_document`, no issue) serialised back, and JCS is RFC 8785. Frozen study value for
|
||||
the current draft: `12d788727c1e532641bd5b3a7ecf39dbdd25e70350f8a76be5ca917b262874e2` (`oracles/r5/fingerprints.json`,
|
||||
`b_rfc8785_validated`). `27c7383c…2660` (sha256 of the Prettier file bytes) becomes the one alias of that content.
|
||||
|
||||
Unchanged: Report v3 shape (`sheet.documentSha256`, 64 hex), the N-1 reader, every PDF fixture, golden and R2-R4 oracle (they
|
||||
keep `27c7…`, valid through the alias), `horse.svg` bytes and asset ids (no re-upload), the document content, the integrity
|
||||
rules, the migration behaviour for a real content change. No view, zone, region, category or reference is renamed.
|
||||
|
||||
## B. Steps (one subtask = one commit, gates at the end)
|
||||
|
||||
1. **Canonicaliser.** `crates/core/src/domain/sheet/fingerprint.rs` (no feature: the core and the tools share it): JCS over
|
||||
`serde_json::Value` (keys sorted by UTF-16 code units, ES number printing through the core's `js_number` (F1-fixed `zmij`),
|
||||
string escaping of RFC 8785 §3.2.2.2, no whitespace), then sha256. Refuses non-finite numbers and lone surrogates (the core's
|
||||
parser already does). Tests: every vector of `oracles/r5/fingerprints.json` `rfc8785_vectors` (RFC appendix inputs) and the
|
||||
44,322-row V8 number oracle `oracles/r5/numbers.json` through the canonical number printer; UTF-16 key order cases
|
||||
(astral vs BMP, `é` vs `é`).
|
||||
2. **Independent differential.** `scripts/r5-fingerprints.mjs` (its own Node JCS, RFC vectors first) is the reference, never
|
||||
the Rust: a new `--json` field per document, then a Rust test reads a frozen output
|
||||
`oracles/r5/fingerprint-differential.json` written once by `MAKE_R5_ORACLES=1` over the draft, the legacy document and
|
||||
N synthetic documents (numbers at the ES formatting edges, Unicode keys, deep nesting). Added, never an existing oracle
|
||||
rewritten.
|
||||
3. **Validation before hashing.** `sheet_fingerprint(document_text) -> Result<Fingerprint, FingerprintError>`: strict parse,
|
||||
R2 shape, `validate_sheet_document` with zero issues, then JCS of the typed reading. A document with an issue has no
|
||||
fingerprint (closed error), so an invalid sheet can never be stamped official.
|
||||
4. **Versioned convention.** `const FINGERPRINT_CONVENTION: &str = "sha256-rfc8785-validated-v1"`; `v1` freezes steps 1 and 3.
|
||||
No default anywhere: a sheet reference without a known convention name is refused at sheet load (`invalid_sheet`), and the
|
||||
pre-switch reference (no convention field) is read as the legacy convention only for the frozen registry of step 5.
|
||||
5. **Trusted alias registry.** A compiled-in table in the core, the only authority:
|
||||
`(sheet_id 503dc301-…, convention "prettier-file-bytes-v0", alias 27c7383c…2660, canonical 12d78872…74e2)`, built from the
|
||||
frozen oracle (P4) and the committed pre-switch `sheet-ref.json`, never from an archive, a PDF, a bundle or the database.
|
||||
`sheet-ref.json` gains an additive `fingerprint` block (`convention`, `aliases: [...]`) for visibility; at sheet load the
|
||||
core checks that every listed alias is in the registry for that `sheet_id` and canonical value, else the sheet is refused.
|
||||
6. **Old archives.** An archive stamped `27c7…` on sheet `503dc301…` is "same document" as the loaded sheet whose canonical
|
||||
value is `12d7…`: no rebase, no message, labels and snapshots kept. The exported envelope is never rewritten.
|
||||
7. **The four gated readers** (owner approved scope; one membership test each, `same_document(report_sha, loaded_ref)`):
|
||||
`report/rebase.rs:40` (skip the rebase on current value or registered alias), `session.rs` (draft restore and import go
|
||||
through the rebase: no own comparison expected, verified), `archive/read.rs:118` (`snapshot_matches`: report vs archived
|
||||
snapshot, both archived; equality kept, alias accepted only when the pair is the registered one), `pdf/write/input.rs:207`
|
||||
(the export's sheet match: a report stamped with an alias exports against the current sheet). Nothing else in them moves.
|
||||
8. **Seed and sheet-ref.** Regenerated once by `import-equine --out` into an empty folder, copied over `draft/` by the owner's
|
||||
go after review of the diff: `sheet-ref.json` (value + block), `report-migration-map.json` (`sheet.documentSha256`, so a
|
||||
migrated v2 report is stamped with the canonical value), nothing else (`equine_draft.sql` carries no fingerprint; checked).
|
||||
`--check` then compares with the regenerated folder; the R5 oracles (TypeScript output) stay as frozen, and the two files'
|
||||
difference from them is a declared rule (fingerprint field and block only), never an oracle regeneration.
|
||||
9. **`.prettierignore`.** `supabase/seed/anatomical-sheets/equine/{legacy-import,draft}/` (the Rust-written paths, narrow),
|
||||
with a reproducible check: `prettier --check` on the repo passes and `git diff` after `pnpm format` is empty on those paths.
|
||||
10. **Non-regression.** Every R2-R4 suite, the 99 PDF fixtures (import, compare), the envelope goldens, the N-1 reader, the
|
||||
612-leaf migration, the app suite on the bundled sheet (wasm rebuilt), `report-compare` on the fixtures, `check:all`.
|
||||
11. **Refusals.** A sheet-ref alias absent from the registry, an alias bound to another `sheet_id`, an alias whose canonical
|
||||
value is not the loaded one, a malformed hex value, an unknown convention: each a closed refusal with a test; a report
|
||||
stamped with an unknown fingerprint rebases as today (never treated as the official sheet).
|
||||
12. **Rollback before merge.** Everything stays on the branch: revert the R5.6 commits (`git revert`, never a reset of pushed
|
||||
history); the seed returns to `27c7…` from git; no database, Storage or asset change exists to undo; an archive exported
|
||||
during the R5.6 tests with `12d7…` would rebase after a rollback (DEV only, labels kept). Rollback trigger: any authentic
|
||||
archive (fixtures, goldens, owner devices) opening with a message or a rebase it did not have before.
|
||||
|
||||
## C. Decisions still needed before the switch
|
||||
|
||||
1. **The fifth comparison.** `crates/core/src/pdf/compare/lines.rs:350` (`report-compare`, the archive check tool) compares
|
||||
fingerprints too and is not in the approved list. Without the membership test, it reports alias-stamped archives as "other
|
||||
sheet". Recommendation: include it (same one-line test).
|
||||
2. **The TypeScript writer.** `pnpm import:equine` and its golden test in `packages/anatomy` would write and expect `27c7…`
|
||||
after the switch. Recommendation: the Rust tool becomes the only writer of the seed at R5.6, `pnpm import:equine` refuses to
|
||||
write (message pointing at `r5:import-equine`) and its golden test is pinned to the pre-switch commit as historical evidence,
|
||||
both deleted at R5.7.
|
||||
3. **Where the alias authority lives.** Recommendation as step 5 (core registry authoritative, `sheet-ref.json` block for
|
||||
visibility, mismatch refused). Alternative: `sheet-ref.json` alone (simpler, but then the bundle is the authority, which the
|
||||
owner excluded).
|
||||
4. **The app side.** `apps/app/src/sheets/provider.ts` reads `ref.document_sha256` only; the additive block needs no TS change.
|
||||
Confirm no TS display of the fingerprint should show the convention.
|
||||
|
||||
## D. Out of scope
|
||||
|
||||
The migration-map cascade debt (§11.12) stays open: no rename in this switch. Step 13 (`anatomical_sheet_versions` storing a
|
||||
fingerprint) is later. No merge, no release, no deploy.
|
||||
|
||||
## E. Owner arbitration 2026-10-24 and preparation state
|
||||
|
||||
Decided: convention confirmed; compiled registry the only authority, bound to sheet + content; five consumers
|
||||
(adds `pdf/compare/lines.rs`); Rust the only generator, `pnpm import:equine` → Rust, TypeScript kept as
|
||||
`import:equine:legacy` until R5.7; no visible change, no migration message. Prepared (RUST.md §11.19):
|
||||
module + tests committed inert; the switch as `.claude/tasks/plans/r5-6-switch.patch` (applies cleanly; on the current seed 4 `curation` tests fail as expected, with the seed switched 3 more: RUST.md §11.19; the earlier "rehearsed green" covered the five consumers only).
|
||||
|
||||
Still to decide at the switch signal:
|
||||
1. `loaded.rs` + `facade.rs` (sheet load refuses a contradicting block): outside the five approved files,
|
||||
not in CLAUDE.md's gated list (bundled sheet, not user data). Needed for "a contradiction is refused".
|
||||
2. The two `curation_draft` cases move to a declared rule for `sheet-ref.json` and `report-migration-map.json`
|
||||
(fingerprint field and block only), the R5.1 oracles untouched.
|
||||
|
||||
3. M1 (security): the export's snapshot reference on an alias match: the report's own stamp (recommended), re-stamp at rebase, or accept with an N-1 test.
|
||||
4. M2 remainder: refuse a `sheet-ref.json` without a fingerprint block after the switch (every test sheet then needs one).
|
||||
|
||||
Switch procedure (on the signal): apply the patch; move the fingerprint after the drawings bound and keep the drawings-bound fixture valid; declared rules for the O7 oracle and the switched-seed preconditions; declared
|
||||
rule in `curation_draft`; `pnpm import:equine` → `scripts/import-equine.sh`, TS as `import:equine:legacy`;
|
||||
`import-equine --out` into an empty folder, diff reviewed (only the two files of §11.19), copied over `draft/`;
|
||||
`--check` 0 differences; `pnpm rust:check`, `pnpm check:all`, security, fresh verifier; owner device check
|
||||
(an old PDF opens with no message). Rollback: `git revert` of the switch commits (seed back to `27c7…`, nothing
|
||||
outside git to undo); an archive exported meanwhile with `12d7…` would then rebase (DEV only).
|
||||
@@ -0,0 +1,43 @@
|
||||
# PLAN: Rust R5.7, consumer census, `packages/anatomy` and `packages/domain` deleted, guarantees kept
|
||||
|
||||
- date: 2026-10-25 | branch `feature/rust-r5-pipeline-core` at 756b7ef (R5.6 active, 9e87016)
|
||||
- spec: owner brief 2026-10-25 « Validation R5.6 et autorisation R5.7 » (§1 to §13); plan R5 §B.5 row R5.7 and §B.6
|
||||
(`.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md`); RUST.md §11; BDR-068, BDR-070
|
||||
- contract: `.claude/tasks/contracts/2026-10-25-rust-r5-7-cleanup.md`
|
||||
- limits: no merge, no 0.2.0, no deploy, no Supabase DEV data, no migration, no SaaS feature, no oracle regenerated,
|
||||
no seed byte changed, no major version bump; stop at the checkpoint (brief §12)
|
||||
|
||||
## Facts at the start (756b7ef)
|
||||
|
||||
- TS cases to retire: anatomy 144 in 22 files, domain 263 in 13 files (vitest JSON, all green).
|
||||
- Consumers outside the two packages: anatomy → root script `import:equine:legacy`, Dockerfile COPY, docs, Rust
|
||||
module comments; domain → anatomy, Dockerfile COPY, the Rust suites reading
|
||||
`packages/domain/tests/fixtures/report-v2.{full,minimal}.json` (`report_rules`, `archive_rules`), the app ESLint ban,
|
||||
`r4.census.test.ts` (xmldom pinned under anatomy), docs, SQL migration comments (frozen, never edited).
|
||||
- `packages/supabase/scripts/upload-sheet-assets.ts` (`assets:upload:legacy`): TS upload tool, replaced by the Rust bin
|
||||
at R5.5 (owner e2e PASS 11/11).
|
||||
- Root `prettier` stays (`pnpm format`, `db:types`, `db:types:check`); `tsx` only under anatomy; `@xmldom/xmldom`
|
||||
only under anatomy.
|
||||
- The `report-v2.*.json` fixtures of domain are byte-identical to the app's copies (sha256 e96b0882…, ba79698c…).
|
||||
- Seed SQL header names `packages/anatomy`: frozen bytes (sha c5a24312…), kept, documented beside.
|
||||
|
||||
## Subtasks
|
||||
|
||||
1. Census + matrices: tests ledger `crates/core/tests/curation/LEDGER.tsv` (407 rows: ported / replaced / kept /
|
||||
deleted, target, oracle, note), files matrix `crates/core/tests/curation/FILES.tsv` (every deleted file → its Rust
|
||||
counterpart or reason). GAPs found by the mapping are ported before any deletion.
|
||||
2. Fixtures: `report-v2.{full,minimal}.json` → `crates/core/tests/fixtures/`, `git mv`, sha256 before = after; Rust
|
||||
readers repointed.
|
||||
3. Ports of uncovered contracts (from the mapping), on frozen oracles, in Rust or durable vitest.
|
||||
4. Deletion: `packages/anatomy`, `packages/domain`, TS upload script + `assets:upload:legacy`,
|
||||
`import:equine:legacy`, Dockerfile COPY lines, lockfile by `pnpm install`; Rust module comments that cite deleted
|
||||
paths point at the reference commit.
|
||||
5. Census tests: `r4.census` xmldom assertion turned into "no importer, no package resolves xmldom", with a non-vacuity
|
||||
check; new R5 census (packages gone, no production path imports a TS engine, ledger and files matrix consistent with
|
||||
the tree); app ESLint ban extended to `@faunosteo/anatomy` and xmldom.
|
||||
6. Historical reproducibility: reference commits, versions, commands, isolated-worktree procedure, expected sha256 of the
|
||||
18 files; rehearsed once in a detached worktree.
|
||||
7. Docs: CLAUDE.md, WORKSPACE.md, packages/README.md, README.md, RUST.md §11.23, CHANGELOG, ANATOMICAL_SHEET_MODEL.md,
|
||||
SUPABASE.md, TEST_STRATEGY.md, PDF-FORMAT.md (xmldom line stale), ANNOTATION-RENDERING.md; follow-ups kept (brief §10).
|
||||
8. Checks: frozen-lockfile install, rust:check, check:all, `import:equine --check`, targeted suites, census, counts.
|
||||
9. Gates: fresh security audit, fresh verifier on the contract; memory; checkpoint, STOP.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -7,3 +7,6 @@ packages/core-wasm/types/
|
||||
packages/core-wasm/pkg/
|
||||
packages/core-wasm/pkg-proofs/
|
||||
packages/core-wasm/tests/n-1/
|
||||
# Written by the Rust pipeline (`pnpm r5:import-equine`, R5.6): byte-exact, never reformatted.
|
||||
supabase/seed/anatomical-sheets/equine/legacy-import/
|
||||
supabase/seed/anatomical-sheets/equine/draft/
|
||||
@@ -6,6 +6,37 @@ Versions follow [Semantic Versioning](https://semver.org/).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Removed (2026-10-25, Rust R5.7: the TypeScript pipeline and domain packages)
|
||||
|
||||
- `packages/anatomy` (legacy importer, curation pipeline, TypeScript renderer, `import-equine` CLI) and `packages/domain` (Zod schemas and rules) are deleted, with the TypeScript upload script `packages/supabase/scripts/upload-sheet-assets.ts` and the scripts `pnpm import:equine:legacy` and `assets:upload:legacy`. The Rust core does all of it: `pnpm import:equine` and `pnpm assets:upload` are unchanged for the user. `@xmldom/xmldom` leaves the lockfile; `tsx` is no longer a dependency of any package (pnpm keeps it only as an optional peer of vite); root Prettier stays (`pnpm format`, `db:types`).
|
||||
- Every one of the 407 TypeScript test cases is dispositioned in `crates/core/tests/curation/LEDGER.tsv` (ported, replaced by a frozen oracle, or deleted with its reason, none uncovered) and every deleted file in `crates/core/tests/curation/FILES.tsv`; the contracts the TypeScript alone still checked are ported to Rust (the 612 legacy leaves and 26 legacy annotations, region inference, the R5.1 artifact ledger, the generated folders' file list, the migration-map oracle entry by entry, review overlay removals, the halo filter, reserved and dollar-tag refusals). The `report-v2.*.json` fixtures moved to `crates/core/tests/fixtures/` with their bytes unchanged. No oracle, golden, digest or seed byte changed. The historical TypeScript pipeline stays replayable from commit 59521dc (official writer) or 756b7ef (last revision holding it) in a detached worktree (`docs/development/RUST.md` §11.23).
|
||||
|
||||
### Changed (2026-10-25, Rust R5.6: the sheet fingerprint convention `sha256-rfc8785-validated-v1`)
|
||||
|
||||
- The equine sheet's fingerprint is now the SHA-256 of the RFC 8785 canonical form of the validated document: `12d78872…74e2` in `draft/sheet-ref.json` (with a `fingerprint` block naming the convention) and in `draft/report-migration-map.json`, the only two seed files changed; `document.json`, the drawing and the SQL seed are byte-identical. The previous value `27c7383c…2660` stays recognised for this sheet only, through the core's compiled registry and a fingerprint recomputed from the loaded document: old reports and PDFs open with no message and no rebase, and a re-export keeps their stamp so older readers keep the archive. A report written after the switch, opened by a build from before it, is rebased with nothing lost. `pnpm import:equine` runs the Rust pipeline; the TypeScript generator stays as `pnpm import:equine:legacy` until R5.7 (`docs/development/RUST.md` §11.22).
|
||||
|
||||
### Changed (2026-10-23, Rust R5.5: the sheet-asset upload tool in Rust)
|
||||
|
||||
- `pnpm assets:upload [--check] <seed folder>` runs the Rust binary `upload-sheet-assets` (`crates/core`, feature `upload`, `ureq`) in place of the TypeScript script (kept as `assets:upload:legacy` until R5.7). Local stack only: no default destination, `http` with an explicit port on loopback or the configured DEV address, a private-bucket handshake before any transfer; every file checked against `assets.json` and the SVG reader before the first request; an existing object is hashed and never overwritten, a missing or foreign asset row stops the run, a row is confirmed only after its object is read back; bounded retries, resumable; the service key is never printed. Runbook `docs/development/SUPABASE.md` §3.1, `pnpm e2e:sheet-assets` for the local stack, `docs/development/RUST.md` §11.16.
|
||||
- `changes.json` and `references.json` are refused above 4 MiB with an explicit message (core and binary). The serde tolerance of the R2 contracts is characterised (test `characterisation_serde_tolerance`), not changed.
|
||||
|
||||
### Changed (2026-10-23, Rust R5.4: the review files in Rust)
|
||||
|
||||
- The review drawings (`review/annotations-*.svg`, `review/app-*.svg`), `CHANGELOG.md` and `REVIEW-REPORT.md` of the draft are produced by the Rust pipeline: the two reports and three drawings byte for byte with the committed files; `review/app-main.svg` carries the head trigger's English name ("Head (left lateral view)"), the name the editor canvas shows since R3, where the committed file repeats the French one. `pnpm r5:import-equine --check` now compares the whole draft (12 files) and fails on any file missing or different without a declared rule; `--out` writes the 18 files. The seed, `horse.svg`, `pnpm import:equine` and `documentSha256` are unchanged (`docs/development/RUST.md` §11.14).
|
||||
|
||||
### Changed (2026-10-22, Rust R5.3: the curated draft in Rust)
|
||||
|
||||
- The draft side of the anatomical pipeline runs in Rust behind the core's `curation` feature (`crates/core/src/curation/draft`): the authored changeset and references files parsed strictly, the six ops of a change applied on the plain JSON document with their records, the review rule of levels B and C, the shape and integrity gate of the R2 core, the report migration map, the document statistics and the seed SQL. `pnpm r5:import-equine --check` reproduces `legacy-import/` and the R5.3 files of `draft/` (`document.json`, `assets.json`, `sheet-ref.json`, `report-migration-map.json`, `equine_draft.sql`, the frozen `assets/horse.svg`) byte for byte, 12 files, 0 differences; `pnpm r5:import-equine --out <dir>` writes both folders into an empty folder and never deletes anything. The review drawings, `CHANGELOG.md` and `REVIEW-REPORT.md` of the draft stay with the TypeScript until R5.4; `pnpm import:equine` and the seed are unchanged; `documentSha256` stays `27c7383c…2660` until R5.6 (`docs/development/RUST.md` §11.12).
|
||||
|
||||
### Changed (2026-10-22, Rust R5.1b and R5.2)
|
||||
|
||||
- The core prints an exact decimal tie as JavaScript does (`840.1692504882812`, round half to even), its digits now from `zmij`, the formatter serde_json already used; no historical output carried such a value, so no oracle, golden or PDF digest changed (`docs/development/RUST.md` §11.9).
|
||||
- The legacy importer runs in Rust behind the core's `curation` feature: `import-equine --check` reproduces the committed `legacy-import/` folder byte for byte, and `--out <dir>` writes it into an empty folder. The TypeScript pipeline, `pnpm import:equine` and the seed are unchanged; `documentSha256` stays as it is until R5.6 (§11.10).
|
||||
|
||||
### Changed (2026-10-22, Rust R5.1: the TypeScript pipeline frozen, not yet ported)
|
||||
|
||||
- `packages/anatomy` and `packages/domain` are unchanged; their outputs are recorded as oracles under `crates/core/tests/oracles/r5/` (intermediate values, ids and slugs, the fingerprint of every generated JSON file under today's rule and the candidate rules of D-R5), generated by `packages/anatomy/tests/r5.pipeline-oracles.test.ts` with `MAKE_R5_ORACLES=1`. Study scripts `scripts/r5-fingerprints.mjs` and `packages/supabase/scripts/r5-jsonb-fingerprint.mjs` (read-only). `documentSha256` and the generated seed are untouched; the D-R5 decision is the owner's (`docs/development/RUST.md` §11). Known defect recorded, not fixed: the core's JavaScript number printing rounds an exact decimal tie upward where JavaScript rounds to even (no value of the curated draft is affected).
|
||||
|
||||
### Fixed (2026-10-21, R4 close)
|
||||
|
||||
- Sign-in redirects: a `next` value whose dot segments normalise into `//host` (`/..//host`, `/%2e%2e//host`) now falls back to the dashboard instead of being returned as a protocol-relative path. The worker image runs on `distroless/static-debian13:nonroot` (the Debian 12 tags are no longer updated). DEPLOY.md shows `http2 on;` for nginx 1.25.1 and later and keeps the `listen ... http2` form for nginx 1.24 (`docs/development/RUST.md` §10.14).
|
||||
|
||||
@@ -15,8 +15,8 @@ writes conclusion, and exports an A4 PDF. PDF carries the full JSON state as a n
|
||||
attachment so importing it restores the session for a follow-up.
|
||||
SPA deployed via Docker + nginx on Scaleway VPS. Backend foundation on Supabase
|
||||
(PostgreSQL + Auth + RLS) since Step 3 of the SaaS plan: profiles, platform roles and
|
||||
audit log exist; Step 4 added the canonical anatomical sheet model (`packages/domain`,
|
||||
tables `species`, `anatomical_sheets` / `_versions` / `_assets`). Step 5 moved the
|
||||
audit log exist; Step 4 added the canonical anatomical sheet model (since R2 in
|
||||
`crates/core/src/domain/sheet`; the TypeScript `packages/domain` left at R5.7), tables `species`, `anatomical_sheets` / `_versions` / `_assets`). Step 5 moved the
|
||||
application onto that model: the editor renders the bundled reference sheet (the
|
||||
curated equine DRAFT, `SHEET_BUNDLE=draft`), writes Report v3 and imports v2 PDFs
|
||||
through a generated migration map. Step 6 made the PDF an autonomous archive: the
|
||||
@@ -40,8 +40,11 @@ Rust R2 moved the application's domain rules into `crates/core` (sheet, report a
|
||||
archive contracts, integrity rules, sheet index, migration, rebase, snapshot, layered
|
||||
archive reading, editor reducers, findings and vertebra rules, report operations),
|
||||
reached through the wasm facade and `packages/core-wasm`, loaded lazily by the sheet
|
||||
pages only (`docs/development/RUST.md` §8); `packages/domain` is frozen for the PDF
|
||||
engine, the pipeline and the compare CLI.
|
||||
pages only (`docs/development/RUST.md` §8). R5 moved the anatomical pipeline (legacy
|
||||
importer, curation, migration map, review files, seed SQL, upload tool) into the core
|
||||
(feature `curation`); R5.6 switched the sheet fingerprint to `sha256-rfc8785-validated-v1`;
|
||||
R5.7 deleted `packages/anatomy` and `packages/domain` (`RUST.md` §11.23, test ledger
|
||||
`crates/core/tests/curation/LEDGER.tsv`).
|
||||
MVP species: equine only. Open-source MIT.
|
||||
|
||||
---
|
||||
@@ -74,8 +77,8 @@ MVP species: equine only. Open-source MIT.
|
||||
|
||||
## Build commands
|
||||
|
||||
pnpm workspace: `apps/app` = the private application, `packages/domain` = pure
|
||||
Zod domain model (anatomical sheet document), `packages/supabase` = Supabase
|
||||
pnpm workspace: `apps/app` = the private application, `packages/core-wasm` = the
|
||||
Rust core in the browser (runtime, generated types, frozen N-1 reader), `packages/supabase` = Supabase
|
||||
access layer (see `docs/development/WORKSPACE.md`). Every command
|
||||
below runs from the repository root and delegates to the packages.
|
||||
|
||||
@@ -118,7 +121,10 @@ pnpm db:types:check # CI diff
|
||||
# Reference sheet in the app: the dev server bundles the curated draft; a build does not
|
||||
pnpm build:preview # SHEET_BUNDLE=draft build (manual validation, preview deployments)
|
||||
pnpm preview:draft # same, then serves it
|
||||
pnpm import:equine # regenerate legacy-import/ and draft/ (document, migration map, review drawings)
|
||||
pnpm import:equine --check | --out <dir> # since R5.6 the official seed writer: the Rust pipeline (same as `pnpm r5:import-equine`), fingerprint `sha256-rfc8785-validated-v1`; the TypeScript generator left at R5.7 (replayed from a historical revision only, RUST.md §11.23)
|
||||
pnpm r5:import-equine --check | --out <dir> # the Rust pipeline (R5.2 importer, R5.3 draft, R5.4 review files): compare the 18 generated files with the committed folders (bytes; app drawings under the declared R3 rule), or write <dir>/{legacy-import,draft} into an empty folder; never deletes (scripts/import-equine.sh, first run compiles)
|
||||
pnpm assets:upload [--check] <seed folder> # R5.5: uploads the sheet drawings of a generated seed folder to the LOCAL stack's private bucket (Rust `upload-sheet-assets`, feature `upload`; SUPABASE_URL local with a port, SUPABASE_SERVICE_ROLE_KEY typed, never printed; no overwrite, confirm after verify; run by hand, scripts/upload-sheet-assets.sh)
|
||||
pnpm e2e:sheet-assets # scripts/sheet-assets-e2e.sh: that tool against the local stack (upload, rerun, anon and wrong key, refused destinations); run by hand
|
||||
pnpm report:compare <pdf|dir>... # archive check: attachment vs its v3 reading, unresolved items flagged (docs/PDF-FORMAT.md); the core's `report-compare` binary since R4.7 (scripts/report-compare.sh, first run compiles)
|
||||
|
||||
# Rust (docs/development/RUST.md) — native toolchain (user-level rustup, channel of rust-toolchain.toml) first
|
||||
@@ -169,7 +175,7 @@ pnpm test # vitest watch — dev loop
|
||||
|
||||
Test files live in `apps/app/tests/unit/*.test.ts(x)`; golden fixtures in
|
||||
`apps/app/tests/fixtures/`. The 612 legacy catalog leaves run through the v2 → v3 migration
|
||||
in both `packages/anatomy` (against the draft document) and the app (sheet index + PDF roundtrip). Setup: `apps/app/tests/setup.ts` imports
|
||||
in the core (`crates/core/tests/curation_leaves.rs`, against the draft document) and the app (sheet index + PDF roundtrip). Setup: `apps/app/tests/setup.ts` imports
|
||||
@testing-library/jest-dom. Suites run on the bundled reference sheet (`tests/helpers/sheet.ts`); frozen v2 fixtures under `tests/fixtures/`.
|
||||
SQL: `supabase/tests/*.sql` (pgTAP) run by `packages/supabase/tests/db/pgtap.test.ts`
|
||||
on PGlite (part of `pnpm test:run`) and by `pnpm db:test` on the real stack.
|
||||
@@ -256,13 +262,7 @@ faunosteo/
|
||||
├─ eslint.config.js, index.html, package.json (@faunosteo/app)
|
||||
├─ tsconfig*.json, vite.config.ts, vitest.config.ts
|
||||
│ (end of apps/app)
|
||||
├─ packages/domain/ ← @faunosteo/domain (pure, no DOM, zod only): anatomical-sheet (document, integrity rules, sheet-index), report (v3, v2 migration, rebase), archive (sheet snapshot, envelope v1, branding, labels, layered reader)
|
||||
│ ├─ src/anatomical-sheet/{schema,validate,create}.ts ← canonical sheet document + integrity rules
|
||||
│ └─ tests/{fixtures/horse-slice.ts,*.test.ts}
|
||||
├─ packages/anatomy/ ← @faunosteo/anatomy: legacy importer, curation pipeline, frozen TS annotation-layer renderer + view composer (`./render`: the pipeline until R5; no app and no PDF consumer since R4.8)
|
||||
│ ├─ src/legacy/{svg,catalog,regions,ids,report,importer}.ts, src/cli/import-equine.ts
|
||||
│ ├─ src/curation/{schema,targets,fields,ops,rename,apply,types,assets,labels,stats,changelog,review-report,overlay,seed,sources,pipeline}.ts ← changeset → draft
|
||||
│ └─ tests/ ← counts preserved, special cases, determinism, curation ops, golden artifacts
|
||||
│ (packages/domain and packages/anatomy deleted at R5.7: their sources resolve at 756b7ef, crates/core/tests/curation/{FILES,LEDGER}.tsv)
|
||||
├─ packages/supabase/ ← @faunosteo/supabase
|
||||
│ ├─ src/{index,client,database.types,anatomical-sheets,assets}.ts ← typed client, generated types, publish gate, avatar flow (closed AssetFailure kinds)
|
||||
│ ├─ tests/e2e/avatar-flow.e2e.ts + vitest.e2e.config.ts ← real-stack end-to-end (`pnpm e2e:avatar`), skipped without the stack env
|
||||
@@ -279,10 +279,10 @@ faunosteo/
|
||||
├─ packages/README.md ← shared packages index
|
||||
├─ packages/core-wasm/ ← @faunosteo/core-wasm: CoreRuntime (lazy, poison, recovery) + CoreExportSession (R4.6) + Node source, pkg/ shipped and pkg-proofs/ (built, git-ignored, BUILD_STAMP), ts-rs types (committed), frozen N-1 reader (tests/n-1), tests
|
||||
├─ crates/ ← Rust workspace (Cargo.toml, rust-toolchain.toml, deny.toml, clippy.toml, .cargo/config.toml at the root)
|
||||
│ ├─ core/ ← faunosteo-core: domain/{js,sheet,report,archive,editor,session} (contracts and rules), anatomy/{svg/{prescan,tree,sanitize,refs,number,serialize},layer,compose} (R3: the one SVG reader, the layer, canvas + highlight composition), pdf/{read,json_js,import,write,compare} (R4: reader, writer and session, the archive check tool behind feature `cli`), ops.rs + facade.rs (`call`, `call_with_sheet`), json (JS number printing), bin/report_compare.rs (`pnpm report:compare`, the one file allowed std::fs); tests/{oracles,conformance,fixtures,pdf/LEDGER.tsv} frozen from the TypeScript (oracles/r3, oracles/r4, fixtures/svg/hostile); no I/O, no clock (lint-enforced)
|
||||
│ ├─ core/ ← faunosteo-core: domain/{js,sheet,report,archive,editor,session} (contracts and rules), anatomy/{svg/{prescan,tree,sanitize,refs,number,serialize},layer,compose} (R3: the one SVG reader, the layer, canvas + highlight composition), pdf/{read,json_js,import,write,compare} (R4: reader, writer and session, the archive check tool behind feature `cli`), ops.rs + facade.rs (`call`, `call_with_sheet`), json (JS number printing), curation/{text,ids,prettier_json,markdown,report_guard,legacy/*,draft/*} (R5, feature `curation`: Prettier layout, the legacy importer, the changeset applier, migration map, stats, seed SQL, CHANGELOG, REVIEW-REPORT, review drawings), curation/upload.rs (R5.5, feature `upload`: the upload rules, pure) + bin/report_compare.rs (`pnpm report:compare`) + bin/import_equine.rs (`pnpm r5:import-equine`) + bin/upload_sheet_assets/ (`pnpm assets:upload`, ureq; the three tools allowed std::fs); tests/{oracles,conformance,fixtures,pdf/LEDGER.tsv} frozen from the TypeScript (oracles/r3, oracles/r4, fixtures/svg/hostile); no I/O, no clock (lint-enforced)
|
||||
│ ├─ wasm/ ← faunosteo-wasm: wasm-bindgen facade (core_version, call, SheetHandle; trap exports in the proofs build only)
|
||||
│ └─ worker/ ← faunosteo-worker: plain main; `decode` sandboxed child (rlimits, seccomp allowlist, image → libwebp), parent Spawner + RIFF check; R1 `run` loop: config, log (JSON lines, allowlist), keys, backend trait + http (ureq, PostgREST rpc + Storage objects), job, sweep, watchdog; Dockerfile, scripts/derive-seccomp.sh
|
||||
├─ scripts/{rust-env,rust-check,wasm-build,check-all,avatar-e2e,report-compare}.sh, wasm-stamp.mjs ← native Rust gate, wasm build (pkg + pkg-proofs + stamp), the pre-merge gate (`pnpm check:all`), the real-stack avatar e2e, the archive check tool (`pnpm report:compare`)
|
||||
├─ scripts/{rust-env,rust-check,wasm-build,check-all,avatar-e2e,report-compare,import-equine,upload-sheet-assets,sheet-assets-e2e,floor-clarification-check}.sh, wasm-stamp.mjs ← native Rust gate, wasm build (pkg + pkg-proofs + stamp), the pre-merge gate (`pnpm check:all`), the real-stack avatar e2e, the archive check tool (`pnpm report:compare`), the Rust pipeline (`pnpm r5:import-equine`)
|
||||
├─ scripts/{dev-env,supabase,dev}.sh ← the DEV origin (LAN address) in one place; CLI wrapper; `pnpm dev` on that origin
|
||||
├─ docker-compose.yml ← `app` service; `worker` service under the `worker` profile (read-only, cap_drop ALL, external Supabase network, `.env.worker`)
|
||||
├─ docker/rust-builder.Dockerfile, rust-run.sh, csp-proof/ ← reproducible Rust builder (alternative) + the CSP proofs (R0 module, R2 app image: app-image.sh)
|
||||
@@ -406,8 +406,8 @@ Database (Supabase)
|
||||
|
||||
### Annotation rendering
|
||||
- docs/ANNOTATION-RENDERING.md is the source of truth: the drawing carries no annotation element, the markers are
|
||||
drawn from the document (Rust core for the canvas since R3, `crates/core/src/anatomy`; the frozen TS renderer
|
||||
`@faunosteo/anatomy/render` for the pipeline until R5, same output), with the attributes listed there
|
||||
drawn from the document (the Rust core, `crates/core/src/anatomy`, for the canvas since R3, the PDF since R4 and
|
||||
the pipeline's review drawings since R5), with the attributes listed there
|
||||
- Every SVG the product draws goes through the core's one reader (`anatomy::svg::read_svg`): caps before any parse,
|
||||
allowlist, reference graph; a product drawing with anything removed is refused at load, never repaired
|
||||
- Never rely on element id, className or a label for click targeting or behaviour — always `data-anno-id`
|
||||
@@ -418,8 +418,8 @@ Database (Supabase)
|
||||
### Schemas
|
||||
- The data contracts live in Rust since R2 (`crates/core/src/domain/{sheet,report,archive}`, Zod semantics
|
||||
reproduced: strict where Zod was strict, optional absent, nullable required, Zod schema field order);
|
||||
`packages/domain` is frozen (no new logic) for the pipeline until R5 (the app no longer depends on it: R4.8),
|
||||
and its Zod copy under `packages/core-wasm/tests/n-1` is the permanent N-1 reader. Never change a contract
|
||||
`packages/domain` was deleted at R5.7; its byte-identical Zod copy under `packages/core-wasm/tests/n-1` is the
|
||||
permanent N-1 reader. Never change a contract
|
||||
without bumping the version on a breaking change (additive optional fields keep it); a change touches the
|
||||
Rust struct, the regenerated ts-rs types and the N-1 test
|
||||
- Every report entering the store comes from the core (new report, draft restore, import, reducer outputs);
|
||||
@@ -491,16 +491,16 @@ Database (Supabase)
|
||||
- Avatar UI follows the validated mockup: photo in Mon compte (Informations personnelles), sidebar, drawer and top bar
|
||||
link to Mon compte; JPEG / PNG / WebP only, never SVG; no signed URL for the owner-only avatar (download under RLS).
|
||||
|
||||
### Anatomical sheet model (packages/domain + migration 20261008110000)
|
||||
### Anatomical sheet model (crates/core/src/domain/sheet + migration 20261008110000)
|
||||
- `docs/development/ANATOMICAL_SHEET_MODEL.md` is the source of truth: document shape, issue codes, tables, RPCs.
|
||||
- Ids are stable slugs never derived from labels; labels are `{ fr, en }` objects. Annotation → findings is the
|
||||
explicit `finding_zone_id`; vertebral regions are ordered data with free codes. No label matching at runtime.
|
||||
- A published version is immutable (trigger, every role). Change = new draft (`create_sheet_draft`) then
|
||||
`publish_sheet_version`. Integrity rule codes are shared by `validateSheetDocument` (TS) and
|
||||
`publish_sheet_version`. Integrity rule codes are shared by `validate_sheet_document` (Rust core) and
|
||||
`private.sheet_document_issues` (SQL); add a rule on both sides and in the spec table.
|
||||
- Document schema change → bump `SHEET_DOCUMENT_SCHEMA_VERSION`, extend the SQL CHECK on `document_schema_version`.
|
||||
- Official equine sheet = generated seed `draft/` (`pnpm import:equine`, never edit `legacy-import/` or `draft/` by hand;
|
||||
the golden test fails on drift). Every content change goes through `curation/changes.json` (traced in CHANGELOG.md).
|
||||
- Official equine sheet = generated seed `draft/` (`pnpm import:equine --out` into an empty folder, then reviewed and copied; never edit `legacy-import/` or `draft/` by hand;
|
||||
`curation_draft` / `curation_review` fail on drift, `pnpm import:equine --check` lists it). Every content change goes through `curation/changes.json` (traced in CHANGELOG.md).
|
||||
It is a DRAFT until a platform admin publishes it after business review of `REVIEW-REPORT.md`.
|
||||
- Sheet SVG bytes live in Storage bucket `anatomical-sheet-assets` as `<sha256>.svg`; uploads are an operator task
|
||||
(`pnpm assets:upload <seed dir>` with the service key in env), never from the browser.
|
||||
@@ -526,7 +526,9 @@ Database (Supabase)
|
||||
Study and roadmap: `docs/development/RUST_ARCHITECTURE_REVIEW.md` (validated with
|
||||
the amendments of Annex C). Done: R0, R1 (worker), R2 (domain core used by the app,
|
||||
`docs/development/RUST.md` §8), R3 (anatomy core: SVG reader, layer, composition; editor canvas on it,
|
||||
RUST.md §9; validated and merged 2026-10-17). Next: R4 (PDF core), R5 (pipeline); release 0.2.0 unaffected.
|
||||
RUST.md §9; validated and merged 2026-10-17), R4 (PDF core, RUST.md §10; merged 2026-10-21). R5 (pipeline,
|
||||
RUST.md §11: R5.1 to R5.7, the TypeScript packages deleted; merged 2026-10-25, not a production authorisation);
|
||||
release 0.2.0 unaffected.
|
||||
|
||||
- Any logic that can run in Rust with no significant loss (features, UX,
|
||||
graphics, data compatibility, security, reliability, useful performance,
|
||||
|
||||
Generated
+3
@@ -364,6 +364,7 @@ checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
|
||||
name = "faunosteo-core"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"icu_normalizer",
|
||||
"lopdf",
|
||||
"miniz_oxide 0.9.1",
|
||||
"roxmltree",
|
||||
@@ -371,7 +372,9 @@ dependencies = [
|
||||
"serde_json",
|
||||
"sha2",
|
||||
"ts-rs",
|
||||
"ureq",
|
||||
"url",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
@@ -42,6 +42,12 @@ libc = "0.2"
|
||||
# Supabase over HTTPS) and the hash of the clean output. No async runtime: the loop is sequential.
|
||||
ureq = { version = "3.4.2", default-features = false, features = ["rustls", "json"] }
|
||||
sha2 = "0.11"
|
||||
# Shortest round-trip float digits with ECMAScript's tie rule (round half to even), the
|
||||
# formatter serde_json already uses (R5.1b, F1): the version serde_json locks.
|
||||
zmij = "1.0.23"
|
||||
# NFD for the pipeline's slugs and region roots (R5.2): the normaliser `url` already locks
|
||||
# through idna, its compiled data only.
|
||||
icu_normalizer = { version = "2.3.0", default-features = false, features = ["compiled_data"] }
|
||||
|
||||
# The library crates deny unwrap/expect so that no panic path depends on content;
|
||||
# the worker binary relaxes disallowed_methods (it owns I/O) in its own [lints].
|
||||
|
||||
@@ -36,8 +36,6 @@ RUN corepack enable && corepack prepare pnpm@10.13.1 --activate
|
||||
# Workspace manifests first for layer caching (install depends only on these)
|
||||
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml .npmrc ./
|
||||
COPY apps/app/package.json apps/app/
|
||||
COPY packages/domain/package.json packages/domain/
|
||||
COPY packages/anatomy/package.json packages/anatomy/
|
||||
COPY packages/supabase/package.json packages/supabase/
|
||||
COPY packages/core-wasm/package.json packages/core-wasm/
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ full report JSON embedded as a native PDF attachment. Re-importing that PDF
|
||||
restores the state for a follow-up.
|
||||
|
||||
Multi-species by design. The MVP ships the **horse** sheet: a canonical anatomical
|
||||
sheet document (drawing + annotations + findings catalog, `packages/domain`)
|
||||
sheet document (drawing + annotations + findings catalog, `crates/core/src/domain/sheet`)
|
||||
generated and curated under `supabase/seed/anatomical-sheets/equine/`. Another
|
||||
species is another sheet document, no code change.
|
||||
See `docs/development/ANATOMICAL_SHEET_MODEL.md` §6 for how a sheet is produced.
|
||||
@@ -93,7 +93,7 @@ docker compose up --build
|
||||
```bash
|
||||
pnpm typecheck # TypeScript strict, no emit
|
||||
pnpm lint # ESLint 9 flat config, 0 warnings
|
||||
pnpm test:run # vitest run (domain, anatomy, supabase/PGlite, app)
|
||||
pnpm test:run # vitest run (core-wasm, supabase/PGlite, app)
|
||||
pnpm build # tsc + vite build → apps/app/dist/
|
||||
pnpm report:compare <pdf|dir>... # archive PDF vs its v3 reading, unresolved items flagged (the core's binary; first run compiles)
|
||||
pnpm rust:check # native: fmt, clippy -D warnings, cargo test, ts-rs export, cargo deny
|
||||
@@ -152,8 +152,8 @@ faunosteo/ pnpm workspace (see docs/development/WORKSPACE.md)
|
||||
├─ src/lib/ id, dates, assert utilities
|
||||
├─ src/styles/ Tailwind v4 CSS + design tokens
|
||||
├─ tests/unit/ vitest tests on the bundled sheet (+ frozen v2 fixtures)
|
||||
├─ packages/domain/ @faunosteo/domain: canonical sheet document + report document (v3, v2 migration)
|
||||
├─ packages/anatomy/ @faunosteo/anatomy: legacy importer, curation pipeline, annotation-layer renderer + view composer
|
||||
├─ packages/core-wasm/ @faunosteo/core-wasm: the Rust core in the browser (runtime, generated types, frozen N-1 reader)
|
||||
├─ crates/ Rust: core (contracts, rules, SVG, PDF, anatomical pipeline), wasm facade, worker
|
||||
├─ packages/supabase/ @faunosteo/supabase: typed client, generated types, PGlite SQL test harness
|
||||
├─ supabase/ CLI config, SQL migrations, local seed (incl. the equine sheet), pgTAP tests
|
||||
├─ docker/ entrypoint.sh + nginx.conf
|
||||
@@ -169,7 +169,7 @@ v0.1.0 MVP — implementation complete.
|
||||
|
||||
- `pnpm typecheck` ✅
|
||||
- `pnpm lint` ✅
|
||||
- `pnpm test:run` ✅ — 367 tests across the workspace (domain, anatomy, supabase, app)
|
||||
- `pnpm test:run` ✅ — the vitest suites of the workspace (core-wasm, supabase, app); `pnpm rust:check` runs the Rust ones
|
||||
- `pnpm build` ✅
|
||||
|
||||
The first species ships as the equine anatomical sheet: the historical
|
||||
|
||||
@@ -55,8 +55,8 @@ export default tseslint.config(
|
||||
},
|
||||
{
|
||||
// R2: the application's rules run in the Rust core; its contracts come from
|
||||
// @faunosteo/core-wasm. The TypeScript domain is no longer a dependency of the
|
||||
// application (R4.8); it stays frozen for the pipeline until R5
|
||||
// @faunosteo/core-wasm. The TypeScript domain left the application at R4.8 and
|
||||
// the repository at R5.7 with the pipeline package and its xmldom
|
||||
// (RUST_ARCHITECTURE_REVIEW VI.4). The rule keeps the door closed.
|
||||
files: ['src/**/*.{ts,tsx}'],
|
||||
rules: {
|
||||
@@ -68,8 +68,16 @@ export default tseslint.config(
|
||||
name: '@faunosteo/domain',
|
||||
message: 'Use @faunosteo/core-wasm (types) and the core operations.',
|
||||
},
|
||||
{
|
||||
name: '@faunosteo/anatomy',
|
||||
message: 'The SVG layer and composition are the core operations.',
|
||||
},
|
||||
{
|
||||
name: '@xmldom/xmldom',
|
||||
message: 'Every SVG goes through the core reader (anatomy::svg).',
|
||||
},
|
||||
],
|
||||
patterns: ['@faunosteo/domain/*'],
|
||||
patterns: ['@faunosteo/domain/*', '@faunosteo/anatomy/*'],
|
||||
},
|
||||
],
|
||||
},
|
||||
|
||||
@@ -43,15 +43,34 @@ function buildV2Unresolved(sheet: SheetIndex) {
|
||||
).read
|
||||
}
|
||||
|
||||
/**
|
||||
* The goldens were frozen before R5.6, when a v2 migration stamped the historical fingerprint
|
||||
* (the pre-switch migration map). Since R5.6 the map stamps the canonical `v1` value and
|
||||
* nothing else of the migrated report changes (Rust `pdf_import`
|
||||
* `o7_after_r5_6_migrated_v2_reports_carry_the_canonical_stamp_and_nothing_else_moves`): the
|
||||
* golden v2 reports are taken in that original context, as reports migrated before the switch
|
||||
* and exported now. Their export keeps the stamp in the snapshot (M1), so the frozen bytes hold.
|
||||
*/
|
||||
export const HISTORICAL_FINGERPRINT =
|
||||
'27c7383c445c73dc2001ebf4794a3ea63d416910bbbaa0d7848dd720de7c2660'
|
||||
|
||||
export function migratedBeforeR56(report: Report): Report {
|
||||
return {
|
||||
...report,
|
||||
sheet: { ...report.sheet, documentSha256: HISTORICAL_FINGERPRINT },
|
||||
}
|
||||
}
|
||||
|
||||
export function buildGoldenReports(sheet: SheetIndex): GoldenReports {
|
||||
const v2unresolved = buildV2Unresolved(sheet)
|
||||
return {
|
||||
reports: {
|
||||
v2full: importAttachment(readJson('report-v2.full.json'), sheet).read
|
||||
.report,
|
||||
v2full: migratedBeforeR56(
|
||||
importAttachment(readJson('report-v2.full.json'), sheet).read.report,
|
||||
),
|
||||
demo: importAttachment(readJson('report-v3.demo.json'), sheet).read
|
||||
.report,
|
||||
v2unresolved: v2unresolved.report,
|
||||
v2unresolved: migratedBeforeR56(v2unresolved.report),
|
||||
},
|
||||
unresolvedOfV2Unresolved: v2unresolved.unresolvedFindings,
|
||||
}
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
* R4.8 census (plan §B.7 R4.8, RUST.md §10.10): the PDF path runs on the core and nothing of
|
||||
* the TypeScript engine is left where the application or its glue could reach it. Grep facts
|
||||
* over the repository, so that a reintroduced import, a dependency added back or a script
|
||||
* repointed fails here before anything else.
|
||||
* repointed fails here before anything else. Since R5.7 xmldom, the TypeScript domain and the
|
||||
* pipeline package are gone too (`r5.census.test.ts` holds the rest of that census).
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs'
|
||||
@@ -43,6 +44,7 @@ describe('the TypeScript PDF engine is gone', () => {
|
||||
'@faunosteo/pdf',
|
||||
'pdf-lib',
|
||||
'@faunosteo/domain',
|
||||
'@faunosteo/anatomy',
|
||||
'@xmldom/xmldom',
|
||||
]
|
||||
for (const file of [
|
||||
@@ -70,6 +72,8 @@ describe('the TypeScript PDF engine is gone', () => {
|
||||
'@faunosteo/pdf',
|
||||
'pdf-lib',
|
||||
'@faunosteo/domain',
|
||||
'@faunosteo/anatomy',
|
||||
'@xmldom/xmldom',
|
||||
'tsx',
|
||||
]) {
|
||||
expect(names, path).not.toContain(banned)
|
||||
@@ -77,16 +81,27 @@ describe('the TypeScript PDF engine is gone', () => {
|
||||
}
|
||||
})
|
||||
|
||||
it('the lockfile holds no pdf-lib, @pdf-lib or resvg; xmldom only under packages/anatomy', () => {
|
||||
it('the lockfile holds no pdf-lib, @pdf-lib, resvg or xmldom (R5.7: the pipeline left with its xmldom)', () => {
|
||||
const lock = read('pnpm-lock.yaml')
|
||||
for (const name of ['pdf-lib@', '@pdf-lib/', '@resvg/resvg-js']) {
|
||||
expect(lock, name).not.toContain(name)
|
||||
}
|
||||
const importers = lock.split(/^ (?=[a-z@][^\n]*:$)/m)
|
||||
const withXmldom = importers
|
||||
.filter((block) => block.includes("'@xmldom/xmldom':"))
|
||||
.map((block) => block.split(':')[0]!.trim())
|
||||
expect(withXmldom).toEqual(['packages/anatomy'])
|
||||
const section = lock.slice(
|
||||
lock.indexOf('\nimporters:\n'),
|
||||
lock.indexOf('\npackages:\n'),
|
||||
)
|
||||
const importers = section
|
||||
.split(/\n(?= [^ \n][^\n]*:\n)/)
|
||||
.slice(1)
|
||||
.map((block) => block.slice(2, block.indexOf(':\n')))
|
||||
// Non-vacuity: the split finds the importers a dependency would sit under.
|
||||
expect(importers).toEqual(
|
||||
expect.arrayContaining(['.', 'apps/app', 'packages/core-wasm']),
|
||||
)
|
||||
expect(importers).not.toContain('packages/anatomy')
|
||||
expect(importers).not.toContain('packages/domain')
|
||||
expect(lock).toContain("'@faunosteo/core-wasm':")
|
||||
expect(lock).not.toContain('@xmldom/xmldom')
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* R5.6 in the application, on the switched bundled sheet (`sha256-rfc8785-validated-v1`):
|
||||
* a v2 report migrated now carries the canonical value and exports one consistent envelope
|
||||
* that the N-1 readers accept; an old report on the historical value opens with no rebase and
|
||||
* re-exports with its own stamp in the snapshot (the goldens prove the bytes); no visible
|
||||
* change. The historical goldens run in their original context (`helpers/goldenReports.ts`).
|
||||
*/
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { resolve } from 'node:path'
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
readArchiveDocument,
|
||||
readAttachmentBytes,
|
||||
ReportArchiveEnvelopeSchema,
|
||||
} from '@faunosteo/core-wasm/n-1'
|
||||
import { exportReportToPdf } from '@/pdf/export'
|
||||
import type { SheetIndex } from '@/services/sheetIndex'
|
||||
import { bundledMigrationMap, loadTestSheet } from '../helpers/sheet'
|
||||
import { FIXTURES_DIR, ONE_PX_PNG } from '../helpers/goldens'
|
||||
import { HISTORICAL_FINGERPRINT } from '../helpers/goldenReports'
|
||||
import { importAttachment } from '../helpers/pdfImport'
|
||||
|
||||
vi.mock('@/pdf/browserRasterizer', () => ({
|
||||
browserRasterizer: vi.fn(async () => ONE_PX_PNG),
|
||||
}))
|
||||
vi.mock('@/pdf/branding', async () => {
|
||||
const { brandingOf } = await import('../helpers/goldens')
|
||||
return { loadBranding: vi.fn(() => brandingOf('nologos')) }
|
||||
})
|
||||
|
||||
const CANONICAL =
|
||||
'12d788727c1e532641bd5b3a7ecf39dbdd25e70350f8a76be5ca917b262874e2'
|
||||
let sheet: SheetIndex
|
||||
|
||||
beforeAll(async () => {
|
||||
sheet = await loadTestSheet()
|
||||
vi.spyOn(console, 'warn').mockImplementation(() => undefined)
|
||||
})
|
||||
afterAll(() => vi.restoreAllMocks())
|
||||
|
||||
interface Stamped {
|
||||
report: { sheet: { documentSha256: string } }
|
||||
sheetSnapshot: { ref: { documentSha256: string } }
|
||||
}
|
||||
|
||||
async function exported(
|
||||
report: Parameters<typeof exportReportToPdf>[0],
|
||||
): Promise<Stamped> {
|
||||
const { pdf } = await exportReportToPdf(report, sheet, { locale: 'fr' })
|
||||
const bytes = new Uint8Array(await pdf.arrayBuffer())
|
||||
return JSON.parse(
|
||||
new TextDecoder().decode(await readAttachmentBytes(bytes)),
|
||||
) as Stamped
|
||||
}
|
||||
|
||||
describe('R5.6 on the switched sheet', () => {
|
||||
it('stamps a new or newly migrated report with the canonical value', async () => {
|
||||
expect(sheet.sheet.ref.documentSha256).toBe(CANONICAL)
|
||||
const v2 = JSON.parse(
|
||||
readFileSync(resolve(FIXTURES_DIR, 'report-v2.full.json'), 'utf8'),
|
||||
)
|
||||
const migrated = importAttachment(v2, sheet).read.report
|
||||
expect(migrated.sheet.documentSha256).toBe(CANONICAL)
|
||||
const envelope = await exported(migrated)
|
||||
expect(envelope.report.sheet.documentSha256).toBe(CANONICAL)
|
||||
expect(envelope.sheetSnapshot.ref.documentSha256).toBe(CANONICAL)
|
||||
expect(ReportArchiveEnvelopeSchema.safeParse(envelope).success).toBe(true)
|
||||
const outcome = readArchiveDocument(envelope, await bundledMigrationMap())
|
||||
expect(outcome.snapshotRejected).toBeUndefined()
|
||||
}, 30_000)
|
||||
|
||||
it('opens an old report with no rebase and re-exports it with one stamp', async () => {
|
||||
const demo = JSON.parse(
|
||||
readFileSync(resolve(FIXTURES_DIR, 'report-v3.demo.json'), 'utf8'),
|
||||
)
|
||||
const imported = importAttachment(demo, sheet)
|
||||
expect(imported.rebase.rebased).toBe(false)
|
||||
expect(imported.rebase.report.sheet.documentSha256).toBe(
|
||||
HISTORICAL_FINGERPRINT,
|
||||
)
|
||||
const envelope = await exported(imported.rebase.report)
|
||||
expect(envelope.report.sheet.documentSha256).toBe(HISTORICAL_FINGERPRINT)
|
||||
expect(envelope.sheetSnapshot.ref.documentSha256).toBe(
|
||||
HISTORICAL_FINGERPRINT,
|
||||
)
|
||||
}, 30_000)
|
||||
})
|
||||
@@ -0,0 +1,346 @@
|
||||
/**
|
||||
* R5.7 census (plan 2026-10-25, RUST.md §11.23): the anatomical pipeline and the
|
||||
* sheet, report and archive rules run in the Rust core only; `packages/anatomy`
|
||||
* and `packages/domain` are gone and nothing reaches them again. Grep facts over
|
||||
* the repository, and the two disposition matrices of the deletion checked
|
||||
* against the tree, so that a reintroduced package, dependency, import, script or
|
||||
* a ledger target that no longer exists fails here.
|
||||
*/
|
||||
import { createHash } from 'node:crypto'
|
||||
import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs'
|
||||
import { dirname, join, resolve } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '../../../..')
|
||||
const read = (path: string) => readFileSync(join(ROOT, path), 'utf8')
|
||||
const exists = (path: string) => existsSync(join(ROOT, path))
|
||||
const sha256 = (path: string) =>
|
||||
createHash('sha256')
|
||||
.update(readFileSync(join(ROOT, path)))
|
||||
.digest('hex')
|
||||
|
||||
const DELETED_PACKAGES = ['packages/anatomy', 'packages/domain']
|
||||
const BANNED_SPECIFIERS = [
|
||||
'@faunosteo/anatomy',
|
||||
'@faunosteo/domain',
|
||||
'@xmldom/xmldom',
|
||||
]
|
||||
const BANNED_DEPENDENCIES = [...BANNED_SPECIFIERS, 'tsx']
|
||||
|
||||
/** Source files under `dir` (relative to the root), node_modules excluded. */
|
||||
function sources(dir: string, pattern = /\.(tsx?|mjs|js)$/): string[] {
|
||||
const out: string[] = []
|
||||
const walk = (at: string) => {
|
||||
for (const entry of readdirSync(join(ROOT, at))) {
|
||||
if (entry === 'node_modules') continue
|
||||
const path = `${at}/${entry}`
|
||||
if (statSync(join(ROOT, path)).isDirectory()) walk(path)
|
||||
else if (pattern.test(entry)) out.push(path)
|
||||
}
|
||||
}
|
||||
walk(dir)
|
||||
return out
|
||||
}
|
||||
|
||||
/** Static, dynamic and `require` module specifiers of a source text. */
|
||||
const specifiersOf = (text: string): string[] =>
|
||||
[
|
||||
...text.matchAll(/\bfrom\s+['"]([^'"]+)['"]/g),
|
||||
...text.matchAll(/\bimport\s*\(\s*['"]([^'"]+)['"]/g),
|
||||
...text.matchAll(/\brequire\s*\(\s*['"]([^'"]+)['"]/g),
|
||||
].map((m) => m[1]!)
|
||||
|
||||
const isBanned = (spec: string) =>
|
||||
BANNED_SPECIFIERS.some((b) => spec === b || spec.startsWith(`${b}/`)) ||
|
||||
DELETED_PACKAGES.some((p) => spec.includes(p)) ||
|
||||
/(^|\/)(anatomy|domain)\/src(\/|$)/.test(spec)
|
||||
|
||||
type Manifest = Record<string, Record<string, string> | undefined>
|
||||
|
||||
/** Every workspace manifest: the root, `apps/*`, `packages/*` (with a manifest). */
|
||||
function manifests(): string[] {
|
||||
const found = ['package.json']
|
||||
for (const group of ['apps', 'packages']) {
|
||||
for (const entry of readdirSync(join(ROOT, group))) {
|
||||
const path = `${group}/${entry}/package.json`
|
||||
if (exists(path)) found.push(path)
|
||||
}
|
||||
}
|
||||
return found.sort()
|
||||
}
|
||||
|
||||
/** `pnpm-lock.yaml` importers (name → block) and the `packages:` keys. */
|
||||
function lockfile(): { importers: Map<string, string>; packages: string[] } {
|
||||
const lock = read('pnpm-lock.yaml')
|
||||
const start = lock.indexOf('\nimporters:\n')
|
||||
const end = lock.indexOf('\npackages:\n')
|
||||
const section = lock.slice(start, end)
|
||||
const importers = new Map<string, string>()
|
||||
for (const block of section.split(/\n(?= [^ \n][^\n]*:\n)/).slice(1)) {
|
||||
importers.set(block.slice(2, block.indexOf(':\n')), block)
|
||||
}
|
||||
const after = lock.slice(end + '\npackages:\n'.length)
|
||||
const packages = [
|
||||
...after
|
||||
.slice(0, after.indexOf('\nsnapshots:\n'))
|
||||
.matchAll(/^ {2}'?([^\s'][^\n]*?)'?:$/gm),
|
||||
].map((m) => m[1]!)
|
||||
return { importers, packages }
|
||||
}
|
||||
|
||||
/** Data rows of a committed TSV, checked against its header. */
|
||||
function tsv(path: string, header: string[]): string[][] {
|
||||
const [head, ...rows] = read(path).trimEnd().split('\n')
|
||||
expect(head!.split('\t'), path).toEqual(header)
|
||||
return rows.map((row) => {
|
||||
const cells = row.split('\t')
|
||||
expect(cells.length, `${path}: ${row}`).toBe(header.length)
|
||||
return cells
|
||||
})
|
||||
}
|
||||
|
||||
/** The file a ledger target names: `cargo <binary>::<path>::<test>`. */
|
||||
function cargoTargetFile(binary: string): string[] {
|
||||
if (binary === 'faunosteo_core') return sources('crates/core/src', /\.rs$/)
|
||||
if (['import_equine', 'report_compare'].includes(binary)) {
|
||||
return [`crates/core/src/bin/${binary}.rs`]
|
||||
}
|
||||
if (binary === 'upload_sheet_assets') {
|
||||
return sources('crates/core/src/bin/upload_sheet_assets', /\.rs$/)
|
||||
}
|
||||
return [`crates/core/tests/${binary}.rs`]
|
||||
}
|
||||
|
||||
describe('the TypeScript pipeline and domain packages are gone', () => {
|
||||
it('packages/anatomy, packages/domain and the TS upload script no longer exist', () => {
|
||||
// A folder may linger with a git-ignored node_modules of a stale install.
|
||||
for (const dir of DELETED_PACKAGES) {
|
||||
for (const part of ['package.json', 'src', 'tests']) {
|
||||
expect(exists(`${dir}/${part}`), `${dir}/${part}`).toBe(false)
|
||||
}
|
||||
}
|
||||
expect(exists('packages/supabase/scripts/upload-sheet-assets.ts')).toBe(
|
||||
false,
|
||||
)
|
||||
})
|
||||
|
||||
it('no workspace manifest depends on them, and the scripts run the Rust tools', () => {
|
||||
const found = manifests()
|
||||
for (const path of [
|
||||
'package.json',
|
||||
'apps/app/package.json',
|
||||
'packages/core-wasm/package.json',
|
||||
'packages/supabase/package.json',
|
||||
]) {
|
||||
expect(found).toContain(path)
|
||||
}
|
||||
for (const path of found) {
|
||||
const pkg = JSON.parse(read(path)) as Manifest
|
||||
const names = Object.keys({
|
||||
...pkg.dependencies,
|
||||
...pkg.devDependencies,
|
||||
...pkg.peerDependencies,
|
||||
...pkg.optionalDependencies,
|
||||
})
|
||||
for (const banned of BANNED_DEPENDENCIES) {
|
||||
expect(names, path).not.toContain(banned)
|
||||
}
|
||||
for (const [name, script] of Object.entries(pkg.scripts ?? {})) {
|
||||
expect(name, path).not.toMatch(/:legacy$/)
|
||||
expect(script, `${path} ${name}`).not.toMatch(
|
||||
/@faunosteo\/(anatomy|domain)|packages\/(anatomy|domain)|\btsx\b/,
|
||||
)
|
||||
}
|
||||
}
|
||||
const root = JSON.parse(read('package.json')) as Manifest
|
||||
expect(root.scripts?.['import:equine']).toBe(
|
||||
'bash scripts/import-equine.sh',
|
||||
)
|
||||
expect(root.scripts?.['r5:import-equine']).toBe(
|
||||
'bash scripts/import-equine.sh',
|
||||
)
|
||||
expect(root.scripts?.['assets:upload']).toBe(
|
||||
'bash scripts/upload-sheet-assets.sh',
|
||||
)
|
||||
expect(read('scripts/import-equine.sh')).toContain(
|
||||
'--features curation --bin import-equine',
|
||||
)
|
||||
expect(read('scripts/upload-sheet-assets.sh')).toContain(
|
||||
'--bin upload-sheet-assets',
|
||||
)
|
||||
})
|
||||
|
||||
it('the lockfile has one importer per manifest and resolves none of them', () => {
|
||||
const { importers, packages } = lockfile()
|
||||
const dirs = manifests().map((path) =>
|
||||
path === 'package.json' ? '.' : path.slice(0, -'/package.json'.length),
|
||||
)
|
||||
// Non-vacuity: the parser sees exactly the workspace the manifests declare.
|
||||
expect([...importers.keys()].sort()).toEqual(dirs.sort())
|
||||
expect(packages.length).toBeGreaterThan(100)
|
||||
expect(packages.some((key) => key.startsWith('zod@'))).toBe(true)
|
||||
for (const [name, block] of importers) {
|
||||
for (const banned of BANNED_DEPENDENCIES) {
|
||||
expect(block, `${name}: ${banned}`).not.toMatch(
|
||||
new RegExp(`^ {6}'?${banned.replace('/', '\\/')}'?:$`, 'm'),
|
||||
)
|
||||
}
|
||||
}
|
||||
for (const banned of BANNED_SPECIFIERS) {
|
||||
expect(
|
||||
packages.filter((key) => key.startsWith(`${banned}@`)),
|
||||
banned,
|
||||
).toEqual([])
|
||||
}
|
||||
// `tsx` left with the pipeline's CLI; it stays resolved only as the optional peer of vite,
|
||||
// which pnpm marks `optional: true` (`pnpm dedupe --check` clean, RUST.md §11.23).
|
||||
const lock = read('pnpm-lock.yaml')
|
||||
const snapshots = lock.slice(lock.indexOf('\nsnapshots:\n'))
|
||||
const tsx = snapshots.split('\n\n').filter((b) => /^ {2}tsx@/.test(b))
|
||||
for (const block of tsx) expect(block).toMatch(/^ {4}optional: true$/m)
|
||||
})
|
||||
})
|
||||
|
||||
describe('no production path reaches a deleted engine', () => {
|
||||
it('the application, the glue, the Supabase package and their configs import none of them', () => {
|
||||
const files = [
|
||||
...sources('apps/app/src'),
|
||||
...sources('apps/app/vite'),
|
||||
'apps/app/vite.config.ts',
|
||||
'apps/app/vitest.config.ts',
|
||||
'apps/app/eslint.config.js',
|
||||
...sources('packages/core-wasm/src'),
|
||||
...sources('packages/supabase/src'),
|
||||
]
|
||||
// Non-vacuity: the specifier scan finds the imports these sources do have.
|
||||
expect(files.length).toBeGreaterThan(100)
|
||||
const specs = files.flatMap((file) => specifiersOf(read(file)))
|
||||
expect(specs).toContain('@faunosteo/core-wasm')
|
||||
for (const file of files) {
|
||||
expect(specifiersOf(read(file)).filter(isBanned), file).toEqual([])
|
||||
}
|
||||
const tsconfig = read('apps/app/tsconfig.json')
|
||||
expect(tsconfig).not.toMatch(/packages\/(anatomy|domain)/)
|
||||
})
|
||||
|
||||
it('the Dockerfile copies only manifests that exist', () => {
|
||||
const dockerfile = read('Dockerfile')
|
||||
expect(dockerfile).not.toMatch(/packages\/(anatomy|domain)/)
|
||||
const copied = [
|
||||
...dockerfile.matchAll(/^COPY (\S+\/package\.json) /gm),
|
||||
].map((m) => m[1]!)
|
||||
expect(copied).toContain('packages/core-wasm/package.json')
|
||||
expect(copied).toContain('packages/supabase/package.json')
|
||||
for (const path of copied) expect(exists(path), path).toBe(true)
|
||||
})
|
||||
|
||||
it('the application lint rule keeps the deleted packages out', () => {
|
||||
const config = read('apps/app/eslint.config.js')
|
||||
for (const name of BANNED_SPECIFIERS) {
|
||||
expect(config).toContain(`name: '${name}'`)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('the disposition of the deleted files and tests', () => {
|
||||
it('every deleted file is absent and its counterpart exists', () => {
|
||||
const rows = tsv('crates/core/tests/curation/FILES.tsv', [
|
||||
'ts_file',
|
||||
'disposition',
|
||||
'counterpart',
|
||||
'note',
|
||||
])
|
||||
expect(rows.length).toBeGreaterThan(80)
|
||||
for (const [file, disposition, counterpart] of rows) {
|
||||
expect(file, file).toMatch(/^packages\/(anatomy|domain|supabase)\//)
|
||||
expect(exists(file!), file).toBe(false)
|
||||
expect(['ported', 'replaced', 'deleted'], file).toContain(disposition)
|
||||
if (disposition === 'deleted') continue
|
||||
for (const path of counterpart!.split(', ')) {
|
||||
expect(exists(path), `${file} → ${path}`).toBe(true)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('every TS case of the two packages has a disposition and a live target', () => {
|
||||
const rows = tsv('crates/core/tests/curation/LEDGER.tsv', [
|
||||
'ts_file',
|
||||
'case_title',
|
||||
'contract',
|
||||
'disposition',
|
||||
'target',
|
||||
'oracle',
|
||||
'note',
|
||||
])
|
||||
const per = (dir: string) =>
|
||||
rows.filter(([file]) => file!.startsWith(dir)).length
|
||||
expect(rows.length).toBe(407)
|
||||
expect(per('packages/anatomy/tests/')).toBe(144)
|
||||
expect(per('packages/domain/tests/')).toBe(263)
|
||||
for (const [file, title, , disposition, target, oracle] of rows) {
|
||||
const id = `${file} > ${title}`
|
||||
expect(exists(file!), id).toBe(false)
|
||||
expect(['ported', 'replaced', 'kept', 'deleted'], id).toContain(
|
||||
disposition,
|
||||
)
|
||||
for (const path of oracle === '-' ? [] : oracle!.split(', ')) {
|
||||
expect(exists(path), `${id}: ${path}`).toBe(true)
|
||||
}
|
||||
if (disposition === 'deleted') continue
|
||||
expect(target, id).not.toBe('-')
|
||||
for (const part of target!.split(' ; ')) {
|
||||
const cargo =
|
||||
/^cargo ([a-z0-9_]+)::(?:[a-z0-9_]+::)*([a-z0-9_]+)$/.exec(part)
|
||||
if (cargo) {
|
||||
const [, binary, test] = cargo
|
||||
const files = cargoTargetFile(binary!)
|
||||
// A `#[test]` function of that name, not any `fn` (attributes and doc lines between),
|
||||
// and never an `#[ignore]` one: `cargo test` would skip it and the row would be passive.
|
||||
const testFn = new RegExp(
|
||||
`#\\[test\\]\\s*(?:(?:#\\[(?!ignore)[^\\]]*\\]|///[^\\n]*)\\s*)*fn ${test}\\(`,
|
||||
)
|
||||
const defined = files.some(
|
||||
(path) => exists(path) && testFn.test(read(path)),
|
||||
)
|
||||
expect(defined, `${id}: ${part}`).toBe(true)
|
||||
continue
|
||||
}
|
||||
const vitest = /^vitest (\S+) > (.+)$/.exec(part)
|
||||
if (vitest) {
|
||||
expect(exists(vitest[1]!), `${id}: ${part}`).toBe(true)
|
||||
continue
|
||||
}
|
||||
expect(part, id).toBe('import-equine --check')
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('the historical references stay byte for byte', () => {
|
||||
it('the moved report fixtures keep their sha256', () => {
|
||||
expect(sha256('crates/core/tests/fixtures/report-v2.full.json')).toBe(
|
||||
'e96b088282b5d20bd5ed925ea980c1faca86cf1f87eb500201834dda714b729c',
|
||||
)
|
||||
expect(sha256('crates/core/tests/fixtures/report-v2.minimal.json')).toBe(
|
||||
'ba79698c5691dd89f8c6a3ada5a9c6ceb6ea41a85e64596e3eec7399c33cfde7',
|
||||
)
|
||||
})
|
||||
|
||||
it('the oracles, the pre-switch seed files and the N-1 readers stay', () => {
|
||||
for (const path of [
|
||||
'crates/core/tests/oracles/sheet.draft.json',
|
||||
'crates/core/tests/oracles/migration-map.draft.json',
|
||||
'crates/core/tests/oracles/r3/highlight.json',
|
||||
'crates/core/tests/oracles/r5/pipeline.json',
|
||||
'crates/core/tests/oracles/r5/ids.json',
|
||||
'crates/core/tests/oracles/r5/fingerprints.json',
|
||||
'crates/core/tests/fixtures/r5-6/pre-switch/sheet-ref.json',
|
||||
'crates/core/tests/fixtures/r5-6/pre-switch/report-migration-map.json',
|
||||
'crates/core/tests/fixtures/horse-slice.json',
|
||||
'packages/core-wasm/tests/n-1/domain',
|
||||
]) {
|
||||
expect(exists(path), path).toBe(true)
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -22,20 +22,41 @@ pdf-write = ["pdf-read"]
|
||||
# The archive check tool (R4.7, plan §B.1.7): `pdf::compare` and the `report-compare` binary,
|
||||
# the one file of the crate that reads the file system. Never enabled by the wasm facade.
|
||||
cli = ["pdf-read", "pdf-write"]
|
||||
# The anatomical pipeline (R5): legacy importer, curation, generated files and their
|
||||
# formatting, the `import-equine` tool. Never enabled by the wasm facade or the worker.
|
||||
curation = ["dep:icu_normalizer"]
|
||||
# The sheet-asset upload tool (R5.5): the `upload-sheet-assets` binary, a local-stack operator
|
||||
# tool. The rules are in `curation::upload`; only the binary reaches the network.
|
||||
upload = ["curation", "dep:ureq"]
|
||||
|
||||
[dependencies]
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
url.workspace = true
|
||||
roxmltree.workspace = true
|
||||
zmij.workspace = true
|
||||
ts-rs = { workspace = true, optional = true }
|
||||
miniz_oxide = { workspace = true, optional = true }
|
||||
# The sheet fingerprint (`sha256-rfc8785-validated-v1`, R5.6): recomputed at sheet load too.
|
||||
sha2.workspace = true
|
||||
icu_normalizer = { workspace = true, optional = true }
|
||||
ureq = { workspace = true, optional = true }
|
||||
|
||||
[[bin]]
|
||||
name = "report-compare"
|
||||
path = "src/bin/report_compare.rs"
|
||||
required-features = ["cli"]
|
||||
|
||||
[[bin]]
|
||||
name = "import-equine"
|
||||
path = "src/bin/import_equine.rs"
|
||||
required-features = ["curation"]
|
||||
|
||||
[[bin]]
|
||||
name = "upload-sheet-assets"
|
||||
path = "src/bin/upload_sheet_assets/main.rs"
|
||||
required-features = ["upload"]
|
||||
|
||||
[dev-dependencies]
|
||||
# Reads the PDFs the tests compare (the R3 fixtures and the R4 output): objects, streams,
|
||||
# annotations; never in the library.
|
||||
@@ -98,3 +119,44 @@ required-features = ["cli"]
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
||||
# The pipeline suites (R5) compile only with the feature.
|
||||
[[test]]
|
||||
name = "curation_format"
|
||||
required-features = ["curation"]
|
||||
|
||||
[[test]]
|
||||
name = "curation_legacy"
|
||||
required-features = ["curation"]
|
||||
|
||||
[[test]]
|
||||
name = "curation_draft"
|
||||
required-features = ["curation"]
|
||||
|
||||
[[test]]
|
||||
name = "curation_review"
|
||||
required-features = ["curation"]
|
||||
|
||||
[[test]]
|
||||
name = "curation_markdown"
|
||||
required-features = ["curation"]
|
||||
|
||||
[[test]]
|
||||
name = "curation_caps"
|
||||
required-features = ["curation"]
|
||||
|
||||
[[test]]
|
||||
name = "curation_upload"
|
||||
required-features = ["curation"]
|
||||
|
||||
[[test]]
|
||||
name = "upload_tool"
|
||||
required-features = ["upload"]
|
||||
|
||||
[[test]]
|
||||
name = "sheet_fingerprint"
|
||||
required-features = ["pdf-read"]
|
||||
|
||||
[[test]]
|
||||
name = "r5_6_switch"
|
||||
required-features = ["pdf-read", "pdf-write"]
|
||||
@@ -1,5 +1,7 @@
|
||||
//! The anatomy core (R3): SVG reading and sanitising, the annotation layer drawn from the
|
||||
//! sheet document, and the composition of a view (canvas and highlight).
|
||||
//! sheet document, and the composition of a view (canvas and highlight). The TypeScript
|
||||
//! renderer it replaced (`packages/anatomy/src/render`) was deleted at R5.7; cited paths resolve
|
||||
//! at commit 756b7ef.
|
||||
|
||||
pub mod compose;
|
||||
pub mod layer;
|
||||
|
||||
@@ -0,0 +1,537 @@
|
||||
//! `import-equine` (R5.2 to R5.4): the anatomical pipeline on the core. Reads the historical
|
||||
//! equine files, the authored curation files and the frozen cleaned drawing of the repository,
|
||||
//! and produces the two generated folders the TypeScript `pnpm import:equine` writes, whole:
|
||||
//! `legacy-import/` (6 files) and `draft/` (12 files).
|
||||
//!
|
||||
//! import-equine --check compare with the committed folders, write nothing: every file
|
||||
//! byte for byte, except the application's drawings, which may
|
||||
//! fall back on parsed trees under the declared R3 rule; a file
|
||||
//! missing, not produced or different fails the check
|
||||
//! import-equine --out <dir> write <dir>/legacy-import and <dir>/draft (<dir> absent or empty)
|
||||
//!
|
||||
//! The repository is found from this crate's manifest folder; a relative `--out` is typed from
|
||||
//! the folder pnpm was run in (`INIT_CWD`), else the current one. Nothing is ever deleted:
|
||||
//! `--out` refuses an empty path, a symbolic link, a path that is not a folder and a folder
|
||||
//! holding any file, so the official seed can never be rewritten by this tool. Like
|
||||
//! `report-compare`, this file is the only one of its tool allowed the file system.
|
||||
#![allow(clippy::disallowed_methods)]
|
||||
|
||||
use std::fs::OpenOptions;
|
||||
use std::io::{Read, Write};
|
||||
use std::path::{Component, Path, PathBuf};
|
||||
use std::{env, fs, process};
|
||||
|
||||
use faunosteo_core::curation::draft::schema::MAX_CURATION_FILE_BYTES;
|
||||
use faunosteo_core::curation::draft::svg_compare::{TriggerNames, compare_drawing};
|
||||
use faunosteo_core::curation::draft::{
|
||||
DraftInput, build_equine_draft, parse_changeset, parse_references, parse_view_box,
|
||||
render_draft_artifacts,
|
||||
};
|
||||
use faunosteo_core::curation::legacy::{
|
||||
EquineLegacyInput, HEAD_VIEW_ID, ImportResult, SvgSource, import_equine_legacy,
|
||||
render_artifacts,
|
||||
};
|
||||
use faunosteo_core::domain::sheet::AnatomicalSheetDocument;
|
||||
|
||||
const SEED: &str = concat!(
|
||||
env!("CARGO_MANIFEST_DIR"),
|
||||
"/../../supabase/seed/anatomical-sheets/equine"
|
||||
);
|
||||
|
||||
const LEGACY_FOLDER: &str = "legacy-import";
|
||||
const DRAFT_FOLDER: &str = "draft";
|
||||
const CLEANED_ASSET: &str = "assets/horse.svg";
|
||||
|
||||
/// The files of the draft whose comparison may fall back on parsed trees under the declared
|
||||
/// R3 trigger-name rule (`svg_compare`): the application's drawings of each view.
|
||||
const APP_DRAWING_PREFIX: &str = "review/app-";
|
||||
|
||||
/// A file of a generated folder: its path relative to the folder and its bytes.
|
||||
type FolderFile = (String, Vec<u8>);
|
||||
|
||||
/// A generated folder: its name under the seed and its files.
|
||||
struct Folder {
|
||||
name: &'static str,
|
||||
files: Vec<FolderFile>,
|
||||
/// The declared tree rule of the application's drawings (the draft only).
|
||||
trigger_names: Option<TriggerNames>,
|
||||
}
|
||||
|
||||
/// A bound no source or generated file approaches (the largest, the draft CHANGELOG, is about
|
||||
/// 1.1 MB): a resource cap on what the tool reads.
|
||||
const MAX_FILE_BYTES: u64 = 16 * 1024 * 1024;
|
||||
|
||||
/// The cap of an authored curation file, the core's own (`MAX_CURATION_FILE_BYTES`): the file
|
||||
/// is never read beyond it, so a larger one is refused before the allocation.
|
||||
const MAX_CURATION_BYTES: u64 = MAX_CURATION_FILE_BYTES as u64;
|
||||
|
||||
/// A regular file of the seed folder, read under the cap: its canonical path (every symbolic
|
||||
/// link resolved, folders included) must lie in the canonical seed folder, so a link never
|
||||
/// leads a read outside it; devices and pipes are refused. The repository is a single user's
|
||||
/// working copy: a swap between the check and the read is not defended (RUST.md §11.13).
|
||||
fn read(path: &Path) -> Result<Vec<u8>, String> {
|
||||
read_capped(path, MAX_FILE_BYTES)
|
||||
}
|
||||
|
||||
/// `read` under a given cap, enforced on the bytes actually read as well as on the size seen
|
||||
/// first, so a file that grows between the two is still refused before it is held.
|
||||
fn read_capped(path: &Path, cap: u64) -> Result<Vec<u8>, String> {
|
||||
let refuse = |why: &str| format!("{}: {why}", path.display());
|
||||
let seed = fs::canonicalize(SEED).map_err(|error| format!("{SEED}: {error}"))?;
|
||||
let canonical = fs::canonicalize(path).map_err(|error| refuse(&error.to_string()))?;
|
||||
if !inside(&canonical, &seed) {
|
||||
return Err(refuse("leads outside the seed folder"));
|
||||
}
|
||||
let metadata = fs::metadata(&canonical).map_err(|error| refuse(&error.to_string()))?;
|
||||
if !metadata.file_type().is_file() {
|
||||
return Err(refuse("not a regular file"));
|
||||
}
|
||||
if metadata.len() > cap {
|
||||
return Err(refuse(&format!("larger than {cap} bytes")));
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
let file = fs::File::open(&canonical).map_err(|error| refuse(&error.to_string()))?;
|
||||
file.take(cap + 1)
|
||||
.read_to_end(&mut bytes)
|
||||
.map_err(|error| refuse(&error.to_string()))?;
|
||||
if bytes.len() as u64 > cap {
|
||||
return Err(refuse(&format!("larger than {cap} bytes")));
|
||||
}
|
||||
Ok(bytes)
|
||||
}
|
||||
|
||||
/// `readFile(path, 'utf8')` of an authored curation file, under the 4 MiB curation cap: invalid
|
||||
/// UTF-8 becomes U+FFFD, as the TypeScript read them.
|
||||
fn read_curation_text(path: &Path) -> Result<String, String> {
|
||||
Ok(String::from_utf8_lossy(&read_capped(path, MAX_CURATION_BYTES)?).into_owned())
|
||||
}
|
||||
|
||||
fn svg(name: &str) -> Result<SvgSource, String> {
|
||||
let bytes = read(&Path::new(SEED).join("legacy-sources").join(name))?;
|
||||
Ok(SvgSource {
|
||||
file_name: name.to_owned(),
|
||||
bytes,
|
||||
})
|
||||
}
|
||||
|
||||
fn legacy_input() -> Result<EquineLegacyInput, String> {
|
||||
let sources = Path::new(SEED).join("legacy-sources");
|
||||
Ok(EquineLegacyInput {
|
||||
main_svg: svg("Horse.svg")?,
|
||||
head_svg: svg("Horse_head.svg")?,
|
||||
catalog: read(&sources.join("pathologies.json"))?,
|
||||
vertebrae: read(&sources.join("vertebrae.ts"))?,
|
||||
})
|
||||
}
|
||||
|
||||
fn texts(files: Vec<(String, String)>) -> Vec<FolderFile> {
|
||||
files
|
||||
.into_iter()
|
||||
.map(|(name, text)| (name, text.into_bytes()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// `legacy-import/`: the four texts, then the two SVG copies.
|
||||
fn legacy_folder(input: &EquineLegacyInput, result: &ImportResult) -> Result<Folder, String> {
|
||||
let mut files = texts(render_artifacts(result).map_err(|error| error.to_string())?);
|
||||
for source in [&input.main_svg, &input.head_svg] {
|
||||
files.push((format!("assets/{}", source.file_name), source.bytes.clone()));
|
||||
}
|
||||
Ok(Folder {
|
||||
name: LEGACY_FOLDER,
|
||||
files,
|
||||
trigger_names: None,
|
||||
})
|
||||
}
|
||||
|
||||
/// `draft/`: the R5.3 texts, then the frozen cleaned drawing (read, never re-cleaned).
|
||||
fn draft_folder(result: &ImportResult) -> Result<Folder, String> {
|
||||
let curation = Path::new(SEED).join("curation");
|
||||
let references = parse_references(&read_curation_text(&curation.join("references.json"))?)
|
||||
.map_err(|error| error.to_string())?;
|
||||
let changeset = parse_changeset(&read_curation_text(&curation.join("changes.json"))?)
|
||||
.map_err(|error| error.to_string())?;
|
||||
let cleaned_svg = read(&Path::new(SEED).join(DRAFT_FOLDER).join(CLEANED_ASSET))?;
|
||||
let head = result.view_boxes.get(HEAD_VIEW_ID).and_then(|v| v.as_str());
|
||||
let head_view_box = parse_view_box(head).map_err(|error| error.to_string())?;
|
||||
let draft = build_equine_draft(&DraftInput {
|
||||
import_result: result,
|
||||
cleaned_svg: &cleaned_svg,
|
||||
head_view_box,
|
||||
references: &references,
|
||||
changes: &changeset.changes,
|
||||
})
|
||||
.map_err(|error| error.to_string())?;
|
||||
let rendered =
|
||||
render_draft_artifacts(&draft, &result.anomalies).map_err(|error| error.to_string())?;
|
||||
let typed: AnatomicalSheetDocument =
|
||||
serde_json::from_value(draft.document.clone()).map_err(|error| error.to_string())?;
|
||||
let mut files = texts(rendered);
|
||||
files.push((CLEANED_ASSET.to_owned(), cleaned_svg));
|
||||
Ok(Folder {
|
||||
name: DRAFT_FOLDER,
|
||||
files,
|
||||
trigger_names: Some(TriggerNames::of(&typed)),
|
||||
})
|
||||
}
|
||||
|
||||
fn folders() -> Result<Vec<Folder>, String> {
|
||||
let input = legacy_input()?;
|
||||
let result = import_equine_legacy(&input).map_err(|error| error.to_string())?;
|
||||
Ok(vec![
|
||||
legacy_folder(&input, &result)?,
|
||||
draft_folder(&result)?,
|
||||
])
|
||||
}
|
||||
|
||||
/// Relative paths of every file under `dir`.
|
||||
fn listed(dir: &Path, prefix: &str, out: &mut Vec<String>) -> Result<(), String> {
|
||||
let entries = fs::read_dir(dir).map_err(|error| format!("{}: {error}", dir.display()))?;
|
||||
for entry in entries {
|
||||
let entry = entry.map_err(|error| error.to_string())?;
|
||||
let name = format!("{prefix}{}", entry.file_name().to_string_lossy());
|
||||
// `file_type` does not follow a symbolic link: a link is listed, never walked into.
|
||||
let is_dir = entry.file_type().is_ok_and(|kind| kind.is_dir());
|
||||
if is_dir {
|
||||
listed(&entry.path(), &format!("{name}/"), out)?;
|
||||
} else {
|
||||
out.push(name);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// How one produced file compares with the committed one: byte identity, or, for the
|
||||
/// application's drawings only, parsed trees under the declared R3 rule (printed as declared).
|
||||
fn compare_file(
|
||||
folder: &Folder,
|
||||
name: &str,
|
||||
produced: &[u8],
|
||||
committed: &[u8],
|
||||
) -> Result<(), String> {
|
||||
if produced == committed {
|
||||
return Ok(());
|
||||
}
|
||||
let rule = folder
|
||||
.trigger_names
|
||||
.as_ref()
|
||||
.filter(|_| name.starts_with(APP_DRAWING_PREFIX));
|
||||
let (Ok(produced), Ok(committed)) = (
|
||||
std::str::from_utf8(produced),
|
||||
std::str::from_utf8(committed),
|
||||
) else {
|
||||
return Err(format!("{}: differs: {name}", folder.name));
|
||||
};
|
||||
match compare_drawing(produced, committed, rule) {
|
||||
Ok(_) => {
|
||||
println!(
|
||||
"{}: declared: {name} (byte-identical once the R3 trigger-name rule is applied)",
|
||||
folder.name
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
Err(why) => Err(format!("{}: differs: {name} ({why})", folder.name)),
|
||||
}
|
||||
}
|
||||
|
||||
/// The differences between a produced folder and the committed one: every committed file must
|
||||
/// be produced and every produced file committed and equal (`compare_file`).
|
||||
fn check(folder: &Folder) -> Result<Vec<String>, String> {
|
||||
let target = Path::new(SEED).join(folder.name);
|
||||
// The generated folder itself must lie in the seed: a link there is never listed through.
|
||||
let seed = fs::canonicalize(SEED).map_err(|error| format!("{SEED}: {error}"))?;
|
||||
let canonical = fs::canonicalize(&target).map_err(|e| format!("{}: {e}", target.display()))?;
|
||||
if canonical != seed.join(folder.name) {
|
||||
return Ok(vec![format!("{}: not a folder of the seed", folder.name)]);
|
||||
}
|
||||
let mut problems = Vec::new();
|
||||
for (name, bytes) in &folder.files {
|
||||
match read(&target.join(name)) {
|
||||
Ok(committed) => {
|
||||
if let Err(problem) = compare_file(folder, name, bytes, &committed) {
|
||||
problems.push(problem);
|
||||
}
|
||||
}
|
||||
Err(_) => problems.push(format!("{}: missing: {name}", folder.name)),
|
||||
}
|
||||
}
|
||||
let mut present = Vec::new();
|
||||
listed(&target, "", &mut present)?;
|
||||
for name in present {
|
||||
if folder.files.iter().any(|(produced, _)| *produced == name) {
|
||||
continue;
|
||||
}
|
||||
problems.push(format!("{}: not produced: {name}", folder.name));
|
||||
}
|
||||
Ok(problems)
|
||||
}
|
||||
|
||||
/// Claims a destination this tool may fill and returns its canonical path. The typed path
|
||||
/// must be non-empty, hold no `..`, have an existing parent folder, and name either nothing
|
||||
/// (the folder is created here, its parent's symbolic links resolved) or an empty folder that
|
||||
/// is not a symbolic link. A destination inside the seed folder is refused: the tool never
|
||||
/// writes into the official seed. The writes that follow are exclusive (`write_exclusive`).
|
||||
fn claim_destination(typed: &Path, seed: &Path) -> Result<PathBuf, String> {
|
||||
let refuse = |why: &str| format!("{}: {why}, nothing written", typed.display());
|
||||
if typed.as_os_str().is_empty() {
|
||||
return Err("--out needs a destination path".to_owned());
|
||||
}
|
||||
if typed.components().any(|c| c == Component::ParentDir) {
|
||||
return Err(refuse("`..` is not accepted"));
|
||||
}
|
||||
// Rebuilt from its components: a trailing `/` would make the system resolve a final
|
||||
// symbolic link before the check below could see it.
|
||||
let out: PathBuf = typed.components().collect();
|
||||
let parent = out.parent().filter(|p| !p.as_os_str().is_empty());
|
||||
let parent = match parent {
|
||||
Some(parent) => {
|
||||
fs::canonicalize(parent).map_err(|_| refuse("the parent folder does not exist"))?
|
||||
}
|
||||
None => return Err(refuse("not a destination folder")),
|
||||
};
|
||||
if inside(&parent, seed) || inside_by_identity(&parent, seed)? {
|
||||
return Err(refuse("inside the seed folder"));
|
||||
}
|
||||
let name = out
|
||||
.file_name()
|
||||
.ok_or_else(|| format!("{}: not a destination folder", typed.display()))?;
|
||||
let root = parent.join(name);
|
||||
match fs::symlink_metadata(&root) {
|
||||
Err(_) => fs::create_dir(&root).map_err(|error| refuse(&error.to_string()))?,
|
||||
Ok(metadata) if metadata.file_type().is_symlink() => return Err(refuse("a symbolic link")),
|
||||
Ok(metadata) if !metadata.is_dir() => return Err(refuse("not a folder")),
|
||||
Ok(_) => {
|
||||
// An existing folder may itself be a second path to a folder of the seed.
|
||||
if inside_by_identity(&root, seed)? {
|
||||
return Err(refuse("inside the seed folder"));
|
||||
}
|
||||
let mut entries =
|
||||
fs::read_dir(&root).map_err(|error| format!("{}: {error}", root.display()))?;
|
||||
if entries.next().is_some() {
|
||||
return Err(refuse("not empty"));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(root)
|
||||
}
|
||||
|
||||
/// Whether a canonical folder is the seed folder or lies under it.
|
||||
fn inside(folder: &Path, seed: &Path) -> bool {
|
||||
folder.starts_with(seed)
|
||||
}
|
||||
|
||||
/// The file identities (device, inode) of the seed folder and of every folder under it, links
|
||||
/// never followed; `None` when the walk fails (the caller then refuses).
|
||||
fn seed_identities(seed: &Path) -> Option<Vec<(u64, u64)>> {
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
let mut identities = Vec::new();
|
||||
let mut pending = vec![seed.to_path_buf()];
|
||||
while let Some(folder) = pending.pop() {
|
||||
let metadata = fs::symlink_metadata(&folder).ok()?;
|
||||
identities.push((metadata.dev(), metadata.ino()));
|
||||
for entry in fs::read_dir(&folder).ok()? {
|
||||
let entry = entry.ok()?;
|
||||
if entry.file_type().ok()?.is_dir() {
|
||||
pending.push(entry.path());
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(identities)
|
||||
}
|
||||
|
||||
/// Whether `folder` or one of its ancestors is the seed folder or a folder under it, by file
|
||||
/// identity: a second path to any of them (a bind mount, a share mounted twice on the same
|
||||
/// device) is caught too. A seed walk that fails is an error (the claim then refuses).
|
||||
fn inside_by_identity(folder: &Path, seed: &Path) -> Result<bool, String> {
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
let identities = seed_identities(seed).ok_or_else(|| {
|
||||
format!(
|
||||
"{}: the seed folder cannot be read, nothing written",
|
||||
seed.display()
|
||||
)
|
||||
})?;
|
||||
Ok(folder.ancestors().any(|ancestor| {
|
||||
fs::metadata(ancestor).is_ok_and(|m| identities.contains(&(m.dev(), m.ino())))
|
||||
}))
|
||||
}
|
||||
|
||||
/// Where a relative destination is typed from: `INIT_CWD` when pnpm set it to an absolute
|
||||
/// path, else the current folder (the repository root when the script is run directly: it
|
||||
/// moves there first).
|
||||
fn base_dir() -> PathBuf {
|
||||
env::var_os("INIT_CWD")
|
||||
.map(PathBuf::from)
|
||||
.filter(|path| path.is_absolute())
|
||||
.or_else(|| env::current_dir().ok())
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Creates `relative` and the folders leading to it under `root`: every folder is one this run
|
||||
/// created (`created`), every file new; an entry that appeared since the claim, file, folder
|
||||
/// or symbolic link, is refused, never followed or truncated (`create_dir`, `create_new`).
|
||||
fn write_exclusive(
|
||||
root: &Path,
|
||||
relative: &Path,
|
||||
bytes: &[u8],
|
||||
created: &mut Vec<PathBuf>,
|
||||
) -> Result<(), String> {
|
||||
// The claimed folder is checked again: a swap for a symbolic link since the claim is refused.
|
||||
if !fs::symlink_metadata(root).is_ok_and(|m| m.file_type().is_dir()) {
|
||||
return Err(format!("{}: no longer the claimed folder", root.display()));
|
||||
}
|
||||
let mut folder = root.to_path_buf();
|
||||
let components: Vec<_> = relative.components().collect();
|
||||
if !components.iter().all(|c| matches!(c, Component::Normal(_))) {
|
||||
return Err(format!(
|
||||
"{}: not a plain relative file name",
|
||||
relative.display()
|
||||
));
|
||||
}
|
||||
let (file, folders) = components.split_last().ok_or("an empty file name")?;
|
||||
for component in folders {
|
||||
folder.push(component);
|
||||
if created.contains(&folder) {
|
||||
continue;
|
||||
}
|
||||
fs::create_dir(&folder).map_err(|error| format!("{}: {error}", folder.display()))?;
|
||||
created.push(folder.clone());
|
||||
}
|
||||
let path = folder.join(file);
|
||||
let mut handle = OpenOptions::new()
|
||||
.write(true)
|
||||
.create_new(true)
|
||||
.open(&path)
|
||||
.map_err(|error| format!("{}: {error}", path.display()))?;
|
||||
handle
|
||||
.write_all(bytes)
|
||||
.map_err(|error| format!("{}: {error}", path.display()))
|
||||
}
|
||||
|
||||
/// Writes every file under the claimed destination; returns the file count.
|
||||
fn write(folders: &[Folder], root: &Path) -> Result<usize, String> {
|
||||
let mut written = 0;
|
||||
let mut created = Vec::new();
|
||||
for folder in folders {
|
||||
for (name, bytes) in &folder.files {
|
||||
let relative = Path::new(folder.name).join(name);
|
||||
write_exclusive(root, &relative, bytes, &mut created)?;
|
||||
written += 1;
|
||||
}
|
||||
}
|
||||
Ok(written)
|
||||
}
|
||||
|
||||
fn run(args: &[String]) -> Result<i32, String> {
|
||||
let folders = folders()?;
|
||||
match args {
|
||||
[flag] if flag == "--check" => {
|
||||
let mut total = 0;
|
||||
for folder in &folders {
|
||||
let problems = check(folder)?;
|
||||
for problem in &problems {
|
||||
eprintln!("{problem}");
|
||||
}
|
||||
println!(
|
||||
"{}: {} files, {} differences",
|
||||
folder.name,
|
||||
folder.files.len(),
|
||||
problems.len()
|
||||
);
|
||||
total += problems.len();
|
||||
}
|
||||
Ok(i32::from(total != 0))
|
||||
}
|
||||
[flag, dir] if flag == "--out" => {
|
||||
if dir.is_empty() {
|
||||
return Err("--out needs a destination path".to_owned());
|
||||
}
|
||||
let seed = fs::canonicalize(SEED).map_err(|error| format!("{SEED}: {error}"))?;
|
||||
let root = claim_destination(&base_dir().join(dir), &seed)?;
|
||||
println!("writing to {}", root.display());
|
||||
let written = write(&folders, &root)?;
|
||||
println!("{written} files written to {}", root.display());
|
||||
Ok(0)
|
||||
}
|
||||
_ => Err("usage: import-equine --check | --out <dir>".to_owned()),
|
||||
}
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let args: Option<Vec<String>> = env::args_os()
|
||||
.skip(1)
|
||||
.map(|a| a.into_string().ok())
|
||||
.collect();
|
||||
let Some(args) = args else {
|
||||
eprintln!("an argument is not valid UTF-8");
|
||||
process::exit(2);
|
||||
};
|
||||
match run(&args) {
|
||||
Ok(code) => process::exit(code),
|
||||
Err(message) => {
|
||||
eprintln!("{message}");
|
||||
process::exit(2);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Fixed fixtures under the workspace's target folder (private to the checkout): created
|
||||
/// once, never deleted, never written by the refusals below.
|
||||
fn fixture_seed() -> PathBuf {
|
||||
let base = Path::new(concat!(env!("CARGO_MANIFEST_DIR"), "/../../target/tmp"))
|
||||
.join("import-equine-claim");
|
||||
fs::create_dir_all(base.join("seed/draft")).unwrap_or_else(|e| panic!("{e}"));
|
||||
fs::canonicalize(base.join("seed")).unwrap_or_else(|e| panic!("{e}"))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_claim_refuses_a_destination_inside_the_seed_and_creates_nothing() {
|
||||
let seed = fixture_seed();
|
||||
for typed in [seed.join("new"), seed.join("draft/new")] {
|
||||
let refused = claim_destination(&typed, &seed).err().unwrap_or_default();
|
||||
assert!(refused.contains("inside the seed folder"), "{refused}");
|
||||
assert!(fs::symlink_metadata(&typed).is_err());
|
||||
}
|
||||
let parent = seed.parent().map(Path::to_path_buf).unwrap_or_default();
|
||||
let refused = claim_destination(&parent, &seed).err().unwrap_or_default();
|
||||
assert!(refused.contains("not empty"), "{refused}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_difference_fails_the_check_unless_a_declared_rule_covers_it() {
|
||||
let draft = Folder {
|
||||
name: DRAFT_FOLDER,
|
||||
files: Vec::new(),
|
||||
trigger_names: None,
|
||||
};
|
||||
assert!(compare_file(&draft, "CHANGELOG.md", b"a\n", b"a\n").is_ok());
|
||||
let differs = compare_file(&draft, "CHANGELOG.md", b"a\n", b"b\n")
|
||||
.err()
|
||||
.unwrap_or_default();
|
||||
assert!(differs.contains("differs: CHANGELOG.md"), "{differs}");
|
||||
// No rule for the folder: even an application drawing must be byte-identical.
|
||||
let svg = b"<svg xmlns=\"http://www.w3.org/2000/svg\"/>";
|
||||
let other = b"<svg xmlns=\"http://www.w3.org/2000/svg\"><g/></svg>";
|
||||
assert!(compare_file(&draft, "review/app-main.svg", svg, other).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_folder_under_the_seed_or_the_seed_itself_is_inside_it() {
|
||||
let seed = Path::new("/repo/supabase/seed/anatomical-sheets/equine");
|
||||
assert!(inside(seed, seed));
|
||||
assert!(inside(&seed.join("draft"), seed));
|
||||
assert!(inside(&seed.join("x/y"), seed));
|
||||
assert!(!inside(
|
||||
Path::new("/repo/supabase/seed/anatomical-sheets"),
|
||||
seed
|
||||
));
|
||||
assert!(!inside(
|
||||
Path::new("/repo/supabase/seed/anatomical-sheets/equine-copy"),
|
||||
seed
|
||||
));
|
||||
assert!(!inside(Path::new("/tmp/out"), seed));
|
||||
}
|
||||
}
|
||||
@@ -37,6 +37,11 @@ struct SeedRef {
|
||||
document_sha256: String,
|
||||
schema_version: f64,
|
||||
species_id: String,
|
||||
#[serde(
|
||||
default,
|
||||
deserialize_with = "faunosteo_core::domain::sheet::loaded::present_block"
|
||||
)]
|
||||
fingerprint: Option<faunosteo_core::domain::sheet::loaded::DeclaredFingerprint>,
|
||||
}
|
||||
|
||||
fn seed_json<T: for<'de> Deserialize<'de>>(name: &str) -> Result<T, String> {
|
||||
@@ -58,6 +63,7 @@ fn load_sheet() -> Result<LoadedSheet, String> {
|
||||
document_sha256: reference.document_sha256,
|
||||
schema_version,
|
||||
species_id: reference.species_id,
|
||||
fingerprint: reference.fingerprint,
|
||||
},
|
||||
document: seed_json("document.json")?,
|
||||
migration_map: seed_json("report-migration-map.json")?,
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
//! `upload-sheet-assets` (R5.5): uploads the SVG drawings of a generated sheet seed folder to
|
||||
//! the private Storage bucket `anatomical-sheet-assets` of a LOCAL Supabase stack, content
|
||||
//! addressed (`<sha256>.svg`), and confirms each asset row once its object is verified.
|
||||
//!
|
||||
//! upload-sheet-assets --check <seed folder> read-only: verify the files, the destination
|
||||
//! and what is already there; write nothing
|
||||
//! upload-sheet-assets <seed folder> the same, then upload and confirm what is missing
|
||||
//!
|
||||
//! Environment: `SUPABASE_URL` (a local stack: loopback, or the DEV address
|
||||
//! `FAUNOSTEO_DEV_HOST`; `http` with an explicit port; no default) and
|
||||
//! `SUPABASE_SERVICE_ROLE_KEY` (never printed). The folder lies under
|
||||
//! `supabase/seed/anatomical-sheets`; a relative path is typed from the folder pnpm was run in
|
||||
//! (`INIT_CWD`), else the current one.
|
||||
//!
|
||||
//! Order of a run: files and manifest checked, destination checked, handshake, then every
|
||||
//! entry planned from the state found; any refusal stops the run before the first write. An
|
||||
//! object that exists is read back and compared with its key, never replaced; an asset row is
|
||||
//! confirmed only after its object is verified. A rerun after a failure resumes safely.
|
||||
//! Like `import-equine`, this tool owns the file system, the environment and the network; the
|
||||
//! rules are the core's (`curation::upload`).
|
||||
#![allow(clippy::disallowed_methods)]
|
||||
|
||||
mod storage;
|
||||
|
||||
use std::io::Read;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::{env, fs, process};
|
||||
|
||||
use faunosteo_core::curation::upload::{
|
||||
AssetEntry, MAX_ASSET_BYTES, MAX_MANIFEST_BYTES, ObjectState, Plan, UploadError, UploadKind,
|
||||
check_destination, parse_manifest, plan_entry, refuse, verify_asset,
|
||||
};
|
||||
use storage::{Secret, Storage};
|
||||
|
||||
const SEED_ROOT: &str = concat!(
|
||||
env!("CARGO_MANIFEST_DIR"),
|
||||
"/../../supabase/seed/anatomical-sheets"
|
||||
);
|
||||
|
||||
/// A local entry whose bytes passed `verify_asset`.
|
||||
struct Local {
|
||||
entry: AssetEntry,
|
||||
bytes: Vec<u8>,
|
||||
}
|
||||
|
||||
/// A regular file under `root` (every link resolved), read under a cap: a file that grows
|
||||
/// between the size check and the read is still refused. The seed is a single user's working
|
||||
/// copy: a swap between the check and the read is not defended.
|
||||
fn read_confined(path: &Path, root: &Path, cap: usize) -> Result<Vec<u8>, UploadError> {
|
||||
let bad = |why: &str| refuse(UploadKind::Content, format!("{}: {why}", path.display()));
|
||||
let canonical = fs::canonicalize(path).map_err(|e| bad(&e.to_string()))?;
|
||||
if !canonical.starts_with(root) {
|
||||
return Err(bad("leads outside the seed folder"));
|
||||
}
|
||||
let metadata = fs::metadata(&canonical).map_err(|e| bad(&e.to_string()))?;
|
||||
if !metadata.is_file() || metadata.len() > cap as u64 {
|
||||
return Err(bad("not a regular file, or too large"));
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
let file = fs::File::open(&canonical).map_err(|e| bad(&e.to_string()))?;
|
||||
file.take(cap as u64 + 1)
|
||||
.read_to_end(&mut bytes)
|
||||
.map_err(|e| bad(&e.to_string()))?;
|
||||
if bytes.len() > cap {
|
||||
return Err(bad("too large"));
|
||||
}
|
||||
Ok(bytes)
|
||||
}
|
||||
|
||||
/// The folder typed on the command line: from `INIT_CWD` when pnpm set it, canonical, a
|
||||
/// directory, inside the seed root.
|
||||
fn resolve_folder(typed: &str) -> Result<PathBuf, UploadError> {
|
||||
let bad = |why: &str| refuse(UploadKind::Content, format!("{typed}: {why}"));
|
||||
let base = env::var_os("INIT_CWD")
|
||||
.map(PathBuf::from)
|
||||
.filter(|p| p.is_absolute())
|
||||
.map_or_else(env::current_dir, Ok)
|
||||
.map_err(|e| bad(&e.to_string()))?;
|
||||
let folder = fs::canonicalize(base.join(typed)).map_err(|e| bad(&e.to_string()))?;
|
||||
let root = fs::canonicalize(SEED_ROOT).map_err(|e| bad(&e.to_string()))?;
|
||||
if !folder.starts_with(&root) || !folder.is_dir() {
|
||||
return Err(bad("not a folder under supabase/seed/anatomical-sheets"));
|
||||
}
|
||||
Ok(folder)
|
||||
}
|
||||
|
||||
/// Manifest, then every drawing, all verified before any request is made.
|
||||
fn load_local(folder: &Path) -> Result<Vec<Local>, UploadError> {
|
||||
let manifest = read_confined(&folder.join("assets.json"), folder, MAX_MANIFEST_BYTES)?;
|
||||
let text = String::from_utf8(manifest)
|
||||
.map_err(|_| refuse(UploadKind::Manifest, "assets.json is not UTF-8"))?;
|
||||
parse_manifest(&text)?
|
||||
.into_iter()
|
||||
.map(|entry| {
|
||||
let path = folder.join("assets").join(&entry.file);
|
||||
let bytes = read_confined(&path, folder, MAX_ASSET_BYTES)?;
|
||||
verify_asset(&entry, &bytes)?;
|
||||
Ok(Local { entry, bytes })
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn required_env(name: &str) -> Result<String, UploadError> {
|
||||
env::var(name)
|
||||
.ok()
|
||||
.filter(|value| !value.is_empty())
|
||||
.ok_or_else(|| refuse(UploadKind::Destination, format!("{name} is required")))
|
||||
}
|
||||
|
||||
/// The DEV address counts only if it is one of this machine's own unicast addresses: a socket
|
||||
/// binds to it, and connecting it to that address (no packet is sent) keeps it as the local
|
||||
/// end. A subnet broadcast binds but refuses the connect (no `SO_BROADCAST`). Otherwise the
|
||||
/// DEV address is ignored and only loopback stays allowed.
|
||||
fn own_address(host: &str) -> bool {
|
||||
let Ok(ip) = host.parse::<std::net::IpAddr>() else {
|
||||
return false;
|
||||
};
|
||||
// A multicast group binds and connects like an own address on Linux: never unicast.
|
||||
if ip.is_multicast() {
|
||||
return false;
|
||||
}
|
||||
let Ok(socket) = std::net::UdpSocket::bind((ip, 0)) else {
|
||||
return false;
|
||||
};
|
||||
socket.connect((ip, 9)).is_ok() && socket.local_addr().is_ok_and(|a| a.ip() == ip)
|
||||
}
|
||||
|
||||
fn connect() -> Result<Storage, UploadError> {
|
||||
let dev_host = env::var("FAUNOSTEO_DEV_HOST")
|
||||
.ok()
|
||||
.filter(|h| own_address(h));
|
||||
let destination = check_destination(&required_env("SUPABASE_URL")?, dev_host.as_deref())?;
|
||||
let key = Secret::new(required_env("SUPABASE_SERVICE_ROLE_KEY")?);
|
||||
Ok(Storage::new(&destination, key))
|
||||
}
|
||||
|
||||
/// What the destination holds for each entry and the steps that follow; refuses on the first
|
||||
/// entry that cannot be planned, so nothing is written for a run that cannot complete.
|
||||
fn plan_all(storage: &Storage, locals: &[Local]) -> Result<Vec<(Plan, ObjectState)>, UploadError> {
|
||||
locals
|
||||
.iter()
|
||||
.map(|local| {
|
||||
let object = storage.object_state(&local.entry)?;
|
||||
let row = storage.row_state(&local.entry)?;
|
||||
Ok((plan_entry(&local.entry, &object, &row)?, object))
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn describe(local: &Local, plan: Plan, check: bool) -> String {
|
||||
let steps: Vec<&str> = match (plan.upload, plan.confirm) {
|
||||
(false, false) => vec!["object verified, row confirmed, nothing to do"],
|
||||
(true, true) => vec!["upload", "confirm"],
|
||||
(true, false) => vec!["upload"],
|
||||
(false, true) => vec!["object verified", "confirm"],
|
||||
};
|
||||
let verb = if check { "would" } else { "will" };
|
||||
format!(
|
||||
"{} {}: {verb} {}",
|
||||
local.entry.file,
|
||||
local.entry.object_key(),
|
||||
steps.join(", ")
|
||||
)
|
||||
}
|
||||
|
||||
/// Uploads (if absent), verifies the object by reading it back, then confirms (if pending).
|
||||
fn apply(storage: &Storage, local: &Local, plan: Plan) -> Result<(), UploadError> {
|
||||
if plan.upload {
|
||||
storage.upload(&local.entry, &local.bytes)?;
|
||||
}
|
||||
if storage.object_state(&local.entry)? != ObjectState::Verified {
|
||||
return Err(refuse(
|
||||
UploadKind::State,
|
||||
format!(
|
||||
"{}: the object is not the expected bytes after upload",
|
||||
local.entry.file
|
||||
),
|
||||
));
|
||||
}
|
||||
if plan.confirm {
|
||||
storage.confirm(&local.entry)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn run(args: &[String]) -> Result<(), UploadError> {
|
||||
let (check, folder) = match args {
|
||||
[flag, folder] if flag == "--check" => (true, folder),
|
||||
[folder] if folder != "--check" => (false, folder),
|
||||
_ => {
|
||||
return Err(refuse(
|
||||
UploadKind::Content,
|
||||
"usage: upload-sheet-assets [--check] <seed folder>",
|
||||
));
|
||||
}
|
||||
};
|
||||
let locals = load_local(&resolve_folder(folder)?)?;
|
||||
let storage = connect()?;
|
||||
storage.handshake()?;
|
||||
let plans = plan_all(&storage, &locals)?;
|
||||
for (local, (plan, _)) in locals.iter().zip(&plans) {
|
||||
println!("{}", describe(local, *plan, check));
|
||||
}
|
||||
if check {
|
||||
return Ok(());
|
||||
}
|
||||
for (local, (plan, _)) in locals.iter().zip(&plans) {
|
||||
apply(&storage, local, *plan)?;
|
||||
println!("{} done", local.entry.file);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let args: Option<Vec<String>> = env::args_os()
|
||||
.skip(1)
|
||||
.map(|a| a.into_string().ok())
|
||||
.collect();
|
||||
let Some(args) = args else {
|
||||
eprintln!("an argument is not valid UTF-8");
|
||||
process::exit(2);
|
||||
};
|
||||
match run(&args) {
|
||||
Ok(()) => println!("ok"),
|
||||
Err(error) => {
|
||||
eprintln!("refused: {error}");
|
||||
process::exit(if error.kind == UploadKind::Network {
|
||||
3
|
||||
} else {
|
||||
1
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::own_address;
|
||||
|
||||
/// `inet` and `brd` addresses of this host, as `ip -4 -o addr` prints them.
|
||||
fn host_addresses() -> (Vec<String>, Vec<String>) {
|
||||
let listing = std::process::Command::new("ip")
|
||||
.args(["-4", "-o", "addr"])
|
||||
.output()
|
||||
.map(|output| String::from_utf8_lossy(&output.stdout).into_owned())
|
||||
.unwrap_or_default();
|
||||
let field = |line: &str, name: &str| {
|
||||
let mut words = line.split_whitespace();
|
||||
words.find(|w| *w == name)?;
|
||||
Some(words.next()?.split('/').next()?.to_owned())
|
||||
};
|
||||
let lines: Vec<&str> = listing.lines().collect();
|
||||
let unicast = lines.iter().filter_map(|l| field(l, "inet")).collect();
|
||||
let broadcast = lines.iter().filter_map(|l| field(l, "brd")).collect();
|
||||
(unicast, broadcast)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_an_own_unicast_address_counts_as_the_dev_address() {
|
||||
assert!(own_address("127.0.0.1"));
|
||||
for refused in [
|
||||
"255.255.255.255",
|
||||
"224.0.0.1",
|
||||
"239.255.255.250",
|
||||
"203.0.113.7",
|
||||
"not-an-ip",
|
||||
"",
|
||||
] {
|
||||
assert!(!own_address(refused), "{refused}");
|
||||
}
|
||||
let (unicast, broadcast) = host_addresses();
|
||||
for address in &unicast {
|
||||
assert!(own_address(address), "{address}");
|
||||
}
|
||||
for address in &broadcast {
|
||||
assert!(!own_address(address), "broadcast {address}");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
//! The destination, over HTTP: Storage objects and the asset rows of a local Supabase stack,
|
||||
//! with the service key. Every status is classified into the closed `UploadKind`s; the key is
|
||||
//! held in a `Secret` and goes in two request headers, nowhere else.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use faunosteo_core::curation::ids::sha256_hex;
|
||||
use faunosteo_core::curation::upload::{
|
||||
AssetEntry, BUCKET, Destination, MAX_ASSET_BYTES, ObjectState, RowState, UploadError,
|
||||
UploadKind, refuse,
|
||||
};
|
||||
use serde_json::{Value, json};
|
||||
use ureq::Agent;
|
||||
|
||||
const MAX_BODY: u64 = 1024 * 1024;
|
||||
/// One try and three retries.
|
||||
const ATTEMPTS: u32 = 4;
|
||||
|
||||
/// A key that never prints: `Debug` hides it and there is no `Display`.
|
||||
pub struct Secret(String);
|
||||
|
||||
impl Secret {
|
||||
pub fn new(text: String) -> Self {
|
||||
Self(text)
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for Secret {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("Secret(<redacted>)")
|
||||
}
|
||||
}
|
||||
|
||||
/// What an HTTP exchange gave back: the status and the (capped) body.
|
||||
struct Reply {
|
||||
status: u16,
|
||||
body: Vec<u8>,
|
||||
}
|
||||
|
||||
pub struct Storage {
|
||||
agent: Agent,
|
||||
origin: String,
|
||||
key: Secret,
|
||||
}
|
||||
|
||||
impl Storage {
|
||||
pub fn new(destination: &Destination, key: Secret) -> Self {
|
||||
let agent: Agent = Agent::config_builder()
|
||||
.timeout_global(Some(Duration::from_secs(30)))
|
||||
.timeout_connect(Some(Duration::from_secs(5)))
|
||||
.max_redirects(0)
|
||||
// The service key never goes through a proxy picked up from the environment.
|
||||
.proxy(None)
|
||||
.http_status_as_error(false)
|
||||
.user_agent("faunosteo-upload-sheet-assets")
|
||||
.build()
|
||||
.into();
|
||||
Self {
|
||||
agent,
|
||||
origin: destination.origin().to_owned(),
|
||||
key,
|
||||
}
|
||||
}
|
||||
|
||||
/// One exchange, retried on a transport error or a 5xx only (a refusal, an oversized body
|
||||
/// and a malformed request are final). The request is built again on each attempt.
|
||||
fn exchange(
|
||||
&self,
|
||||
limit: u64,
|
||||
send: impl Fn(&Agent, &Self) -> Result<ureq::http::Response<ureq::Body>, ureq::Error>,
|
||||
) -> Result<Reply, UploadError> {
|
||||
let mut last = refuse(UploadKind::Network, "no attempt made");
|
||||
for attempt in 1..=ATTEMPTS {
|
||||
if attempt > 1 {
|
||||
std::thread::sleep(Duration::from_millis(500 * u64::from(attempt)));
|
||||
}
|
||||
match send(&self.agent, self).and_then(|mut response| read_reply(&mut response, limit))
|
||||
{
|
||||
Ok(reply) if reply.status < 500 => return Ok(reply),
|
||||
Ok(reply) => last = refuse(UploadKind::Network, format!("HTTP {}", reply.status)),
|
||||
Err(error) if is_transient(&error) => {
|
||||
last = refuse(UploadKind::Network, error.to_string());
|
||||
}
|
||||
Err(error) => return Err(final_error(&error)),
|
||||
}
|
||||
}
|
||||
Err(last)
|
||||
}
|
||||
|
||||
fn authorised<B>(&self, request: ureq::RequestBuilder<B>) -> ureq::RequestBuilder<B> {
|
||||
request
|
||||
.header("apikey", &self.key.0)
|
||||
.header("Authorization", &format!("Bearer {}", self.key.0))
|
||||
}
|
||||
|
||||
fn object_url(&self, entry: &AssetEntry) -> String {
|
||||
format!(
|
||||
"{}/storage/v1/object/{BUCKET}/{}",
|
||||
self.origin,
|
||||
entry.object_key()
|
||||
)
|
||||
}
|
||||
|
||||
/// The destination is the local Faunosteo stack: the sheet bucket exists and is private,
|
||||
/// and the key is accepted. Nothing is written.
|
||||
pub fn handshake(&self) -> Result<(), UploadError> {
|
||||
let url = format!("{}/storage/v1/bucket/{BUCKET}", self.origin);
|
||||
let reply = self.exchange(MAX_BODY, |agent, me| me.authorised(agent.get(&url)).call())?;
|
||||
check_access(&reply)?;
|
||||
let bucket: Value = serde_json::from_slice(&reply.body)
|
||||
.map_err(|_| refuse(UploadKind::Handshake, "the bucket answer is not JSON"))?;
|
||||
let name = bucket.get("name").and_then(Value::as_str);
|
||||
let public = bucket.get("public").and_then(Value::as_bool);
|
||||
if reply.status != 200 || name != Some(BUCKET) || public != Some(false) {
|
||||
return Err(refuse(
|
||||
UploadKind::Handshake,
|
||||
format!("{BUCKET} is missing, or not a private bucket"),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Absent, or present with bytes that hash to the key, or present with other bytes.
|
||||
pub fn object_state(&self, entry: &AssetEntry) -> Result<ObjectState, UploadError> {
|
||||
let url = self.object_url(entry);
|
||||
let limit = MAX_ASSET_BYTES as u64 + 1;
|
||||
let reply = self.exchange(limit, |agent, me| me.authorised(agent.get(&url)).call())?;
|
||||
if is_not_found(&reply) {
|
||||
return Ok(ObjectState::Absent);
|
||||
}
|
||||
check_access(&reply)?;
|
||||
expect_status(&reply, 200, "object read")?;
|
||||
let found = sha256_hex(&reply.body);
|
||||
Ok(if found == entry.sha256 {
|
||||
ObjectState::Verified
|
||||
} else {
|
||||
ObjectState::Different(found)
|
||||
})
|
||||
}
|
||||
|
||||
/// Creates the object; an existing key is never replaced (`x-upsert: false`). Returns
|
||||
/// whether this call created it.
|
||||
pub fn upload(&self, entry: &AssetEntry, bytes: &[u8]) -> Result<bool, UploadError> {
|
||||
let url = self.object_url(entry);
|
||||
let reply = self.exchange(MAX_BODY, |agent, me| {
|
||||
me.authorised(agent.post(&url))
|
||||
.header("Content-Type", "image/svg+xml")
|
||||
.header("x-upsert", "false")
|
||||
.send(bytes)
|
||||
})?;
|
||||
check_access(&reply)?;
|
||||
if is_duplicate(&reply) {
|
||||
return Ok(false);
|
||||
}
|
||||
expect_status(&reply, 200, "upload")?;
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// The row of the asset: absent, another hash, pending or confirmed.
|
||||
pub fn row_state(&self, entry: &AssetEntry) -> Result<RowState, UploadError> {
|
||||
let url = format!(
|
||||
"{}/rest/v1/anatomical_sheet_assets?id=eq.{}&select=sha256,uploaded_at",
|
||||
self.origin, entry.svg_asset_id
|
||||
);
|
||||
let reply = self.exchange(MAX_BODY, |agent, me| me.authorised(agent.get(&url)).call())?;
|
||||
check_access(&reply)?;
|
||||
expect_status(&reply, 200, "row read")?;
|
||||
let rows: Vec<Value> = serde_json::from_slice(&reply.body)
|
||||
.map_err(|_| refuse(UploadKind::Handshake, "the row answer is not a JSON list"))?;
|
||||
Ok(match rows.first() {
|
||||
None => RowState::Unknown,
|
||||
Some(row) => row_state_of(row, &entry.sha256),
|
||||
})
|
||||
}
|
||||
|
||||
/// Marks the asset bytes as present (idempotent on the server, service role only).
|
||||
pub fn confirm(&self, entry: &AssetEntry) -> Result<(), UploadError> {
|
||||
let url = format!("{}/rest/v1/rpc/confirm_sheet_asset_upload", self.origin);
|
||||
let body = json!({ "asset_id": entry.svg_asset_id });
|
||||
let reply = self.exchange(MAX_BODY, |agent, me| {
|
||||
me.authorised(agent.post(&url)).send_json(&body)
|
||||
})?;
|
||||
check_access(&reply)?;
|
||||
expect_status(&reply, 200, "confirm")
|
||||
}
|
||||
}
|
||||
|
||||
/// The connection could not be made or broke: worth another attempt.
|
||||
fn is_transient(error: &ureq::Error) -> bool {
|
||||
matches!(
|
||||
error,
|
||||
ureq::Error::Io(_)
|
||||
| ureq::Error::Timeout(_)
|
||||
| ureq::Error::HostNotFound
|
||||
| ureq::Error::ConnectionFailed
|
||||
)
|
||||
}
|
||||
|
||||
/// Not retried: a body over its cap is a destination holding something it should not (an
|
||||
/// object larger than any drawing); anything else is a request the tool cannot build or an
|
||||
/// answer it cannot read (a key a header refuses, a malformed HTTP answer: `Protocol`), the
|
||||
/// same on every attempt. The key itself is never part of the message.
|
||||
fn final_error(error: &ureq::Error) -> UploadError {
|
||||
match error {
|
||||
ureq::Error::BodyExceedsLimit(limit) => refuse(
|
||||
UploadKind::State,
|
||||
format!("the destination answered more than {limit} bytes"),
|
||||
),
|
||||
_ => refuse(
|
||||
UploadKind::Handshake,
|
||||
"the request failed and is not retried (a key a header refuses, a malformed answer)",
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
fn row_state_of(row: &Value, expected_sha256: &str) -> RowState {
|
||||
let found = row
|
||||
.get("sha256")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or_default();
|
||||
if found != expected_sha256 {
|
||||
return RowState::OtherHash(found.to_owned());
|
||||
}
|
||||
match row.get("uploaded_at") {
|
||||
Some(Value::Null) | None => RowState::Pending,
|
||||
Some(_) => RowState::Confirmed,
|
||||
}
|
||||
}
|
||||
|
||||
fn read_reply(
|
||||
response: &mut ureq::http::Response<ureq::Body>,
|
||||
limit: u64,
|
||||
) -> Result<Reply, ureq::Error> {
|
||||
let body = response
|
||||
.body_mut()
|
||||
.with_config()
|
||||
.limit(limit)
|
||||
.read_to_vec()?;
|
||||
Ok(Reply {
|
||||
status: response.status().as_u16(),
|
||||
body,
|
||||
})
|
||||
}
|
||||
|
||||
/// The `statusCode` of a Storage error body (a string, "404" or "409" here).
|
||||
fn storage_code(reply: &Reply) -> Option<String> {
|
||||
let body: Value = serde_json::from_slice(&reply.body).ok()?;
|
||||
body.get("statusCode")?.as_str().map(str::to_owned)
|
||||
}
|
||||
|
||||
/// Storage answers a missing object with HTTP 400 and `statusCode` "404" in this image family.
|
||||
fn is_not_found(reply: &Reply) -> bool {
|
||||
reply.status == 404 || (reply.status == 400 && storage_code(reply).as_deref() == Some("404"))
|
||||
}
|
||||
|
||||
/// A key that already exists answers 409, or 400 with `statusCode` "409" ("Duplicate").
|
||||
fn is_duplicate(reply: &Reply) -> bool {
|
||||
reply.status == 409 || (reply.status == 400 && storage_code(reply).as_deref() == Some("409"))
|
||||
}
|
||||
|
||||
fn check_access(reply: &Reply) -> Result<(), UploadError> {
|
||||
if matches!(reply.status, 401 | 403) {
|
||||
return Err(refuse(
|
||||
UploadKind::Access,
|
||||
format!("HTTP {}: the destination refused the key", reply.status),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn expect_status(reply: &Reply, wanted: u16, what: &str) -> Result<(), UploadError> {
|
||||
if reply.status == wanted {
|
||||
return Ok(());
|
||||
}
|
||||
// The server's free text is not echoed: only the step and the status.
|
||||
Err(refuse(
|
||||
UploadKind::Handshake,
|
||||
format!("{what}: unexpected HTTP {}", reply.status),
|
||||
))
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
//! The drawing of one view exactly as the application renders it
|
||||
//! (`packages/anatomy/src/curation/app-review.ts`): the cleaned asset framed on the view's box
|
||||
//! plus the annotation layer generated from the document, by the R3 layer the editor canvas
|
||||
//! uses. Written under `draft/review/app-<view>.svg` so the markers and the framing can be
|
||||
//! compared outside the browser.
|
||||
//!
|
||||
//! Declared difference: the R3 owner decision (2026-10-15) gives a trigger without a name of
|
||||
//! its own the opened view's name in each language; the TypeScript renderer wrote the French
|
||||
//! name twice (`svg_compare::TriggerNames`).
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use super::drawing::{CleanedDrawing, drawing_text};
|
||||
use super::targets::fail;
|
||||
use crate::anatomy::layer::{find_view, render_layer, view_box_of};
|
||||
use crate::anatomy::svg::Node;
|
||||
use crate::curation::PipelineError;
|
||||
use crate::domain::sheet::AnatomicalSheetDocument;
|
||||
|
||||
pub fn render_app_review_drawing(
|
||||
drawing: &CleanedDrawing,
|
||||
document: &AnatomicalSheetDocument,
|
||||
view_id: &str,
|
||||
) -> Result<String, PipelineError> {
|
||||
let view =
|
||||
find_view(document, view_id).ok_or_else(|| fail(format!("unknown view \"{view_id}\"")))?;
|
||||
let mut root = drawing.root.clone();
|
||||
if let Some(view_box) = view_box_of(view) {
|
||||
root.set_attr("viewBox", view_box);
|
||||
}
|
||||
root.remove_attr("width");
|
||||
root.remove_attr("height");
|
||||
let layer = render_layer(document, view_id, &HashSet::new())
|
||||
.map_err(|_| fail(format!("unknown view \"{view_id}\"")))?;
|
||||
root.children.push(Node::Element(layer));
|
||||
drawing_text(&drawing.prolog, &root)
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
//! Applies a changeset on a baseline document (`packages/anatomy/src/curation/apply.ts`),
|
||||
//! deterministically and with a record per elementary edit. The references file replaces the
|
||||
//! document registry first (it is the source of truth). Any op that cannot apply exactly as
|
||||
//! written, and any integrity error left at the end, refuses: a curated document is never
|
||||
//! produced from a partially applied changeset.
|
||||
//!
|
||||
//! Review rule: a level B or C change never leaves what it touched (findings, finding
|
||||
//! categories, annotations: every item a set, add, cite, rename or move resolved to, whole
|
||||
//! added subtrees and subtree scopes included) as inherited or reviewed. After its ops, the
|
||||
//! applier sets `review_status = 'to_confirm'` on each one and records that edit under the
|
||||
//! same change id. An explicit 'reviewed' in such a change refuses. Level A is untouched by
|
||||
//! the rule.
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
use super::ops::{apply_op, assert_declared};
|
||||
use super::records::{
|
||||
ChangeRecord, CurationResult, RecordParts, Recorder, charge, strip_marks, touch_order,
|
||||
};
|
||||
use super::schema::{Change, ChangeSummary, Op, SetOp};
|
||||
use super::shape::assert_same_shape;
|
||||
use super::targets::{
|
||||
Path, Step, TargetKind, all_findings, at, fail, item_at_mut, list_at, top_collection_path,
|
||||
};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::domain::sheet::{
|
||||
AnatomicalSheetDocument, Reference, SheetIssue, SheetIssueSeverity, validate_sheet_document,
|
||||
};
|
||||
|
||||
/// A finding or an annotation a change touched: its touch order, its path, its target.
|
||||
struct Touched {
|
||||
order: u64,
|
||||
path: Path,
|
||||
target: String,
|
||||
}
|
||||
|
||||
/// Findings and annotations of the document the change touched, in first-touch order.
|
||||
fn touched_reviewable(doc: &Value) -> Result<Vec<Touched>, PipelineError> {
|
||||
let mut candidates: Vec<(Path, String)> = all_findings(doc)?
|
||||
.into_iter()
|
||||
.map(|entry| (entry.path, entry.target))
|
||||
.collect();
|
||||
let annotations = top_collection_path(TargetKind::Annotation);
|
||||
for (index, annotation) in list_at(doc, &annotations)?.iter().enumerate() {
|
||||
let mut path = annotations.clone();
|
||||
path.push(Step::Index(index));
|
||||
let id = super::records::js_string(annotation.get("id"));
|
||||
candidates.push((path, format!("annotation:{id}")));
|
||||
}
|
||||
let mut touched = Vec::new();
|
||||
for (path, target) in candidates {
|
||||
let Some(order) = at(doc, &path)?.as_object().and_then(touch_order) else {
|
||||
continue;
|
||||
};
|
||||
touched.push(Touched {
|
||||
order,
|
||||
path,
|
||||
target,
|
||||
});
|
||||
}
|
||||
touched.sort_by_key(|item| item.order);
|
||||
Ok(touched)
|
||||
}
|
||||
|
||||
/// Marks every touched item still in the document 'to_confirm'.
|
||||
fn mark_to_confirm(
|
||||
doc: &mut Value,
|
||||
change: &ChangeSummary,
|
||||
records: &mut Vec<ChangeRecord>,
|
||||
spent: &mut usize,
|
||||
) -> Result<(), PipelineError> {
|
||||
for touched in touched_reviewable(doc)? {
|
||||
let item = item_at_mut(doc, &touched.path)?;
|
||||
let status = item.get("review_status").cloned();
|
||||
if status == Some(Value::from("to_confirm")) {
|
||||
continue;
|
||||
}
|
||||
let before = match status {
|
||||
None | Some(Value::Null) => Value::from("inherited"),
|
||||
Some(value) => value,
|
||||
};
|
||||
item.insert("review_status".to_owned(), Value::from("to_confirm"));
|
||||
let op = Op::Set(SetOp {
|
||||
target: touched.target.clone(),
|
||||
field: "review_status".to_owned(),
|
||||
value: Value::from("to_confirm"),
|
||||
scope: None,
|
||||
});
|
||||
let after = Value::from("to_confirm");
|
||||
let parts = RecordParts {
|
||||
change,
|
||||
op: &op,
|
||||
target: &touched.target,
|
||||
field: Some("review_status"),
|
||||
before: &before,
|
||||
after: &after,
|
||||
};
|
||||
charge(spent, parts)?;
|
||||
records.push(ChangeRecord {
|
||||
change: change.clone(),
|
||||
op,
|
||||
target: touched.target,
|
||||
field: Some("review_status".to_owned()),
|
||||
before,
|
||||
after,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn apply_change(
|
||||
doc: &mut Value,
|
||||
change: &Change,
|
||||
records: &mut Vec<ChangeRecord>,
|
||||
spent: &mut usize,
|
||||
) -> Result<(), PipelineError> {
|
||||
let summary = change.summary();
|
||||
assert_declared(doc, &change.reference_ids)
|
||||
.map_err(|error| fail(format!("{}: {error}", change.id)))?;
|
||||
let mut next_touch = 0;
|
||||
for (index, op) in change.ops.iter().enumerate() {
|
||||
let mut recorder = Recorder::new(&summary, op, records, (&mut next_touch, &mut *spent));
|
||||
apply_op(doc, &mut recorder)
|
||||
.and_then(|()| assert_depth(doc))
|
||||
.map_err(|error| fail(format!("{} op {index}: {error}", change.id)))?;
|
||||
}
|
||||
if change.level.needs_confirmation() {
|
||||
mark_to_confirm(doc, &summary, records, spent)
|
||||
.map_err(|error| fail(format!("{}: {error}", change.id)))?;
|
||||
}
|
||||
strip_marks(doc);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Nesting a valid sheet document never approaches (the equine document has 10 levels): a
|
||||
/// resource bound checked after every op, so repeated adds, moves or deep values cannot grow
|
||||
/// the document until a recursive walk overflows the stack.
|
||||
pub const MAX_DOCUMENT_DEPTH: usize = 64;
|
||||
|
||||
/// Values a valid sheet document never approaches (the equine document holds 6,374): a
|
||||
/// resource bound checked with the depth, so the document, and the work done after each op,
|
||||
/// cannot grow without limit.
|
||||
pub const MAX_DOCUMENT_VALUES: usize = 200_000;
|
||||
|
||||
/// The nesting and the number of values of `value`, walked with an explicit stack (no
|
||||
/// recursion); the walk stops counting past the value bound.
|
||||
fn measure(value: &Value) -> (usize, usize) {
|
||||
let (mut deepest, mut count) = (0, 0);
|
||||
let mut pending = vec![(value, 1)];
|
||||
while let Some((value, depth)) = pending.pop() {
|
||||
count += 1;
|
||||
if count > MAX_DOCUMENT_VALUES {
|
||||
break;
|
||||
}
|
||||
let children: Box<dyn Iterator<Item = &Value>> = match value {
|
||||
Value::Object(map) => Box::new(map.values()),
|
||||
Value::Array(items) => Box::new(items.iter()),
|
||||
_ => continue,
|
||||
};
|
||||
deepest = deepest.max(depth);
|
||||
pending.extend(children.map(|child| (child, depth + 1)));
|
||||
}
|
||||
(deepest, count)
|
||||
}
|
||||
|
||||
fn assert_depth(doc: &Value) -> Result<(), PipelineError> {
|
||||
let (depth, count) = measure(doc);
|
||||
if count > MAX_DOCUMENT_VALUES {
|
||||
return Err(fail(format!(
|
||||
"the document holds more than {MAX_DOCUMENT_VALUES} values"
|
||||
)));
|
||||
}
|
||||
if depth > MAX_DOCUMENT_DEPTH {
|
||||
return Err(fail(format!(
|
||||
"the document nests more than {MAX_DOCUMENT_DEPTH} levels"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn describe(issues: &[SheetIssue]) -> String {
|
||||
issues
|
||||
.iter()
|
||||
.map(|issue| format!("{}: {}", issue.path, issue.message))
|
||||
.collect::<Vec<_>>()
|
||||
.join("; ")
|
||||
}
|
||||
|
||||
/// Shape (the strict R2 contract) then integrity check of a curated document. Refuses on any
|
||||
/// shape issue or integrity error; returns the warnings, which are allowed.
|
||||
pub fn curated_issues(doc: &Value) -> Result<Vec<SheetIssue>, PipelineError> {
|
||||
let parsed: AnatomicalSheetDocument = serde_json::from_value(doc.clone())
|
||||
.map_err(|error| fail(format!("curated document is not shape-valid: {error}")))?;
|
||||
assert_same_shape("curated document is not shape-valid", doc, &parsed)?;
|
||||
let issues = validate_sheet_document(&parsed);
|
||||
let errors: Vec<SheetIssue> = issues
|
||||
.iter()
|
||||
.filter(|issue| issue.severity == SheetIssueSeverity::Error)
|
||||
.cloned()
|
||||
.collect();
|
||||
if !errors.is_empty() {
|
||||
return Err(fail(format!(
|
||||
"curated document has integrity errors: {}",
|
||||
describe(&errors)
|
||||
)));
|
||||
}
|
||||
Ok(issues)
|
||||
}
|
||||
|
||||
pub fn apply_changeset(
|
||||
baseline: &Value,
|
||||
references: &[Reference],
|
||||
changes: &[Change],
|
||||
) -> Result<CurationResult, PipelineError> {
|
||||
let mut doc = baseline.clone();
|
||||
let registry =
|
||||
serde_json::to_value(references).map_err(|error| fail(format!("references: {error}")))?;
|
||||
doc.as_object_mut()
|
||||
.ok_or_else(|| fail("the baseline is not an object"))?
|
||||
.insert("references".to_owned(), registry);
|
||||
let mut records = Vec::new();
|
||||
let mut spent = 0;
|
||||
for change in changes {
|
||||
apply_change(&mut doc, change, &mut records, &mut spent)?;
|
||||
}
|
||||
curated_issues(&doc)?;
|
||||
Ok(CurationResult {
|
||||
document: doc,
|
||||
records,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
//! French CHANGELOG of the working revision (`packages/anatomy/src/curation/changelog.ts`):
|
||||
//! volumes before / after, then one table per change level with every elementary edit. Pure
|
||||
//! formatting of the records, deterministic (no dates). The text returned here is the raw
|
||||
//! report; `markdown::format_markdown` aligns its tables as Prettier did.
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
use super::labels::{LEVELS, kind_label, level_label};
|
||||
use super::migration_map::MigrationMapStats;
|
||||
use super::records::ChangeRecord;
|
||||
use super::schema::ChangeLevel;
|
||||
use super::stats::{DocumentStats, stat_rows};
|
||||
use super::targets::fail;
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::markdown::{Cell, assert_table_bound, code, table};
|
||||
use crate::curation::report_guard::assert_authored;
|
||||
use crate::json::to_js_compact;
|
||||
|
||||
/// The longest value text a table cell shows, in UTF-16 units (`text.length`).
|
||||
const MAX_VALUE_LENGTH: usize = 120;
|
||||
|
||||
/// `isLabel`: a non-empty object whose keys are `fr` / `en` with string values.
|
||||
fn is_label(value: &Value) -> bool {
|
||||
value.as_object().is_some_and(|map| {
|
||||
!map.is_empty()
|
||||
&& map
|
||||
.iter()
|
||||
.all(|(key, value)| (key == "fr" || key == "en") && value.is_string())
|
||||
})
|
||||
}
|
||||
|
||||
/// `compactValue`: labels as text, other non-strings as JSON, cut to 120 UTF-16 units with an
|
||||
/// ellipsis. A cut inside a surrogate pair leaves a lone surrogate, which the file write turned
|
||||
/// into U+FFFD: `from_utf16_lossy` gives the same text.
|
||||
pub fn compact_value(value: &Value) -> Result<String, PipelineError> {
|
||||
let text = match value {
|
||||
Value::String(text) => text.clone(),
|
||||
Value::Object(map) if is_label(value) => map
|
||||
.values()
|
||||
.filter_map(Value::as_str)
|
||||
.collect::<Vec<_>>()
|
||||
.join(" / "),
|
||||
other => to_js_compact(other).map_err(|error| fail(format!("value: {error}")))?,
|
||||
};
|
||||
let units: Vec<u16> = text.encode_utf16().collect();
|
||||
if units.len() <= MAX_VALUE_LENGTH {
|
||||
return Ok(text);
|
||||
}
|
||||
Ok(format!(
|
||||
"{}…",
|
||||
String::from_utf16_lossy(&units[..MAX_VALUE_LENGTH - 1])
|
||||
))
|
||||
}
|
||||
|
||||
fn authored(text: String) -> Result<String, PipelineError> {
|
||||
assert_authored(&text)?;
|
||||
Ok(text)
|
||||
}
|
||||
|
||||
/// `recordSubject`: the target, with its field when there is one.
|
||||
pub fn record_subject(record: &ChangeRecord) -> String {
|
||||
match &record.field {
|
||||
None => record.target.clone(),
|
||||
Some(field) => format!("{}.{field}", record.target),
|
||||
}
|
||||
}
|
||||
|
||||
fn record_row(record: &ChangeRecord) -> Result<Vec<Cell>, PipelineError> {
|
||||
// Authored texts reach the cells raw: no backtick (code spans are the generator's).
|
||||
assert_authored(&record.change.title.fr)?;
|
||||
assert_authored(&record.change.rationale)?;
|
||||
let change = &record.change;
|
||||
let references: Vec<String> = change
|
||||
.reference_ids
|
||||
.iter()
|
||||
.map(|id| code(Some(id)))
|
||||
.collect();
|
||||
let references = if references.is_empty() {
|
||||
"-".to_owned()
|
||||
} else {
|
||||
references.join(", ")
|
||||
};
|
||||
let confirm = if change.level.needs_confirmation() {
|
||||
"oui"
|
||||
} else {
|
||||
"non"
|
||||
};
|
||||
Ok(vec![
|
||||
Cell::from(format!("{} {}", code(Some(&change.id)), change.title.fr)),
|
||||
Cell::from(code(Some(&record_subject(record)))),
|
||||
Cell::from(authored(compact_value(&record.before)?)?),
|
||||
Cell::from(authored(compact_value(&record.after)?)?),
|
||||
Cell::from(kind_label(change.kind)),
|
||||
Cell::from(change.rationale.as_str()),
|
||||
Cell::from(references),
|
||||
Cell::from(confirm),
|
||||
])
|
||||
}
|
||||
|
||||
fn level_section(
|
||||
level: ChangeLevel,
|
||||
records: &[ChangeRecord],
|
||||
spent: &mut usize,
|
||||
) -> Result<String, PipelineError> {
|
||||
let rows = records
|
||||
.iter()
|
||||
.filter(|record| record.change.level == level)
|
||||
.map(record_row)
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
let head = [
|
||||
"Changement",
|
||||
"Objet",
|
||||
"Avant",
|
||||
"Après",
|
||||
"Nature",
|
||||
"Justification",
|
||||
"Références",
|
||||
"Confirmation requise",
|
||||
];
|
||||
// The rows are bounded through the records' weight; the alignment of the three level
|
||||
// tables together is checked here, before each raw table is joined.
|
||||
assert_table_bound(&head, &rows, spent)?;
|
||||
let body = if rows.is_empty() {
|
||||
"Aucun changement de ce niveau.".to_owned()
|
||||
} else {
|
||||
table(&head, &rows)
|
||||
};
|
||||
Ok(format!("## {}\n\n{body}", level_label(level)))
|
||||
}
|
||||
|
||||
fn comparison(before: &DocumentStats, after: &DocumentStats) -> String {
|
||||
let after_rows = stat_rows(after);
|
||||
let rows: Vec<Vec<Cell>> = stat_rows(before)
|
||||
.into_iter()
|
||||
.map(|(label, value)| {
|
||||
let later = after_rows
|
||||
.iter()
|
||||
.find(|(other, _)| *other == label)
|
||||
.map_or(0, |(_, value)| *value);
|
||||
vec![Cell::from(label), Cell::from(value), Cell::from(later)]
|
||||
})
|
||||
.collect();
|
||||
format!(
|
||||
"## Comparaison quantitative\n\n{}",
|
||||
table(&["Indicateur", "Avant", "Après"], &rows)
|
||||
)
|
||||
}
|
||||
|
||||
fn intro() -> String {
|
||||
[
|
||||
"# Journal des changements : révision de travail",
|
||||
"",
|
||||
"Généré par `pnpm import:equine` : import historique (`legacy-import/`) puis changements \
|
||||
rédigés (`curation/changes.json`). Ne pas modifier à la main : relancer la commande.",
|
||||
"",
|
||||
"Niveaux : A = erreur démontrée, B = amélioration étayée, C = point incertain. Les \
|
||||
niveaux B et C demandent une confirmation professionnelle.",
|
||||
]
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
/// Fate of the legacy finding nodes a Report v2 may cite (`report-migration-map.json`).
|
||||
fn migration(stats: &MigrationMapStats) -> String {
|
||||
let row = |label: &str, value: usize| vec![Cell::from(label), Cell::from(value.to_string())];
|
||||
let rows = vec![
|
||||
row("Nœuds du catalogue historique", stats.nodes),
|
||||
row("Chemin identique", stats.identical),
|
||||
row(
|
||||
"Chemin renommé ou déplacé (suivi des renommages)",
|
||||
stats.remapped,
|
||||
),
|
||||
row(
|
||||
"Relogé sous une autre catégorie (correspondance unique)",
|
||||
stats.rehomed,
|
||||
),
|
||||
row(
|
||||
"Retiré : le bilan garde le constat avec ses libellés figés",
|
||||
stats.removed,
|
||||
),
|
||||
];
|
||||
format!(
|
||||
"## Migration des bilans v2\n\n{}",
|
||||
table(&["Nœuds", "Nombre"], &rows)
|
||||
)
|
||||
}
|
||||
|
||||
/// `renderChangelog`: the raw text, before the table alignment.
|
||||
pub fn render_changelog(
|
||||
records: &[ChangeRecord],
|
||||
before: &DocumentStats,
|
||||
after: &DocumentStats,
|
||||
migration_stats: &MigrationMapStats,
|
||||
) -> Result<String, PipelineError> {
|
||||
let mut sections = vec![
|
||||
intro(),
|
||||
comparison(before, after),
|
||||
migration(migration_stats),
|
||||
];
|
||||
let mut spent = 0;
|
||||
for level in LEVELS {
|
||||
sections.push(level_section(level, records, &mut spent)?);
|
||||
}
|
||||
Ok(format!("{}\n", sections.join("\n\n")))
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
//! The frozen cleaned drawing as the review files read it: the text of
|
||||
//! `draft/assets/horse.svg` through the core's one SVG reader (refused if the reader removes
|
||||
//! anything: the product's drawings come out untouched), and its prolog (the bytes before the
|
||||
//! root element: the XML declaration and its line break), kept verbatim in front of every
|
||||
//! drawing the pipeline writes, as xmldom's serialiser kept it.
|
||||
|
||||
use crate::anatomy::svg::serialize::{is_xml_text, serialize};
|
||||
use crate::anatomy::svg::{Element, Node, read_svg};
|
||||
use crate::curation::PipelineError;
|
||||
|
||||
pub struct CleanedDrawing {
|
||||
pub prolog: String,
|
||||
pub root: Element,
|
||||
}
|
||||
|
||||
fn refuse(why: impl std::fmt::Display) -> PipelineError {
|
||||
PipelineError::Source(format!("horse.svg: {why}"))
|
||||
}
|
||||
|
||||
/// Reads the cleaned drawing (UTF-8, refused otherwise, as `TextDecoder` fatal was).
|
||||
pub fn read_cleaned_drawing(bytes: &[u8]) -> Result<CleanedDrawing, PipelineError> {
|
||||
let text = std::str::from_utf8(bytes).map_err(|_| refuse("invalid UTF-8"))?;
|
||||
let sanitized = read_svg(text).map_err(|error| refuse(error.reason()))?;
|
||||
if !sanitized.removed.is_empty() {
|
||||
return Err(refuse(format!(
|
||||
"the reader removed {:?}",
|
||||
sanitized.removed
|
||||
)));
|
||||
}
|
||||
let document = roxmltree::Document::parse(text).map_err(refuse)?;
|
||||
let start = document.root_element().range().start;
|
||||
Ok(CleanedDrawing {
|
||||
prolog: text[..start].to_owned(),
|
||||
root: sanitized.root,
|
||||
})
|
||||
}
|
||||
|
||||
/// A value or text the drawing cannot carry as the TypeScript wrote it: a code point XML 1.0
|
||||
/// refuses, or a carriage return (xmldom wrote it raw in text, a parser reads it back as a line
|
||||
/// feed; the core writes ` `, which a parser keeps). Walked with an explicit stack.
|
||||
fn first_unwritable(root: &Element) -> Option<String> {
|
||||
let bad = |text: &str| !is_xml_text(text) || text.contains('\r');
|
||||
let mut pending = vec![root];
|
||||
while let Some(element) = pending.pop() {
|
||||
if let Some(attr) = element.attrs.iter().find(|a| bad(&a.value)) {
|
||||
return Some(attr.value.clone());
|
||||
}
|
||||
for child in &element.children {
|
||||
match child {
|
||||
Node::Element(child) => pending.push(child),
|
||||
Node::Text(text) if bad(text) => return Some(text.clone()),
|
||||
Node::Text(_) => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// The file text of a drawing built on the cleaned root; refused when a label, an id or a path
|
||||
/// holds what the drawing cannot carry (`first_unwritable`).
|
||||
pub fn drawing_text(prolog: &str, root: &Element) -> Result<String, PipelineError> {
|
||||
if let Some(text) = first_unwritable(root) {
|
||||
return Err(PipelineError::Guard(format!(
|
||||
"a review drawing cannot carry {text:?}"
|
||||
)));
|
||||
}
|
||||
Ok(format!("{prolog}{}", serialize(root)))
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
//! Fields a `set` may write, per kind of item (`packages/anatomy/src/curation/fields.ts`). Ids
|
||||
//! are left out on purpose: an id changes through `rename`, which also rewrites every
|
||||
//! reference to it. Unlisted field → the op fails instead of writing a stray key.
|
||||
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use super::targets::{Item, ParsedTarget, TargetKind, fail};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::text::js_insert;
|
||||
|
||||
/// Item kind of the settable table; a finding is a category or a node.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum FieldKind {
|
||||
Document,
|
||||
Region,
|
||||
Reference,
|
||||
Category,
|
||||
View,
|
||||
Annotation,
|
||||
Zone,
|
||||
FindingCategory,
|
||||
FindingNode,
|
||||
}
|
||||
|
||||
impl FieldKind {
|
||||
pub fn of(target: &ParsedTarget, is_category: Option<bool>) -> Self {
|
||||
match target.kind {
|
||||
TargetKind::Document => Self::Document,
|
||||
TargetKind::Region => Self::Region,
|
||||
TargetKind::Reference => Self::Reference,
|
||||
TargetKind::Category => Self::Category,
|
||||
TargetKind::View => Self::View,
|
||||
TargetKind::Annotation => Self::Annotation,
|
||||
TargetKind::Zone => Self::Zone,
|
||||
TargetKind::Finding => {
|
||||
if is_category.unwrap_or(target.ids.len() == 2) {
|
||||
Self::FindingCategory
|
||||
} else {
|
||||
Self::FindingNode
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn settable(self) -> &'static [&'static str] {
|
||||
match self {
|
||||
Self::Document => &["name", "description", "scope_disclaimer", "attribution"],
|
||||
Self::Region => &["name", "count", "count_range"],
|
||||
Self::Reference => &["type", "citation", "url", "doi"],
|
||||
Self::Category => &["name", "color"],
|
||||
Self::View => &["name", "svg_asset_id", "view_box", "parent_view_id"],
|
||||
Self::Annotation => &[
|
||||
"name",
|
||||
"kind",
|
||||
"view_id",
|
||||
"category_id",
|
||||
"laterality",
|
||||
"geometry",
|
||||
"review_status",
|
||||
"finding_zone_id",
|
||||
"vertebral_region_code",
|
||||
"opens_view_id",
|
||||
"color",
|
||||
],
|
||||
Self::Zone => &["name"],
|
||||
Self::FindingCategory => &[
|
||||
"kind",
|
||||
"name",
|
||||
"description",
|
||||
"reference_ids",
|
||||
"review_status",
|
||||
],
|
||||
Self::FindingNode => &[
|
||||
"name",
|
||||
"description",
|
||||
"requires_vet_referral",
|
||||
"reference_ids",
|
||||
"review_status",
|
||||
],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// More segments than any settable path needs (the deepest has two): a resource bound, so a
|
||||
/// field path cannot nest the document without limit.
|
||||
pub const MAX_FIELD_SEGMENTS: usize = 8;
|
||||
|
||||
/// The segments of a dot path. A segment starting with NUL is reserved for the applier's
|
||||
/// touch marks (`records::TOUCH_MARK`): written, it would be stripped with them and the
|
||||
/// author's value lost, so it is refused (the TypeScript refused such a key at the gate).
|
||||
pub fn split_field(field: &str) -> Result<Vec<&str>, PipelineError> {
|
||||
let path: Vec<&str> = field.split('.').collect();
|
||||
if path.iter().any(|key| key.is_empty()) {
|
||||
return Err(fail(format!("malformed field \"{field}\"")));
|
||||
}
|
||||
if path.len() > MAX_FIELD_SEGMENTS {
|
||||
return Err(fail(format!(
|
||||
"field \"{field}\" has more than {MAX_FIELD_SEGMENTS} segments"
|
||||
)));
|
||||
}
|
||||
if let Some(key) = path.iter().find(|key| key.starts_with('\0')) {
|
||||
return Err(fail(format!("reserved key {key:?} in field \"{field}\"")));
|
||||
}
|
||||
// A segment reaches the reports' code spans through the record: the alphabet of every
|
||||
// settable path (`name.fr`, `view_box.min_x`, `count_range.min`), nothing a Markdown parser
|
||||
// could pair with the text around the span (a `]` closed an author's `[` there).
|
||||
let id_alphabet = |key: &str| {
|
||||
key.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || "_-".contains(c))
|
||||
};
|
||||
if !path.iter().all(|key| id_alphabet(key)) {
|
||||
return Err(fail(format!(
|
||||
"field {field:?} holds a segment outside [A-Za-z0-9_-]"
|
||||
)));
|
||||
}
|
||||
// A JavaScript assignment through `__proto__` never wrote a key: it set a prototype,
|
||||
// polluted `Object.prototype` or did nothing. No faithful reading exists, so it is refused.
|
||||
if path.contains(&"__proto__") {
|
||||
return Err(fail(format!(
|
||||
"reserved key \"__proto__\" in field \"{field}\""
|
||||
)));
|
||||
}
|
||||
Ok(path)
|
||||
}
|
||||
|
||||
pub fn is_settable(kind: FieldKind, field: &str) -> Result<bool, PipelineError> {
|
||||
Ok(kind.settable().contains(&split_field(field)?[0]))
|
||||
}
|
||||
|
||||
/// The value at a dot path; `None` where JavaScript read `undefined`.
|
||||
pub fn read_field(item: &Item, field: &str) -> Result<Option<Value>, PipelineError> {
|
||||
let mut value: Option<&Map<String, Value>> = Some(item);
|
||||
let mut found: Option<&Value> = None;
|
||||
for key in split_field(field)? {
|
||||
let Some(holder) = value else {
|
||||
return Ok(None);
|
||||
};
|
||||
found = holder.get(key);
|
||||
value = found.and_then(Value::as_object);
|
||||
}
|
||||
Ok(found.cloned())
|
||||
}
|
||||
|
||||
/// Writes a deep copy of value; missing intermediate objects are created (a non-object value
|
||||
/// on the way is replaced, as `holder[key] = {}` did).
|
||||
pub fn write_field(item: &mut Item, field: &str, value: &Value) -> Result<(), PipelineError> {
|
||||
let path = split_field(field)?;
|
||||
let (last, holders) = path.split_last().ok_or_else(|| fail("empty field"))?;
|
||||
let mut holder = item;
|
||||
for key in holders {
|
||||
if !holder.get(*key).is_some_and(Value::is_object) {
|
||||
js_insert(holder, key, Value::Object(Map::new()));
|
||||
}
|
||||
holder = holder
|
||||
.get_mut(*key)
|
||||
.and_then(Value::as_object_mut)
|
||||
.ok_or_else(|| fail("field holder vanished"))?;
|
||||
}
|
||||
js_insert(holder, last, value.clone());
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
//! French labels shared by the CHANGELOG and the review report
|
||||
//! (`packages/anatomy/src/curation/labels.ts`).
|
||||
|
||||
use super::schema::{ChangeKind, ChangeLevel, ChangePriority};
|
||||
|
||||
pub fn kind_label(kind: ChangeKind) -> &'static str {
|
||||
match kind {
|
||||
ChangeKind::Correction => "Correction",
|
||||
ChangeKind::Terminology => "Terminologie",
|
||||
ChangeKind::Translation => "Traduction",
|
||||
ChangeKind::Restructure => "Réorganisation",
|
||||
ChangeKind::Enrichment => "Ajout",
|
||||
ChangeKind::Reference => "Référence",
|
||||
ChangeKind::Geometry => "Géométrie",
|
||||
ChangeKind::Removal => "Suppression",
|
||||
ChangeKind::Technical => "Technique",
|
||||
}
|
||||
}
|
||||
|
||||
pub fn level_label(level: ChangeLevel) -> &'static str {
|
||||
match level {
|
||||
ChangeLevel::A => "Niveau A : erreurs démontrées",
|
||||
ChangeLevel::B => "Niveau B : améliorations étayées (confirmation requise)",
|
||||
ChangeLevel::C => "Niveau C : points incertains (question au professionnel)",
|
||||
}
|
||||
}
|
||||
|
||||
pub fn priority_label(priority: ChangePriority) -> &'static str {
|
||||
match priority {
|
||||
ChangePriority::High => "haute",
|
||||
ChangePriority::Medium => "moyenne",
|
||||
ChangePriority::Low => "basse",
|
||||
}
|
||||
}
|
||||
|
||||
pub const LEVELS: [ChangeLevel; 3] = [ChangeLevel::A, ChangeLevel::B, ChangeLevel::C];
|
||||
|
||||
/// The letter of a level, as the reports print it.
|
||||
pub fn level_letter(level: ChangeLevel) -> &'static str {
|
||||
match level {
|
||||
ChangeLevel::A => "A",
|
||||
ChangeLevel::B => "B",
|
||||
ChangeLevel::C => "C",
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,310 @@
|
||||
//! Report migration map (`packages/anatomy/src/curation/migration-map.ts`): where every
|
||||
//! identifier a Report v2 may carry lands in the curated draft. Annotations come from the
|
||||
//! legacy id map (then follow annotation renames), finding paths from replaying the applier's
|
||||
//! rename, move and remove records over every node of the legacy baseline. A node removed with
|
||||
//! its category and re-added under another category at the same relative place (the sphenoid
|
||||
//! restructure) is re-homed when the match is unique; anything else stays null = "removed",
|
||||
//! and the report keeps the finding with its snapshots.
|
||||
|
||||
use serde::Serialize;
|
||||
use serde_json::{Map, Value, json};
|
||||
|
||||
use super::records::ChangeRecord;
|
||||
use super::schema::Op;
|
||||
use super::targets::{TargetKind, all_findings, list_at, top_collection_path};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::text::js_object;
|
||||
|
||||
const FINDING: &str = "finding:";
|
||||
|
||||
/// The sheet a report references: id, database version (null for a bundled draft), document
|
||||
/// fingerprint (the `SheetRef` of Report v3, camelCase).
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct SheetRef {
|
||||
pub sheet_id: String,
|
||||
pub version_id: Option<String>,
|
||||
pub document_sha256: String,
|
||||
}
|
||||
|
||||
pub struct MigrationMapInput<'a> {
|
||||
pub baseline: &'a Value,
|
||||
pub final_document: &'a Value,
|
||||
pub legacy_id_map: &'a Map<String, Value>,
|
||||
pub records: &'a [ChangeRecord],
|
||||
pub sheet: &'a SheetRef,
|
||||
pub main_view_id: &'a str,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||
pub struct MigrationMapStats {
|
||||
pub nodes: usize,
|
||||
pub identical: usize,
|
||||
pub remapped: usize,
|
||||
pub rehomed: usize,
|
||||
pub removed: usize,
|
||||
}
|
||||
|
||||
struct PathEntry {
|
||||
legacy: String,
|
||||
current: Option<String>,
|
||||
rehomed: bool,
|
||||
}
|
||||
|
||||
/// `target.slice(FINDING.length)`: the first eight UTF-16 units dropped, whatever they are.
|
||||
fn strip(target: &str) -> String {
|
||||
let mut units: Vec<u16> = target.encode_utf16().collect();
|
||||
units.drain(..units.len().min(FINDING.len()));
|
||||
String::from_utf16_lossy(&units)
|
||||
}
|
||||
|
||||
fn last_segment(path: &str) -> &str {
|
||||
path.rsplit('/').next().unwrap_or(path)
|
||||
}
|
||||
|
||||
fn with_last_segment(path: &str, id: &str) -> String {
|
||||
match path.rfind('/') {
|
||||
None => id.to_owned(),
|
||||
Some(cut) => format!("{}{id}", &path[..=cut]),
|
||||
}
|
||||
}
|
||||
|
||||
fn replace_prefix(entries: &mut [PathEntry], from: &str, to: Option<&str>) {
|
||||
for entry in entries {
|
||||
let Some(current) = &entry.current else {
|
||||
continue;
|
||||
};
|
||||
if current == from {
|
||||
entry.current = to.map(str::to_owned);
|
||||
} else if let Some(rest) = current.strip_prefix(&format!("{from}/")) {
|
||||
entry.current = to.map(|to| format!("{to}/{rest}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Legacy id → current id, for the annotations and the views (`Map` in the TypeScript).
|
||||
struct Ids {
|
||||
annotations: Vec<(String, String)>,
|
||||
views: Vec<(String, String)>,
|
||||
}
|
||||
|
||||
fn id_entries(map: &Map<String, Value>, key: &str) -> Vec<(String, String)> {
|
||||
map.get(key)
|
||||
.and_then(Value::as_object)
|
||||
.map(|object| {
|
||||
object
|
||||
.iter()
|
||||
.map(|(legacy, current)| {
|
||||
(
|
||||
legacy.clone(),
|
||||
current.as_str().unwrap_or_default().to_owned(),
|
||||
)
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
fn apply_record(entries: &mut [PathEntry], ids: &mut Ids, record: &ChangeRecord) {
|
||||
let target = record.target.as_str();
|
||||
let (rename_to, move_to) = match &record.op {
|
||||
Op::Rename(op) => (Some(op.id.as_str()), None),
|
||||
Op::Remove(_) => (None, None),
|
||||
Op::Move(op) => (None, Some(op.to.as_str())),
|
||||
_ => return,
|
||||
};
|
||||
if let Some(from) = target.strip_prefix(FINDING) {
|
||||
match &record.op {
|
||||
Op::Rename(op) => replace_prefix(entries, from, Some(&with_last_segment(from, &op.id))),
|
||||
Op::Remove(_) => replace_prefix(entries, from, None),
|
||||
_ => {
|
||||
let to = format!(
|
||||
"{}/{}",
|
||||
strip(move_to.unwrap_or_default()),
|
||||
last_segment(from)
|
||||
);
|
||||
replace_prefix(entries, from, Some(&to));
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
// `target.split(':', 2)`: the kind, then what lies before the next colon.
|
||||
let mut parts = target.splitn(3, ':');
|
||||
let kind = parts.next().unwrap_or_default();
|
||||
let id = parts.next().unwrap_or_default();
|
||||
if kind == "zone" {
|
||||
for entry in entries {
|
||||
let Some(current) = &entry.current else {
|
||||
continue;
|
||||
};
|
||||
let Some(rest) = current.strip_prefix(&format!("{id}/")) else {
|
||||
continue;
|
||||
};
|
||||
entry.current = match &record.op {
|
||||
Op::Remove(_) => None,
|
||||
Op::Rename(op) => Some(format!("{}/{rest}", op.id)),
|
||||
_ => Some(format!("{id}/{rest}")),
|
||||
};
|
||||
}
|
||||
return;
|
||||
}
|
||||
if let Some(to) = rename_to
|
||||
&& (kind == "annotation" || kind == "view")
|
||||
{
|
||||
let map = if kind == "annotation" {
|
||||
&mut ids.annotations
|
||||
} else {
|
||||
&mut ids.views
|
||||
};
|
||||
for (_, current) in map.iter_mut() {
|
||||
if current == id {
|
||||
*current = to.to_owned();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `zone/<any category>/<rest>` present exactly once in the final document.
|
||||
fn rehome(entry: &mut PathEntry, final_paths: &[String]) {
|
||||
let mut segments = entry.legacy.split('/');
|
||||
let zone = segments.next().unwrap_or_default();
|
||||
let _category = segments.next();
|
||||
let rest: Vec<&str> = segments.collect();
|
||||
if rest.is_empty() {
|
||||
return;
|
||||
}
|
||||
let tail = format!("/{}", rest.join("/"));
|
||||
let matches: Vec<&String> = final_paths
|
||||
.iter()
|
||||
.filter(|path| {
|
||||
path.starts_with(&format!("{zone}/"))
|
||||
&& path.ends_with(&tail)
|
||||
&& path.split('/').count() == rest.len() + 2
|
||||
})
|
||||
.collect();
|
||||
if let [only] = matches[..] {
|
||||
entry.current = Some(only.clone());
|
||||
entry.rehomed = true;
|
||||
}
|
||||
}
|
||||
|
||||
fn text(value: Option<&Value>) -> String {
|
||||
value.and_then(Value::as_str).unwrap_or_default().to_owned()
|
||||
}
|
||||
|
||||
/// Annotation id → `{ fr, en }` of its name, empty strings where the name is missing.
|
||||
fn annotation_names(doc: &Value) -> Result<Vec<(String, Value)>, PipelineError> {
|
||||
let annotations = list_at(doc, &top_collection_path(TargetKind::Annotation))?;
|
||||
Ok(annotations
|
||||
.iter()
|
||||
.map(|annotation| {
|
||||
let name = annotation.get("name");
|
||||
let id = super::records::js_string(annotation.get("id"));
|
||||
let fr = text(name.and_then(|n| n.get("fr")));
|
||||
let en = text(name.and_then(|n| n.get("en")));
|
||||
(id, json!({ "fr": fr, "en": en }))
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
fn initial_entries(baseline: &Value) -> Result<Vec<PathEntry>, PipelineError> {
|
||||
Ok(all_findings(baseline)?
|
||||
.into_iter()
|
||||
.map(|entry| {
|
||||
let legacy = strip(&entry.target);
|
||||
PathEntry {
|
||||
current: Some(legacy.clone()),
|
||||
legacy,
|
||||
rehomed: false,
|
||||
}
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
fn stats_of(entries: &[PathEntry]) -> MigrationMapStats {
|
||||
MigrationMapStats {
|
||||
nodes: entries.len(),
|
||||
identical: entries
|
||||
.iter()
|
||||
.filter(|e| e.current.as_deref() == Some(e.legacy.as_str()))
|
||||
.count(),
|
||||
remapped: entries
|
||||
.iter()
|
||||
.filter(|e| {
|
||||
e.current.is_some() && e.current.as_deref() != Some(e.legacy.as_str()) && !e.rehomed
|
||||
})
|
||||
.count(),
|
||||
rehomed: entries.iter().filter(|e| e.rehomed).count(),
|
||||
removed: entries.iter().filter(|e| e.current.is_none()).count(),
|
||||
}
|
||||
}
|
||||
|
||||
fn pairs(entries: &[(String, String)]) -> Map<String, Value> {
|
||||
js_object(
|
||||
entries
|
||||
.iter()
|
||||
.map(|(legacy, current)| (legacy.clone(), Value::from(current.as_str()))),
|
||||
)
|
||||
}
|
||||
|
||||
/// The map (in the TypeScript key order) and its stats.
|
||||
pub fn build_report_migration_map(
|
||||
input: &MigrationMapInput<'_>,
|
||||
) -> Result<(Value, MigrationMapStats), PipelineError> {
|
||||
let mut entries = initial_entries(input.baseline)?;
|
||||
let mut ids = Ids {
|
||||
annotations: id_entries(input.legacy_id_map, "annotations"),
|
||||
views: id_entries(input.legacy_id_map, "views"),
|
||||
};
|
||||
for record in input.records {
|
||||
apply_record(&mut entries, &mut ids, record);
|
||||
}
|
||||
// `new Set(...)`: a path met twice counts once.
|
||||
let mut final_paths: Vec<String> = Vec::new();
|
||||
for entry in all_findings(input.final_document)? {
|
||||
let path = strip(&entry.target);
|
||||
if !final_paths.contains(&path) {
|
||||
final_paths.push(path);
|
||||
}
|
||||
}
|
||||
for entry in &mut entries {
|
||||
if entry
|
||||
.current
|
||||
.as_ref()
|
||||
.is_some_and(|current| !final_paths.contains(current))
|
||||
{
|
||||
entry.current = None;
|
||||
}
|
||||
if entry.current.is_none() {
|
||||
rehome(entry, &final_paths);
|
||||
}
|
||||
}
|
||||
let legacy_names = annotation_names(input.baseline)?;
|
||||
let import_ids = id_entries(input.legacy_id_map, "annotations");
|
||||
let annotations = js_object(ids.annotations.iter().map(|(legacy, final_id)| {
|
||||
let imported = import_ids
|
||||
.iter()
|
||||
.find(|(key, _)| key == legacy)
|
||||
.map(|(_, id)| id.as_str());
|
||||
let name = imported
|
||||
.and_then(|id| legacy_names.iter().find(|(key, _)| key == id))
|
||||
.map_or_else(|| json!({ "fr": "", "en": "" }), |(_, name)| name.clone());
|
||||
(legacy.clone(), json!({ "id": final_id, "name": name }))
|
||||
}));
|
||||
let finding_paths = js_object(entries.iter().map(|entry| {
|
||||
let current = entry
|
||||
.current
|
||||
.as_ref()
|
||||
.map_or(Value::Null, |c| Value::from(c.as_str()));
|
||||
(entry.legacy.clone(), current)
|
||||
}));
|
||||
let map = json!({
|
||||
"schema_version": 1,
|
||||
"sheet": input.sheet,
|
||||
"main_view_id": input.main_view_id,
|
||||
"views": pairs(&ids.views),
|
||||
"annotations": annotations,
|
||||
"finding_paths": finding_paths,
|
||||
});
|
||||
Ok((map, stats_of(&entries)))
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
//! The draft side of the pipeline (`packages/anatomy/src/curation`): the authored changeset
|
||||
//! applied on the legacy baseline, the report migration map, the document statistics and the
|
||||
//! seed SQL (R5.3), the CHANGELOG, the review report and the review drawings (R5.4). Parity with
|
||||
//! the TypeScript is proven by `tests/curation_draft.rs` and `tests/curation_review.rs` against
|
||||
//! the committed `draft/` folder and the `oracles/r5` files.
|
||||
|
||||
pub mod app_review;
|
||||
pub mod apply;
|
||||
pub mod changelog;
|
||||
pub mod drawing;
|
||||
pub mod fields;
|
||||
pub mod labels;
|
||||
pub mod migration_map;
|
||||
pub mod ops;
|
||||
pub mod overlay;
|
||||
pub mod pipeline;
|
||||
pub mod records;
|
||||
pub mod rename;
|
||||
pub mod review_report;
|
||||
pub mod schema;
|
||||
pub mod seed;
|
||||
pub mod shape;
|
||||
pub mod stats;
|
||||
pub mod svg_compare;
|
||||
pub mod targets;
|
||||
|
||||
pub use apply::{apply_changeset, curated_issues};
|
||||
pub use pipeline::{
|
||||
CLEANED_ASSET_FILE, DRAFT_SEED_SQL_FILE, DraftInput, DraftResult, MIGRATION_MAP_FILE,
|
||||
SHEET_REF_FILE, build_equine_draft, cleaned_asset, parse_view_box, render_draft_artifacts,
|
||||
};
|
||||
pub use schema::{ASSET_CHANGE_ID, Change, Changeset, parse_changeset, parse_references};
|
||||
@@ -0,0 +1,466 @@
|
||||
//! The six op kinds of a change (`packages/anatomy/src/curation/ops.ts`). Each one resolves
|
||||
//! its target, edits the working document in place and records what it changed; anything it
|
||||
//! cannot do exactly as written refuses (the applier adds the change id and op index).
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
use super::fields::{FieldKind, is_settable, read_field, write_field};
|
||||
use super::records::{Recorder, js_deep_equal, js_string};
|
||||
use super::rename::{Rename, propagate_rename};
|
||||
use super::schema::{
|
||||
AddOp, CiteOp, DocumentCollection, MoveOp, Op, Position, RemoveOp, RenameOp, Scope, SetOp,
|
||||
};
|
||||
use super::targets::{
|
||||
Located, Path, Step, TargetKind, at, child_list, child_prefix, fail, finding_scope, has_key,
|
||||
id_text, item_at_mut, list_at, list_at_mut, locate, top_collection_path,
|
||||
};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::domain::js;
|
||||
|
||||
fn collection_kind(field: DocumentCollection) -> TargetKind {
|
||||
match field {
|
||||
DocumentCollection::References => TargetKind::Reference,
|
||||
DocumentCollection::Categories => TargetKind::Category,
|
||||
DocumentCollection::Views => TargetKind::View,
|
||||
DocumentCollection::Annotations => TargetKind::Annotation,
|
||||
DocumentCollection::FindingZones => TargetKind::Zone,
|
||||
DocumentCollection::Regions => TargetKind::Region,
|
||||
}
|
||||
}
|
||||
|
||||
fn collection_name(field: DocumentCollection) -> &'static str {
|
||||
match field {
|
||||
DocumentCollection::References => "references",
|
||||
DocumentCollection::Categories => "categories",
|
||||
DocumentCollection::Views => "views",
|
||||
DocumentCollection::Annotations => "annotations",
|
||||
DocumentCollection::FindingZones => "finding_zones",
|
||||
DocumentCollection::Regions => "regions",
|
||||
}
|
||||
}
|
||||
|
||||
/// No sibling but `excluded` may already carry the id.
|
||||
fn assert_free(
|
||||
list: &[Value],
|
||||
key: &str,
|
||||
id: &str,
|
||||
excluded: Option<usize>,
|
||||
) -> Result<(), PipelineError> {
|
||||
let taken = list
|
||||
.iter()
|
||||
.enumerate()
|
||||
.any(|(index, entry)| Some(index) != excluded && has_key(entry, key, id));
|
||||
if taken {
|
||||
return Err(fail(format!("\"{id}\" already exists among its siblings")));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn insertion_index(
|
||||
list: &[Value],
|
||||
key: &str,
|
||||
position: Option<&Position>,
|
||||
) -> Result<usize, PipelineError> {
|
||||
let Some(position) = position else {
|
||||
return Ok(list.len());
|
||||
};
|
||||
let (anchor, after) = match position {
|
||||
Position::Index(index) => {
|
||||
let index = usize::try_from(*index).unwrap_or(usize::MAX);
|
||||
if index > list.len() {
|
||||
return Err(fail(format!("position {index} out of range")));
|
||||
}
|
||||
return Ok(index);
|
||||
}
|
||||
Position::After { after } => (after, true),
|
||||
Position::Before { before } => (before, false),
|
||||
};
|
||||
let index = list
|
||||
.iter()
|
||||
.position(|entry| has_key(entry, key, anchor))
|
||||
.ok_or_else(|| fail(format!("unknown sibling \"{anchor}\"")))?;
|
||||
Ok(if after { index + 1 } else { index })
|
||||
}
|
||||
|
||||
fn unknown_field(op: &SetOp) -> PipelineError {
|
||||
fail(format!(
|
||||
"unknown field \"{}\" on \"{}\"",
|
||||
op.field, op.target
|
||||
))
|
||||
}
|
||||
|
||||
/// A B or C change leaves findings to the professional: no 'reviewed'.
|
||||
fn assert_not_reviewed(recorder: &Recorder<'_>, op: &SetOp) -> Result<(), PipelineError> {
|
||||
if op.field != "review_status" || op.value != "reviewed" {
|
||||
return Ok(());
|
||||
}
|
||||
if recorder.change.level.needs_confirmation() {
|
||||
return Err(fail(format!(
|
||||
"level {:?} cannot set \"{}\" review_status to \"reviewed\"",
|
||||
recorder.change.level, op.target
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn set_self(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
located: &Located,
|
||||
op: &SetOp,
|
||||
) -> Result<(), PipelineError> {
|
||||
if !is_settable(FieldKind::of(&located.target, None), &op.field)? {
|
||||
return Err(unknown_field(op));
|
||||
}
|
||||
let item = item_at_mut(doc, &located.item_path())?;
|
||||
recorder.touch(item);
|
||||
let before = read_field(item, &op.field)?;
|
||||
write_field(item, &op.field, &op.value)?;
|
||||
recorder.record(&op.target, Some(&op.field), before, Some(op.value.clone()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Target and descendants that carry the field; unchanged items are skipped.
|
||||
fn set_subtree(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
located: &Located,
|
||||
op: &SetOp,
|
||||
) -> Result<(), PipelineError> {
|
||||
if located.target.kind != TargetKind::Finding {
|
||||
return Err(fail(format!(
|
||||
"scope subtree needs a finding target, got \"{}\"",
|
||||
op.target
|
||||
)));
|
||||
}
|
||||
let mut entries = Vec::new();
|
||||
for entry in finding_scope(doc, located, true)? {
|
||||
let kind = FieldKind::of(&located.target, Some(entry.is_category));
|
||||
if is_settable(kind, &op.field)? {
|
||||
entries.push(entry);
|
||||
}
|
||||
}
|
||||
if entries.is_empty() {
|
||||
return Err(unknown_field(op));
|
||||
}
|
||||
for entry in &entries {
|
||||
recorder.touch(item_at_mut(doc, &entry.path)?);
|
||||
}
|
||||
for entry in &entries {
|
||||
let item = item_at_mut(doc, &entry.path)?;
|
||||
let before = read_field(item, &op.field)?;
|
||||
if before
|
||||
.as_ref()
|
||||
.is_some_and(|value| js_deep_equal(value, &op.value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
write_field(item, &op.field, &op.value)?;
|
||||
recorder.record(
|
||||
&entry.target,
|
||||
Some(&op.field),
|
||||
before,
|
||||
Some(op.value.clone()),
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn apply_set(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
op: &SetOp,
|
||||
) -> Result<(), PipelineError> {
|
||||
assert_not_reviewed(recorder, op)?;
|
||||
let located = locate(doc, &op.target)?;
|
||||
match op.scope {
|
||||
None | Some(Scope::OwnItem) => set_self(doc, recorder, &located, op),
|
||||
Some(Scope::Subtree) => set_subtree(doc, recorder, &located, op),
|
||||
}
|
||||
}
|
||||
|
||||
fn apply_rename(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
op: &RenameOp,
|
||||
) -> Result<(), PipelineError> {
|
||||
let located = locate(doc, &op.target)?;
|
||||
let Some(list) = &located.list else {
|
||||
return Err(fail("the document has no id to rename"));
|
||||
};
|
||||
let key = located.target.kind.key_field();
|
||||
assert_stable_id(Some(&Value::from(op.id.as_str())), key)?;
|
||||
assert_free(list_at(doc, list)?, key, &op.id, Some(located.index))?;
|
||||
let item = item_at_mut(doc, &located.item_path())?;
|
||||
let before = Value::Object(item.clone());
|
||||
let from = js_string(item.get(key));
|
||||
item.insert(key.to_owned(), Value::from(op.id.clone()));
|
||||
recorder.touch(item);
|
||||
let after = Value::Object(item.clone());
|
||||
recorder.record(&op.target, None, Some(before), Some(after))?;
|
||||
let rename = Rename {
|
||||
kind: located.target.kind,
|
||||
from,
|
||||
to: op.id.clone(),
|
||||
};
|
||||
propagate_rename(doc, recorder, &rename)
|
||||
}
|
||||
|
||||
struct Destination {
|
||||
list: Path,
|
||||
key: &'static str,
|
||||
prefix: String,
|
||||
}
|
||||
|
||||
fn add_destination(doc: &mut Value, op: &AddOp) -> Result<Destination, PipelineError> {
|
||||
let parent = locate(doc, &op.target)?;
|
||||
if parent.target.kind != TargetKind::Document {
|
||||
if let Some(field) = op.field {
|
||||
return Err(fail(format!(
|
||||
"field \"{}\" is only valid on document",
|
||||
collection_name(field)
|
||||
)));
|
||||
}
|
||||
return Ok(Destination {
|
||||
list: child_list(doc, &parent)?,
|
||||
key: "id",
|
||||
prefix: child_prefix(&parent),
|
||||
});
|
||||
}
|
||||
let Some(field) = op.field else {
|
||||
return Err(fail("add on document needs a field"));
|
||||
};
|
||||
let kind = collection_kind(field);
|
||||
let list = top_collection_path(kind);
|
||||
list_at(doc, &list)?;
|
||||
Ok(Destination {
|
||||
list,
|
||||
key: kind.key_field(),
|
||||
prefix: format!("{}:", kind.name()),
|
||||
})
|
||||
}
|
||||
|
||||
/// An id an op writes must already be what the final document requires (a stable id; a region
|
||||
/// code for a region): an id that passed only because a later op removed or renamed its item
|
||||
/// would still reach the records, and from them the reports' code spans (a line break there
|
||||
/// forged sections). Refused at the op (RUST.md §11.14).
|
||||
fn assert_stable_id(value: Option<&Value>, key: &str) -> Result<(), PipelineError> {
|
||||
let valid = match value.and_then(Value::as_str) {
|
||||
Some(id) if key == "code" => js::is_region_code(id),
|
||||
Some(id) => js::is_stable_id(id),
|
||||
None => false,
|
||||
};
|
||||
if !valid {
|
||||
return Err(fail(format!(
|
||||
"the {} {:?} is not a valid {}",
|
||||
key,
|
||||
value.map(js_string_value).unwrap_or_default(),
|
||||
if key == "code" {
|
||||
"region code"
|
||||
} else {
|
||||
"stable id"
|
||||
}
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn js_string_value(value: &Value) -> String {
|
||||
super::records::js_string(Some(value))
|
||||
}
|
||||
|
||||
/// A `children` or `categories` of an added value must be a list (a later add would otherwise
|
||||
/// replace the author's value by `[]`, as the TypeScript's `childrenOf` did), each entry an
|
||||
/// object with a stable id. The TypeScript met a null or a primitive there inconsistently (a
|
||||
/// TypeError when it walked it, silently skipped when a rename walked past it, refused by the
|
||||
/// gate when it stayed); the port refuses it at the add (RUST.md §11.12). Explicit stack.
|
||||
fn assert_object_children(value: &Value) -> Result<(), PipelineError> {
|
||||
let mut pending = vec![value];
|
||||
while let Some(item) = pending.pop() {
|
||||
let Some(map) = item.as_object() else {
|
||||
continue;
|
||||
};
|
||||
for key in ["children", "categories"] {
|
||||
let list = match map.get(key) {
|
||||
None => continue,
|
||||
Some(Value::Array(list)) => list,
|
||||
Some(_) => {
|
||||
return Err(fail(format!(
|
||||
"the added item holds a \"{key}\" that is not a list"
|
||||
)));
|
||||
}
|
||||
};
|
||||
if list.iter().any(|entry| !entry.is_object()) {
|
||||
return Err(fail(format!(
|
||||
"the added item holds a \"{key}\" entry that is not an object"
|
||||
)));
|
||||
}
|
||||
for entry in list {
|
||||
assert_stable_id(entry.get("id"), "id")?;
|
||||
}
|
||||
pending.extend(list);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn apply_add(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
op: &AddOp,
|
||||
) -> Result<(), PipelineError> {
|
||||
let Some(value) = op.value.as_object() else {
|
||||
return Err(fail("add needs an object value"));
|
||||
};
|
||||
assert_object_children(&op.value)?;
|
||||
let dest = add_destination(doc, op)?;
|
||||
let Some(id) = value.get(dest.key).and_then(Value::as_str) else {
|
||||
return Err(fail(format!(
|
||||
"the added item needs a string \"{}\"",
|
||||
dest.key
|
||||
)));
|
||||
};
|
||||
assert_stable_id(value.get(dest.key), dest.key)?;
|
||||
let list = list_at(doc, &dest.list)?;
|
||||
assert_free(list, dest.key, id, None)?;
|
||||
let index = insertion_index(list, dest.key, op.position.as_ref())?;
|
||||
list_at_mut(doc, &dest.list)?.insert(index, op.value.clone());
|
||||
let mut added = dest.list.clone();
|
||||
added.push(Step::Index(index));
|
||||
recorder.touch_tree(doc, &added)?;
|
||||
let target = format!("{}{id}", dest.prefix);
|
||||
recorder.record(&target, None, None, Some(op.value.clone()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn apply_remove(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
op: &RemoveOp,
|
||||
) -> Result<(), PipelineError> {
|
||||
let located = locate(doc, &op.target)?;
|
||||
let Some(list) = &located.list else {
|
||||
return Err(fail("the document cannot be removed"));
|
||||
};
|
||||
let item = list_at_mut(doc, list)?.remove(located.index);
|
||||
recorder.record(&op.target, None, Some(item), None)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `after` of a move record is the new target path of the item.
|
||||
fn apply_move(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
op: &MoveOp,
|
||||
) -> Result<(), PipelineError> {
|
||||
let source = locate(doc, &op.target)?;
|
||||
if source.target.kind != TargetKind::Finding {
|
||||
return Err(fail(format!("only findings move, got \"{}\"", op.target)));
|
||||
}
|
||||
if op.to == op.target || op.to.starts_with(&format!("{}/", op.target)) {
|
||||
return Err(fail(format!("cannot move \"{}\" into itself", op.target)));
|
||||
}
|
||||
let parent = locate(doc, &op.to)?;
|
||||
let list = child_list(doc, &parent)?;
|
||||
let id = id_text(at(doc, &source.item_path())?);
|
||||
let excluded = (Some(&list) == source.list.as_ref()).then_some(source.index);
|
||||
assert_free(list_at(doc, &list)?, "id", &id, excluded)?;
|
||||
let source_list = source
|
||||
.list
|
||||
.as_ref()
|
||||
.ok_or_else(|| fail("a finding has siblings"))?;
|
||||
let item = list_at_mut(doc, source_list)?.remove(source.index);
|
||||
// The removal may have shifted the destination's indices: resolve it again.
|
||||
let parent = locate(doc, &op.to)?;
|
||||
let list = child_list(doc, &parent)?;
|
||||
let index = insertion_index(list_at(doc, &list)?, "id", op.position.as_ref())?;
|
||||
list_at_mut(doc, &list)?.insert(index, item);
|
||||
let mut moved = list;
|
||||
moved.push(Step::Index(index));
|
||||
let item = item_at_mut(doc, &moved)?;
|
||||
recorder.touch(item);
|
||||
let before = Value::Object(item.clone());
|
||||
let after = format!("{}{id}", child_prefix(&parent));
|
||||
recorder.record(&op.target, None, Some(before), Some(Value::from(after)))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every cited id must be declared in the document references.
|
||||
pub fn assert_declared(doc: &Value, ids: &[String]) -> Result<(), PipelineError> {
|
||||
let declared: HashSet<&str> = list_at(doc, &top_collection_path(TargetKind::Reference))?
|
||||
.iter()
|
||||
.filter_map(|reference| reference.get("id").and_then(Value::as_str))
|
||||
.collect();
|
||||
let missing: Vec<&str> = ids
|
||||
.iter()
|
||||
.map(String::as_str)
|
||||
.filter(|id| !declared.contains(id))
|
||||
.collect();
|
||||
if !missing.is_empty() {
|
||||
return Err(fail(format!(
|
||||
"undeclared reference(s): {}",
|
||||
missing.join(", ")
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `[...new Set(ids)]`: first occurrences, in order.
|
||||
fn distinct(ids: &[String]) -> Vec<&String> {
|
||||
let mut seen: HashSet<&str> = HashSet::new();
|
||||
ids.iter().filter(|id| seen.insert(id.as_str())).collect()
|
||||
}
|
||||
|
||||
/// Appends citations in order, never twice on the same item.
|
||||
fn apply_cite(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
op: &CiteOp,
|
||||
) -> Result<(), PipelineError> {
|
||||
assert_declared(doc, &op.reference_ids)?;
|
||||
let located = locate(doc, &op.target)?;
|
||||
let scope = finding_scope(doc, &located, op.scope == Some(Scope::Subtree))?;
|
||||
for entry in &scope {
|
||||
recorder.touch(item_at_mut(doc, &entry.path)?);
|
||||
}
|
||||
for entry in &scope {
|
||||
let item = item_at_mut(doc, &entry.path)?;
|
||||
let before: Vec<Value> = item
|
||||
.get("reference_ids")
|
||||
.and_then(Value::as_array)
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
// `before.includes(id)`: only an equal string matches; hashed once per item.
|
||||
let existing: HashSet<&str> = before.iter().filter_map(Value::as_str).collect();
|
||||
let added: Vec<Value> = distinct(&op.reference_ids)
|
||||
.into_iter()
|
||||
.filter(|id| !existing.contains(id.as_str()))
|
||||
.map(|id| Value::from(id.clone()))
|
||||
.collect();
|
||||
if added.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let after: Vec<Value> = before.iter().cloned().chain(added).collect();
|
||||
item.insert("reference_ids".to_owned(), Value::Array(after.clone()));
|
||||
recorder.record(
|
||||
&entry.target,
|
||||
Some("reference_ids"),
|
||||
Some(Value::Array(before)),
|
||||
Some(Value::Array(after)),
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn apply_op(doc: &mut Value, recorder: &mut Recorder<'_>) -> Result<(), PipelineError> {
|
||||
match recorder.op {
|
||||
Op::Set(op) => apply_set(doc, recorder, op),
|
||||
Op::Rename(op) => apply_rename(doc, recorder, op),
|
||||
Op::Add(op) => apply_add(doc, recorder, op),
|
||||
Op::Remove(op) => apply_remove(doc, recorder, op),
|
||||
Op::Move(op) => apply_move(doc, recorder, op),
|
||||
Op::Cite(op) => apply_cite(doc, recorder, op),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,376 @@
|
||||
//! Before / after drawing of one view for the professional review
|
||||
//! (`packages/anatomy/src/curation/overlay.ts`): the cleaned anatomy, the curated annotations
|
||||
//! as an SVG layer, and the baseline position of every moved or removed annotation beside it
|
||||
//! (dashed). Pure data, deterministic, opens in any browser.
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
use super::drawing::{CleanedDrawing, drawing_text};
|
||||
use super::records::{js_deep_equal, js_string};
|
||||
use super::targets::fail;
|
||||
use crate::anatomy::svg::number::{is_js_space, js_number_of};
|
||||
use crate::anatomy::svg::{Element, Node};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::text::js_round;
|
||||
use crate::json::js_number;
|
||||
|
||||
pub const REVIEW_OVERLAY_DIR: &str = "review";
|
||||
const LAYER_ID: &str = "review-annotations";
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum Status {
|
||||
Unchanged,
|
||||
Moved,
|
||||
Added,
|
||||
Removed,
|
||||
}
|
||||
|
||||
impl Status {
|
||||
fn name(self) -> &'static str {
|
||||
match self {
|
||||
Self::Unchanged => "unchanged",
|
||||
Self::Moved => "moved",
|
||||
Self::Added => "added",
|
||||
Self::Removed => "removed",
|
||||
}
|
||||
}
|
||||
|
||||
/// The marker style, attribute by attribute in the TypeScript order.
|
||||
fn style(self) -> &'static [(&'static str, &'static str)] {
|
||||
match self {
|
||||
Self::Unchanged => &[
|
||||
("stroke", "#1d4ed8"),
|
||||
("fill", "#1d4ed8"),
|
||||
("fill-opacity", "0.12"),
|
||||
],
|
||||
Self::Moved | Self::Added => &[
|
||||
("stroke", "#15803d"),
|
||||
("fill", "#15803d"),
|
||||
("fill-opacity", "0.2"),
|
||||
],
|
||||
// fill-opacity rather than fill="none": some rasterizers drop unfilled shapes.
|
||||
Self::Removed => &[
|
||||
("stroke", "#b91c1c"),
|
||||
("fill", "#b91c1c"),
|
||||
("fill-opacity", "0.04"),
|
||||
("stroke-dasharray", "8 6"),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
fn stroke(self) -> &'static str {
|
||||
self.style()[0].1
|
||||
}
|
||||
}
|
||||
|
||||
struct Marker<'a> {
|
||||
id: String,
|
||||
geometry: &'a Value,
|
||||
status: Status,
|
||||
}
|
||||
|
||||
fn annotations(doc: &Value) -> &[Value] {
|
||||
doc.get("annotations")
|
||||
.and_then(Value::as_array)
|
||||
.map_or(&[], Vec::as_slice)
|
||||
}
|
||||
|
||||
/// Annotations of the view that carry a geometry (`withGeometry`).
|
||||
fn with_geometry<'a>(doc: &'a Value, view_id: &str) -> impl Iterator<Item = &'a Value> {
|
||||
annotations(doc).iter().filter(move |a| {
|
||||
a.get("view_id").and_then(Value::as_str) == Some(view_id)
|
||||
&& a.get("geometry").is_some_and(|g| !g.is_null())
|
||||
})
|
||||
}
|
||||
|
||||
/// `new Map(entries)`: the first position of an id, its last value.
|
||||
struct Previous<'a> {
|
||||
entries: Vec<(String, &'a Value)>,
|
||||
index: HashMap<String, usize>,
|
||||
}
|
||||
|
||||
fn previous_geometries<'a>(before: &'a Value, view_id: &str) -> Previous<'a> {
|
||||
let mut previous = Previous {
|
||||
entries: Vec::new(),
|
||||
index: HashMap::new(),
|
||||
};
|
||||
for annotation in with_geometry(before, view_id) {
|
||||
let id = js_string(annotation.get("id"));
|
||||
let geometry = &annotation["geometry"];
|
||||
match previous.index.get(&id) {
|
||||
Some(&at) => previous.entries[at].1 = geometry,
|
||||
None => {
|
||||
previous.index.insert(id.clone(), previous.entries.len());
|
||||
previous.entries.push((id, geometry));
|
||||
}
|
||||
}
|
||||
}
|
||||
previous
|
||||
}
|
||||
|
||||
/// `overlayMarkers`: curated annotations with their status, then the baseline shapes they left.
|
||||
fn overlay_markers<'a>(before: &'a Value, after: &'a Value, view_id: &str) -> Vec<Marker<'a>> {
|
||||
let previous = previous_geometries(before, view_id);
|
||||
let mut markers = Vec::new();
|
||||
for annotation in with_geometry(after, view_id) {
|
||||
let id = js_string(annotation.get("id"));
|
||||
let geometry = &annotation["geometry"];
|
||||
let old = previous.index.get(&id).map(|&at| previous.entries[at].1);
|
||||
let status = match old {
|
||||
None => Status::Added,
|
||||
Some(old) if js_deep_equal(old, geometry) => Status::Unchanged,
|
||||
Some(_) => Status::Moved,
|
||||
};
|
||||
markers.push(Marker {
|
||||
id: id.clone(),
|
||||
geometry,
|
||||
status,
|
||||
});
|
||||
if let (Status::Moved, Some(old)) = (status, old) {
|
||||
markers.push(Marker {
|
||||
id,
|
||||
geometry: old,
|
||||
status: Status::Removed,
|
||||
});
|
||||
}
|
||||
}
|
||||
let kept: HashSet<String> = annotations(after)
|
||||
.iter()
|
||||
.map(|a| js_string(a.get("id")))
|
||||
.collect();
|
||||
for (id, geometry) in previous.entries {
|
||||
if !kept.contains(&id) {
|
||||
markers.push(Marker {
|
||||
id,
|
||||
geometry,
|
||||
status: Status::Removed,
|
||||
});
|
||||
}
|
||||
}
|
||||
markers
|
||||
}
|
||||
|
||||
fn number(geometry: &Value, key: &str) -> f64 {
|
||||
geometry
|
||||
.get(key)
|
||||
.and_then(Value::as_f64)
|
||||
.unwrap_or(f64::NAN)
|
||||
}
|
||||
|
||||
/// `String(number)`, NaN and the infinities included.
|
||||
fn text_of(value: f64) -> String {
|
||||
match value {
|
||||
v if v.is_nan() => "NaN".to_owned(),
|
||||
f64::INFINITY => "Infinity".to_owned(),
|
||||
f64::NEG_INFINITY => "-Infinity".to_owned(),
|
||||
v => js_number(v),
|
||||
}
|
||||
}
|
||||
|
||||
fn points(geometry: &Value) -> Vec<(f64, f64)> {
|
||||
let list = geometry.get("points").and_then(Value::as_array);
|
||||
list.into_iter()
|
||||
.flatten()
|
||||
.map(|point| {
|
||||
let at = |i: usize| point.get(i).and_then(Value::as_f64).unwrap_or(f64::NAN);
|
||||
(at(0), at(1))
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn shape_element(geometry: &Value) -> Result<Element, PipelineError> {
|
||||
match geometry.get("type").and_then(Value::as_str) {
|
||||
Some("circle") => Ok(Element::svg(
|
||||
"circle",
|
||||
vec![
|
||||
("cx", text_of(number(geometry, "cx"))),
|
||||
("cy", text_of(number(geometry, "cy"))),
|
||||
("r", text_of(number(geometry, "r"))),
|
||||
],
|
||||
Vec::new(),
|
||||
)),
|
||||
Some("polygon") => {
|
||||
let list: Vec<String> = points(geometry)
|
||||
.into_iter()
|
||||
.map(|(x, y)| format!("{},{}", text_of(x), text_of(y)))
|
||||
.collect();
|
||||
Ok(Element::svg(
|
||||
"polygon",
|
||||
vec![("points", list.join(" "))],
|
||||
Vec::new(),
|
||||
))
|
||||
}
|
||||
Some("path") => Ok(Element::svg(
|
||||
"path",
|
||||
vec![("d", js_string(geometry.get("d")))],
|
||||
Vec::new(),
|
||||
)),
|
||||
_ => Err(fail("a geometry of an unknown type")),
|
||||
}
|
||||
}
|
||||
|
||||
fn label_anchor(geometry: &Value) -> Option<(f64, f64)> {
|
||||
match geometry.get("type").and_then(Value::as_str) {
|
||||
Some("circle") => Some((
|
||||
number(geometry, "cx"),
|
||||
number(geometry, "cy") - number(geometry, "r") - 4.0,
|
||||
)),
|
||||
Some("polygon") => {
|
||||
let list = points(geometry);
|
||||
let (sx, sy) = list
|
||||
.iter()
|
||||
.fold((0.0, 0.0), |(ax, ay), (x, y)| (ax + x, ay + y));
|
||||
let n = list.len() as f64;
|
||||
Some((sx / n, sy / n))
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn text_element(attrs: Vec<(&str, String)>, text: String) -> Element {
|
||||
Element::svg("text", attrs, vec![Node::Text(text)])
|
||||
}
|
||||
|
||||
fn marker_group(marker: &Marker<'_>, font_size: f64) -> Result<Element, PipelineError> {
|
||||
let mut shape = shape_element(marker.geometry)?;
|
||||
for (key, value) in marker.status.style() {
|
||||
shape.set_attr(key, (*value).to_owned());
|
||||
}
|
||||
shape.set_attr("stroke-width", text_of(font_size / 8.0));
|
||||
let title = Element::svg(
|
||||
"title",
|
||||
Vec::new(),
|
||||
vec![Node::Text(format!(
|
||||
"{} ({})",
|
||||
marker.id,
|
||||
marker.status.name()
|
||||
))],
|
||||
);
|
||||
let mut children = vec![Node::Element(title), Node::Element(shape)];
|
||||
if let Some((x, y)) = label_anchor(marker.geometry).filter(|_| marker.status != Status::Removed)
|
||||
{
|
||||
children.push(Node::Element(text_element(
|
||||
vec![
|
||||
("x", text_of(x)),
|
||||
("y", text_of(y)),
|
||||
("text-anchor", "middle".to_owned()),
|
||||
("font-family", "sans-serif".to_owned()),
|
||||
("font-size", text_of(font_size)),
|
||||
("fill", marker.status.stroke().to_owned()),
|
||||
],
|
||||
marker.id.clone(),
|
||||
)));
|
||||
}
|
||||
Ok(Element::svg(
|
||||
"g",
|
||||
vec![
|
||||
("data-review-id", marker.id.clone()),
|
||||
("data-review-status", marker.status.name().to_owned()),
|
||||
],
|
||||
children,
|
||||
))
|
||||
}
|
||||
|
||||
fn legend(layer: &mut Element, (x, y): (f64, f64), font_size: f64) {
|
||||
let lines = [
|
||||
(Status::Unchanged, "bleu : inchangé"),
|
||||
(
|
||||
Status::Moved,
|
||||
"vert : déplacé ou ajouté (pointillé rouge = position précédente)",
|
||||
),
|
||||
(Status::Removed, "rouge pointillé : retiré"),
|
||||
];
|
||||
for (index, (status, label)) in lines.into_iter().enumerate() {
|
||||
let line = (index + 1) as f64;
|
||||
layer.children.push(Node::Element(text_element(
|
||||
vec![
|
||||
("x", text_of(x)),
|
||||
("y", text_of(y + line * font_size * 1.3)),
|
||||
("font-family", "sans-serif".to_owned()),
|
||||
("font-size", text_of(font_size)),
|
||||
("fill", status.stroke().to_owned()),
|
||||
],
|
||||
label.to_owned(),
|
||||
)));
|
||||
}
|
||||
}
|
||||
|
||||
/// `raw.split(/[\s,]+/).map(Number)`, without the trim the TypeScript did not do: a run of
|
||||
/// separators splits once, a leading or trailing run leaves an empty part (`Number("")` is 0).
|
||||
fn split_numbers(raw: &str) -> Vec<f64> {
|
||||
let mut parts = Vec::new();
|
||||
let mut current = String::new();
|
||||
let mut in_separator = false;
|
||||
for c in raw.chars() {
|
||||
if is_js_space(c) || c == ',' {
|
||||
if !in_separator {
|
||||
parts.push(std::mem::take(&mut current));
|
||||
in_separator = true;
|
||||
}
|
||||
} else {
|
||||
in_separator = false;
|
||||
current.push(c);
|
||||
}
|
||||
}
|
||||
parts.push(current);
|
||||
parts.iter().map(|part| js_number_of(part)).collect()
|
||||
}
|
||||
|
||||
/// `Math.max(Math.round(width / 90), 10)` with `Math.max`'s NaN.
|
||||
fn font_size_of(width: f64) -> f64 {
|
||||
let rounded = js_round(width / 90.0);
|
||||
if rounded.is_nan() {
|
||||
f64::NAN
|
||||
} else {
|
||||
rounded.max(10.0)
|
||||
}
|
||||
}
|
||||
|
||||
pub struct OverlayInput<'a> {
|
||||
pub drawing: &'a CleanedDrawing,
|
||||
pub before: &'a Value,
|
||||
pub after: &'a Value,
|
||||
pub view_id: &'a str,
|
||||
}
|
||||
|
||||
/// The review drawing of one view, framed on that view's box (`renderReviewOverlay`).
|
||||
pub fn render_review_overlay(input: &OverlayInput<'_>) -> Result<String, PipelineError> {
|
||||
let view = input
|
||||
.after
|
||||
.get("views")
|
||||
.and_then(Value::as_array)
|
||||
.and_then(|views| {
|
||||
views
|
||||
.iter()
|
||||
.find(|v| v.get("id").and_then(Value::as_str) == Some(input.view_id))
|
||||
})
|
||||
.ok_or_else(|| fail(format!("unknown view \"{}\"", input.view_id)))?;
|
||||
let mut root = input.drawing.root.clone();
|
||||
if let Some(view_box) = view.get("view_box").filter(|b| b.is_object()) {
|
||||
let part = |key: &str| text_of(number(view_box, key));
|
||||
let text = format!(
|
||||
"{} {} {} {}",
|
||||
part("min_x"),
|
||||
part("min_y"),
|
||||
part("width"),
|
||||
part("height")
|
||||
);
|
||||
root.set_attr("viewBox", text);
|
||||
}
|
||||
root.remove_attr("width");
|
||||
root.remove_attr("height");
|
||||
let frame = split_numbers(root.attr("viewBox").unwrap_or("0 0 1000 1000"));
|
||||
let at = |i: usize| frame.get(i).copied().unwrap_or(f64::NAN);
|
||||
let font_size = font_size_of(at(2));
|
||||
let mut layer = Element::svg("g", vec![("id", LAYER_ID.to_owned())], Vec::new());
|
||||
for marker in overlay_markers(input.before, input.after, input.view_id) {
|
||||
layer
|
||||
.children
|
||||
.push(Node::Element(marker_group(&marker, font_size)?));
|
||||
}
|
||||
legend(&mut layer, (at(0) + font_size, at(1)), font_size);
|
||||
root.children.push(Node::Element(layer));
|
||||
drawing_text(&input.drawing.prolog, &root)
|
||||
}
|
||||
@@ -0,0 +1,371 @@
|
||||
//! Legacy import → curated working draft (`packages/anatomy/src/curation/pipeline.ts`). The
|
||||
//! asset step runs first as the reserved change CHG-000 (one cleaned drawing, head view =
|
||||
//! crop), then the authored changeset; both land in the same records, so the CHANGELOG lists
|
||||
//! them alike. Deterministic: same inputs, same bytes.
|
||||
//!
|
||||
//! The cleaned drawing is an input here, not a product: the owner froze the bytes of
|
||||
//! `draft/assets/horse.svg` (BDR-068), so the Rust pipeline describes them (sha256, byte size,
|
||||
//! asset id) and copies them, and never re-cleans the legacy drawing. The review drawings and
|
||||
//! the two Markdown reports are R5.4.
|
||||
|
||||
use crate::domain::sheet::fingerprint::{
|
||||
FINGERPRINT_CONVENTION, TRUSTED_ALIASES, sheet_fingerprint,
|
||||
};
|
||||
use serde_json::{Map, Value, json};
|
||||
|
||||
use crate::json::to_js_compact;
|
||||
|
||||
use super::app_review::render_app_review_drawing;
|
||||
use super::apply::{apply_changeset, curated_issues};
|
||||
use super::changelog::render_changelog;
|
||||
use super::drawing::read_cleaned_drawing;
|
||||
use super::migration_map::{MigrationMapInput, SheetRef, build_report_migration_map};
|
||||
use super::overlay::{OverlayInput, REVIEW_OVERLAY_DIR, render_review_overlay};
|
||||
use super::records::ChangeRecord;
|
||||
use super::review_report::{ReviewReportInput, render_review_report};
|
||||
use super::schema::{
|
||||
ASSET_CHANGE_ID, Change, ChangeKind, ChangeLevel, ChangePriority, ChangeSummary, ChangeTitle,
|
||||
Op, SetOp,
|
||||
};
|
||||
use super::seed::{DRAFT_CHANGE_NOTE, DraftSeedParams, render_draft_seed_sql};
|
||||
use super::stats::{DocumentStats, document_stats};
|
||||
use super::targets::fail;
|
||||
use crate::anatomy::svg::number::{js_number_of, js_split_list};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::ids::{asset_id_from_sha256, sha256_hex};
|
||||
use crate::curation::legacy::types::{Anomaly, AssetRecord, ImportResult};
|
||||
use crate::curation::legacy::{HEAD_VIEW_ID, MAIN_VIEW_ID};
|
||||
use crate::curation::prettier_json::format_json;
|
||||
use crate::curation::report_guard::format_report;
|
||||
use crate::domain::sheet::{AnatomicalSheetDocument, Reference, SheetIssue, ViewBox};
|
||||
|
||||
pub const DRAFT_SEED_SQL_FILE: &str = "equine_draft.sql";
|
||||
pub const SHEET_REF_FILE: &str = "sheet-ref.json";
|
||||
pub const MIGRATION_MAP_FILE: &str = "report-migration-map.json";
|
||||
pub const CLEANED_ASSET_FILE: &str = "horse.svg";
|
||||
|
||||
pub struct DraftInput<'a> {
|
||||
pub import_result: &'a ImportResult,
|
||||
/// The frozen cleaned drawing, `draft/assets/horse.svg`.
|
||||
pub cleaned_svg: &'a [u8],
|
||||
pub head_view_box: ViewBox,
|
||||
pub references: &'a [Reference],
|
||||
pub changes: &'a [Change],
|
||||
}
|
||||
|
||||
pub struct DraftResult {
|
||||
pub document: Value,
|
||||
/// Legacy import the changeset was applied to.
|
||||
pub baseline: Value,
|
||||
pub legacy_id_map: Map<String, Value>,
|
||||
pub records: Vec<ChangeRecord>,
|
||||
pub changes: Vec<ChangeSummary>,
|
||||
pub assets: Vec<AssetRecord>,
|
||||
pub stats_before: DocumentStats,
|
||||
pub stats_after: DocumentStats,
|
||||
/// Integrity warnings of the curated document (errors refuse).
|
||||
pub issues: Vec<SheetIssue>,
|
||||
pub sheet_id: String,
|
||||
/// The frozen cleaned drawing, `draft/assets/horse.svg`, the review drawings' base.
|
||||
pub cleaned_svg: Vec<u8>,
|
||||
}
|
||||
|
||||
/// Manifest entry of the cleaned drawing; id derived from its sha256 (`cleanedAsset`).
|
||||
pub fn cleaned_asset(bytes: &[u8]) -> AssetRecord {
|
||||
let sha256 = sha256_hex(bytes);
|
||||
AssetRecord {
|
||||
file: CLEANED_ASSET_FILE.to_owned(),
|
||||
svg_asset_id: asset_id_from_sha256(&sha256),
|
||||
sha256,
|
||||
byte_size: bytes.len(),
|
||||
}
|
||||
}
|
||||
|
||||
/// SVG viewBox attribute ("minX minY width height") → document view_box (`parseViewBox`).
|
||||
pub fn parse_view_box(raw: Option<&str>) -> Result<ViewBox, PipelineError> {
|
||||
let numbers: Vec<f64> = js_split_list(raw.unwrap_or_default())
|
||||
.into_iter()
|
||||
.map(js_number_of)
|
||||
.collect();
|
||||
if numbers.len() != 4 || numbers.iter().any(|n| !n.is_finite()) {
|
||||
return Err(fail(format!("invalid viewBox {:?}", raw.unwrap_or("null"))));
|
||||
}
|
||||
Ok(ViewBox {
|
||||
min_x: numbers[0],
|
||||
min_y: numbers[1],
|
||||
width: numbers[2],
|
||||
height: numbers[3],
|
||||
})
|
||||
}
|
||||
|
||||
fn set(target: String, field: &str, value: Value) -> Op {
|
||||
Op::Set(SetOp {
|
||||
target,
|
||||
field: field.to_owned(),
|
||||
value,
|
||||
scope: None,
|
||||
})
|
||||
}
|
||||
|
||||
fn asset_change(asset: &AssetRecord, head_view_box: &ViewBox) -> Result<Change, PipelineError> {
|
||||
let view_box = serde_json::to_value(head_view_box)
|
||||
.map_err(|error| fail(format!("head view box: {error}")))?;
|
||||
Ok(Change {
|
||||
id: ASSET_CHANGE_ID.to_owned(),
|
||||
level: ChangeLevel::A,
|
||||
kind: ChangeKind::Technical,
|
||||
title: ChangeTitle {
|
||||
fr: "Un seul dessin, vue tête = recadrage".to_owned(),
|
||||
en: "Single drawing, head view = crop".to_owned(),
|
||||
},
|
||||
rationale: "Les deux fichiers historiques portent le même dessin (tracés identiques) et \
|
||||
les marqueurs sont désormais des données de la fiche : un seul dessin \
|
||||
nettoyé, la vue tête en est un recadrage."
|
||||
.to_owned(),
|
||||
reference_ids: Vec::new(),
|
||||
question: None,
|
||||
priority: ChangePriority::Medium,
|
||||
ops: vec![
|
||||
set(
|
||||
format!("view:{MAIN_VIEW_ID}"),
|
||||
"svg_asset_id",
|
||||
Value::from(asset.svg_asset_id.as_str()),
|
||||
),
|
||||
set(format!("view:{HEAD_VIEW_ID}"), "svg_asset_id", Value::Null),
|
||||
set(format!("view:{HEAD_VIEW_ID}"), "view_box", view_box),
|
||||
],
|
||||
})
|
||||
}
|
||||
|
||||
fn assert_not_reserved(changes: &[Change]) -> Result<(), PipelineError> {
|
||||
if changes.iter().any(|change| change.id == ASSET_CHANGE_ID) {
|
||||
return Err(fail(format!(
|
||||
"{ASSET_CHANGE_ID} is reserved for the asset step"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn build_equine_draft(input: &DraftInput<'_>) -> Result<DraftResult, PipelineError> {
|
||||
assert_not_reserved(input.changes)?;
|
||||
let asset = cleaned_asset(input.cleaned_svg);
|
||||
let mut changes = vec![asset_change(&asset, &input.head_view_box)?];
|
||||
changes.extend(input.changes.iter().cloned());
|
||||
let baseline = &input.import_result.document;
|
||||
let result = apply_changeset(baseline, input.references, &changes)?;
|
||||
Ok(DraftResult {
|
||||
stats_before: document_stats(baseline)?,
|
||||
stats_after: document_stats(&result.document)?,
|
||||
issues: curated_issues(&result.document)?,
|
||||
document: result.document,
|
||||
baseline: baseline.clone(),
|
||||
legacy_id_map: input.import_result.legacy_id_map.clone(),
|
||||
records: result.records,
|
||||
changes: changes.iter().map(Change::summary).collect(),
|
||||
assets: vec![asset],
|
||||
sheet_id: input.import_result.sheet_id.clone(),
|
||||
cleaned_svg: input.cleaned_svg.to_vec(),
|
||||
})
|
||||
}
|
||||
|
||||
/// What a report references: the sheet and its `sha256-rfc8785-validated-v1` fingerprint
|
||||
/// (R5.6; the pre-switch rule hashed the formatted bytes, kept as a registered alias).
|
||||
fn sheet_ref(draft: &DraftResult, document_json: &str) -> Result<SheetRef, PipelineError> {
|
||||
let fingerprint = sheet_fingerprint(document_json)
|
||||
.map_err(|error| fail(format!("document fingerprint: {error:?}")))?;
|
||||
Ok(SheetRef {
|
||||
sheet_id: draft.sheet_id.clone(),
|
||||
version_id: None,
|
||||
document_sha256: fingerprint,
|
||||
})
|
||||
}
|
||||
|
||||
/// The registered aliases of this sheet's canonical value (descriptive in `sheet-ref.json`).
|
||||
fn registered_aliases(reference: &SheetRef) -> Vec<&'static str> {
|
||||
TRUSTED_ALIASES
|
||||
.iter()
|
||||
.filter(|e| e.sheet_id == reference.sheet_id && e.canonical == reference.document_sha256)
|
||||
.map(|e| e.alias)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The migration map (`report-migration-map.json` value) and its stats.
|
||||
pub fn draft_migration_map(
|
||||
draft: &DraftResult,
|
||||
reference: &SheetRef,
|
||||
) -> Result<(Value, super::migration_map::MigrationMapStats), PipelineError> {
|
||||
build_report_migration_map(&MigrationMapInput {
|
||||
baseline: &draft.baseline,
|
||||
final_document: &draft.document,
|
||||
legacy_id_map: &draft.legacy_id_map,
|
||||
records: &draft.records,
|
||||
sheet: reference,
|
||||
main_view_id: MAIN_VIEW_ID,
|
||||
})
|
||||
}
|
||||
|
||||
/// The views of the curated document, in document order.
|
||||
fn view_ids(document: &Value) -> Vec<String> {
|
||||
document
|
||||
.get("views")
|
||||
.and_then(Value::as_array)
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.filter_map(|view| view.get("id").and_then(Value::as_str).map(str::to_owned))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The review drawings' bytes, projected as an upper bound before any is rendered (the equine
|
||||
/// draft projects 1.41 MB for 0.56 MB written): every drawing repeats the whole cleaned drawing,
|
||||
/// two per view, and each annotation of a view adds its markup. Views added by a changeset
|
||||
/// multiply the drawing (2,000 views asked for 547 MB, security gate R5.4 round 8); unnamed
|
||||
/// triggers repeat the opened view's names in theirs (a 485 KB changeset wrote a 131 MB
|
||||
/// drawing, round 10).
|
||||
pub const MAX_REVIEW_DRAWINGS_BYTES: usize = 64 * 1024 * 1024;
|
||||
|
||||
/// An upper bound of what one annotation writes in a drawing: a fixed part (tags, styles,
|
||||
/// numbers the renderer adds), then its own JSON text and the longest view name, each copied
|
||||
/// at most four times (data attributes and titles of the hit box and the visible circle) and
|
||||
/// escaped at most six times longer (`"` → `"`).
|
||||
fn annotation_bytes(annotation: &Value, view_names: usize) -> Result<usize, PipelineError> {
|
||||
let text = to_js_compact(annotation).map_err(|error| fail(format!("annotation: {error}")))?;
|
||||
Ok(1024 + 24 * (text.len() + view_names))
|
||||
}
|
||||
|
||||
/// The bound of the markup a document's annotations add to one view's drawing.
|
||||
fn layer_bytes(document: &Value, view_id: &str, view_names: usize) -> Result<usize, PipelineError> {
|
||||
let annotations = document.get("annotations").and_then(Value::as_array);
|
||||
let mut total = 0usize;
|
||||
for annotation in annotations.into_iter().flatten() {
|
||||
if annotation.get("view_id").and_then(Value::as_str) == Some(view_id) {
|
||||
total = total.saturating_add(annotation_bytes(annotation, view_names)?);
|
||||
}
|
||||
}
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
/// The longest JSON text of a view name, in either document.
|
||||
fn longest_view_name(documents: [&Value; 2]) -> Result<usize, PipelineError> {
|
||||
let mut longest = 0;
|
||||
for document in documents {
|
||||
let views = document.get("views").and_then(Value::as_array);
|
||||
for name in views
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.filter_map(|view| view.get("name"))
|
||||
{
|
||||
let text = to_js_compact(name).map_err(|error| fail(format!("view: {error}")))?;
|
||||
longest = longest.max(text.len());
|
||||
}
|
||||
}
|
||||
Ok(longest)
|
||||
}
|
||||
|
||||
/// What a drawing writes besides the cleaned drawing and its annotations: the layer group, the
|
||||
/// overlay's group and legend (under 1 KB each; a view without annotations carries them alone).
|
||||
const DRAWING_FIXED_BYTES: usize = 4096;
|
||||
|
||||
/// The projected bytes of the review drawings: per view, the overlay (both documents' markers)
|
||||
/// and the application's drawing (the curated layer), each over the cleaned drawing.
|
||||
pub fn review_drawings_bytes(draft: &DraftResult) -> Result<usize, PipelineError> {
|
||||
let names = longest_view_name([&draft.baseline, &draft.document])?;
|
||||
let mut total = 0usize;
|
||||
for view_id in view_ids(&draft.document) {
|
||||
let after = layer_bytes(&draft.document, &view_id, names)?;
|
||||
let before = layer_bytes(&draft.baseline, &view_id, names)?;
|
||||
let drawings = draft.cleaned_svg.len() + DRAWING_FIXED_BYTES;
|
||||
let view = drawings.saturating_mul(2) + before + after.saturating_mul(2);
|
||||
total = total.saturating_add(view);
|
||||
}
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
/// One review overlay per view (`review/annotations-<view>.svg`), then the application's
|
||||
/// drawing of each view (`review/app-<view>.svg`).
|
||||
pub fn review_drawings(draft: &DraftResult) -> Result<Vec<(String, String)>, PipelineError> {
|
||||
let projected = review_drawings_bytes(draft)?;
|
||||
if projected > MAX_REVIEW_DRAWINGS_BYTES {
|
||||
return Err(fail(format!(
|
||||
"the review drawings would write up to {projected} bytes, past the \
|
||||
{MAX_REVIEW_DRAWINGS_BYTES} bound"
|
||||
)));
|
||||
}
|
||||
let views = view_ids(&draft.document);
|
||||
let drawing = read_cleaned_drawing(&draft.cleaned_svg)?;
|
||||
let typed: AnatomicalSheetDocument = serde_json::from_value(draft.document.clone())
|
||||
.map_err(|error| fail(format!("curated document: {error}")))?;
|
||||
let mut files = Vec::new();
|
||||
for view_id in &views {
|
||||
let overlay = render_review_overlay(&OverlayInput {
|
||||
drawing: &drawing,
|
||||
before: &draft.baseline,
|
||||
after: &draft.document,
|
||||
view_id,
|
||||
})?;
|
||||
files.push((
|
||||
format!("{REVIEW_OVERLAY_DIR}/annotations-{view_id}.svg"),
|
||||
overlay,
|
||||
));
|
||||
}
|
||||
for view_id in &views {
|
||||
let app = render_app_review_drawing(&drawing, &typed, view_id)?;
|
||||
files.push((format!("{REVIEW_OVERLAY_DIR}/app-{view_id}.svg"), app));
|
||||
}
|
||||
Ok(files)
|
||||
}
|
||||
|
||||
/// The text artifacts of the draft folder, keyed by file name, in the TypeScript order
|
||||
/// (`renderDraftArtifacts`); the frozen `assets/horse.svg` is the caller's byte copy.
|
||||
pub fn render_draft_artifacts(
|
||||
draft: &DraftResult,
|
||||
legacy_anomalies: &[Anomaly],
|
||||
) -> Result<Vec<(String, String)>, PipelineError> {
|
||||
let document_json = format_json(&draft.document)?;
|
||||
// The drawings' resource bound applies before the fingerprint validates the document.
|
||||
let drawings = review_drawings(draft)?;
|
||||
let reference = sheet_ref(draft, &document_json)?;
|
||||
let (map, migration_stats) = draft_migration_map(draft, &reference)?;
|
||||
let assets = serde_json::to_value(&draft.assets)
|
||||
.map_err(|error| fail(format!("assets.json: {error}")))?;
|
||||
let sheet_ref_json = json!({
|
||||
"sheet_id": reference.sheet_id,
|
||||
"version_id": reference.version_id,
|
||||
"document_sha256": reference.document_sha256,
|
||||
"schema_version": draft.document.get("schema_version"),
|
||||
"species_id": draft.document.get("species_id"),
|
||||
"fingerprint": {
|
||||
"convention": FINGERPRINT_CONVENTION,
|
||||
"aliases": registered_aliases(&reference),
|
||||
},
|
||||
});
|
||||
let changelog = render_changelog(
|
||||
&draft.records,
|
||||
&draft.stats_before,
|
||||
&draft.stats_after,
|
||||
&migration_stats,
|
||||
)?;
|
||||
let review = render_review_report(&ReviewReportInput {
|
||||
changes: &draft.changes,
|
||||
records: &draft.records,
|
||||
document: &draft.document,
|
||||
stats_before: &draft.stats_before,
|
||||
legacy_anomalies,
|
||||
})?;
|
||||
let seed = render_draft_seed_sql(&DraftSeedParams {
|
||||
sheet_id: &draft.sheet_id,
|
||||
assets: &draft.assets,
|
||||
document_json: &document_json,
|
||||
change_note: DRAFT_CHANGE_NOTE,
|
||||
})?;
|
||||
let mut files = vec![
|
||||
("document.json".to_owned(), document_json),
|
||||
("assets.json".to_owned(), format_json(&assets)?),
|
||||
(SHEET_REF_FILE.to_owned(), format_json(&sheet_ref_json)?),
|
||||
(MIGRATION_MAP_FILE.to_owned(), format_json(&map)?),
|
||||
];
|
||||
files.extend(drawings);
|
||||
files.push(("CHANGELOG.md".to_owned(), format_report(&changelog)?));
|
||||
files.push(("REVIEW-REPORT.md".to_owned(), format_report(&review)?));
|
||||
files.push((DRAFT_SEED_SQL_FILE.to_owned(), seed));
|
||||
Ok(files)
|
||||
}
|
||||
@@ -0,0 +1,299 @@
|
||||
//! One elementary edit of the curated document, kept for the CHANGELOG
|
||||
//! (`packages/anatomy/src/curation/types.ts`), and the state shared by the ops of one change.
|
||||
//!
|
||||
//! The TypeScript remembered the items a change touched by object identity. The port marks
|
||||
//! each touched item with a transient key (`TOUCH_MARK`, holding the touch order) that the
|
||||
//! applier removes at the end of the change and that never reaches a record.
|
||||
|
||||
use serde::Serialize;
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use super::schema::{ChangeSummary, Op, Position};
|
||||
use super::targets::{Item, Path, Step, at_mut};
|
||||
use crate::curation::PipelineError;
|
||||
|
||||
/// A key no authored document holds: a NUL character starts it.
|
||||
pub const TOUCH_MARK: &str = "\u{0}touched";
|
||||
|
||||
/// `before` / `after` are deep copies: the field value for a set, the whole item for a rename,
|
||||
/// remove or move, null on the missing side.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize)]
|
||||
pub struct ChangeRecord {
|
||||
pub change: ChangeSummary,
|
||||
pub op: Op,
|
||||
pub target: String,
|
||||
pub field: Option<String>,
|
||||
pub before: Value,
|
||||
pub after: Value,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct CurationResult {
|
||||
pub document: Value,
|
||||
pub records: Vec<ChangeRecord>,
|
||||
}
|
||||
|
||||
/// The total weight of the records (`record_weight`): about 16 MiB of text, or a quarter of a
|
||||
/// million JSON values, far past the equine changeset (test `the_records_cannot_outgrow_their_bound`)
|
||||
/// and small enough that the reports built from them stay bounded too. Charged as each record
|
||||
/// is made, before the change and the op are cloned into it: one op touching many items is
|
||||
/// refused at the record that passes the bound.
|
||||
pub const MAX_RECORDED_WEIGHT: usize = 16 * 1024 * 1024;
|
||||
|
||||
/// The records' running weight; refuses the record that would pass `MAX_RECORDED_WEIGHT`.
|
||||
pub fn charge(spent: &mut usize, record: RecordParts<'_>) -> Result<(), PipelineError> {
|
||||
*spent = spent.saturating_add(parts_weight(&record));
|
||||
if *spent > MAX_RECORDED_WEIGHT {
|
||||
return Err(PipelineError::Guard(format!(
|
||||
"the records weigh more than {MAX_RECORDED_WEIGHT} units"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// What a record is made of, borrowed: weighed before anything is cloned.
|
||||
pub struct RecordParts<'a> {
|
||||
pub change: &'a ChangeSummary,
|
||||
pub op: &'a Op,
|
||||
pub target: &'a str,
|
||||
pub field: Option<&'a str>,
|
||||
pub before: &'a Value,
|
||||
pub after: &'a Value,
|
||||
}
|
||||
|
||||
/// What the op functions of one change share besides the document; the touch counter spans
|
||||
/// the ops of the change (the first touch of an item fixes its place in the review marking).
|
||||
pub struct Recorder<'a> {
|
||||
pub change: &'a ChangeSummary,
|
||||
pub op: &'a Op,
|
||||
pub records: &'a mut Vec<ChangeRecord>,
|
||||
next_touch: &'a mut u64,
|
||||
spent: &'a mut usize,
|
||||
}
|
||||
|
||||
impl<'a> Recorder<'a> {
|
||||
pub fn new(
|
||||
change: &'a ChangeSummary,
|
||||
op: &'a Op,
|
||||
records: &'a mut Vec<ChangeRecord>,
|
||||
counters: (&'a mut u64, &'a mut usize),
|
||||
) -> Self {
|
||||
Self {
|
||||
change,
|
||||
op,
|
||||
records,
|
||||
next_touch: counters.0,
|
||||
spent: counters.1,
|
||||
}
|
||||
}
|
||||
|
||||
/// Absent values are recorded as null so records stay plain JSON. Refused when the
|
||||
/// records would pass their bound (`charge`), before the change and the op are cloned.
|
||||
pub fn record(
|
||||
&mut self,
|
||||
target: &str,
|
||||
field: Option<&str>,
|
||||
before: Option<Value>,
|
||||
after: Option<Value>,
|
||||
) -> Result<(), PipelineError> {
|
||||
let (before, after) = (unmarked(before), unmarked(after));
|
||||
let parts = RecordParts {
|
||||
change: self.change,
|
||||
op: self.op,
|
||||
target,
|
||||
field,
|
||||
before: &before,
|
||||
after: &after,
|
||||
};
|
||||
charge(self.spent, parts)?;
|
||||
self.records.push(ChangeRecord {
|
||||
change: self.change.clone(),
|
||||
op: self.op.clone(),
|
||||
target: target.to_owned(),
|
||||
field: field.map(str::to_owned),
|
||||
before,
|
||||
after,
|
||||
});
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Marks an item as touched by the change, once (the first touch keeps its order).
|
||||
pub fn touch(&mut self, item: &mut Item) {
|
||||
if item.contains_key(TOUCH_MARK) {
|
||||
return;
|
||||
}
|
||||
item.insert(TOUCH_MARK.to_owned(), Value::from(*self.next_touch));
|
||||
*self.next_touch += 1;
|
||||
}
|
||||
|
||||
/// Touches the item at `path` and every descendant node, depth first (`descendants`).
|
||||
pub fn touch_tree(&mut self, doc: &mut Value, path: &Path) -> Result<(), PipelineError> {
|
||||
let holder = at_mut(doc, path)?;
|
||||
let Some(item) = holder.as_object_mut() else {
|
||||
return Ok(());
|
||||
};
|
||||
self.touch(item);
|
||||
let count = item
|
||||
.get("children")
|
||||
.and_then(Value::as_array)
|
||||
.map_or(0, Vec::len);
|
||||
for index in 0..count {
|
||||
let mut child = path.clone();
|
||||
child.push(Step::Key("children"));
|
||||
child.push(Step::Index(index));
|
||||
self.touch_tree(doc, &child)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// What a record holds, in weight units, about its bytes in memory: 64 per JSON value and per
|
||||
/// string (a `Value` node or a `String` header and its allocation cost tens of bytes, so a run of
|
||||
/// one-character ids is not free) plus one per byte of text: the values, and the change and the
|
||||
/// op, which each record clones. A rename or a remove records the whole item and one op can record once per
|
||||
/// item it reaches, so the records can grow far past the document: `charge` bounds their total.
|
||||
pub fn record_weight(record: &ChangeRecord) -> usize {
|
||||
parts_weight(&RecordParts {
|
||||
change: &record.change,
|
||||
op: &record.op,
|
||||
target: &record.target,
|
||||
field: record.field.as_deref(),
|
||||
before: &record.before,
|
||||
after: &record.after,
|
||||
})
|
||||
}
|
||||
|
||||
fn parts_weight(parts: &RecordParts<'_>) -> usize {
|
||||
let change = parts.change;
|
||||
let texts = [
|
||||
change.id.as_str(),
|
||||
&change.title.fr,
|
||||
&change.title.en,
|
||||
&change.rationale,
|
||||
change.question.as_deref().unwrap_or_default(),
|
||||
parts.target,
|
||||
parts.field.unwrap_or_default(),
|
||||
];
|
||||
let text: usize = texts.iter().map(|text| text_weight(text)).sum::<usize>()
|
||||
+ change
|
||||
.reference_ids
|
||||
.iter()
|
||||
.map(|id| text_weight(id))
|
||||
.sum::<usize>();
|
||||
text + op_weight(parts.op) + value_weight(parts.before) + value_weight(parts.after)
|
||||
}
|
||||
|
||||
/// The op a record clones: its target, its value or its ids.
|
||||
fn op_weight(op: &Op) -> usize {
|
||||
let carried = match op {
|
||||
Op::Set(op) => text_weight(&op.field) + value_weight(&op.value),
|
||||
Op::Add(op) => value_weight(&op.value) + position_weight(op.position.as_ref()),
|
||||
Op::Rename(op) => text_weight(&op.id),
|
||||
Op::Move(op) => text_weight(&op.to) + position_weight(op.position.as_ref()),
|
||||
Op::Cite(op) => op.reference_ids.iter().map(|id| text_weight(id)).sum(),
|
||||
Op::Remove(_) => 0,
|
||||
};
|
||||
64 + text_weight(op.target()) + carried
|
||||
}
|
||||
|
||||
/// The anchor an add or a move clones with its op.
|
||||
fn position_weight(position: Option<&Position>) -> usize {
|
||||
match position {
|
||||
Some(Position::After { after: id } | Position::Before { before: id }) => text_weight(id),
|
||||
Some(Position::Index(_)) | None => 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// A cloned string: its header and allocation, then its bytes.
|
||||
fn text_weight(text: &str) -> usize {
|
||||
64 + text.len()
|
||||
}
|
||||
|
||||
/// The weight of a value, walked with an explicit stack (no recursion).
|
||||
fn value_weight(value: &Value) -> usize {
|
||||
let mut weight = 0;
|
||||
let mut pending = vec![value];
|
||||
while let Some(value) = pending.pop() {
|
||||
weight += 64;
|
||||
match value {
|
||||
Value::String(text) => weight += text.len(),
|
||||
Value::Object(map) => {
|
||||
weight += map.keys().map(|key| text_weight(key)).sum::<usize>();
|
||||
pending.extend(map.values());
|
||||
}
|
||||
Value::Array(items) => pending.extend(items),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
weight
|
||||
}
|
||||
|
||||
/// The touch order of a marked item.
|
||||
pub fn touch_order(item: &Item) -> Option<u64> {
|
||||
item.get(TOUCH_MARK).and_then(Value::as_u64)
|
||||
}
|
||||
|
||||
/// Removes the marks from a value and every value under it, keeping the key order.
|
||||
pub fn strip_marks(value: &mut Value) {
|
||||
match value {
|
||||
Value::Object(map) => {
|
||||
map.shift_remove(TOUCH_MARK);
|
||||
map.values_mut().for_each(strip_marks);
|
||||
}
|
||||
Value::Array(items) => items.iter_mut().for_each(strip_marks),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
fn unmarked(value: Option<Value>) -> Value {
|
||||
let mut value = value.unwrap_or(Value::Null);
|
||||
strip_marks(&mut value);
|
||||
value
|
||||
}
|
||||
|
||||
/// `String(value)` of a JSON value read from the document.
|
||||
pub fn js_string(value: Option<&Value>) -> String {
|
||||
match value {
|
||||
None => "undefined".to_owned(),
|
||||
Some(Value::Null) => "null".to_owned(),
|
||||
Some(Value::Bool(flag)) => flag.to_string(),
|
||||
Some(Value::Number(number)) => number
|
||||
.as_f64()
|
||||
.map_or_else(|| number.to_string(), crate::json::js_number),
|
||||
Some(Value::String(text)) => text.clone(),
|
||||
Some(Value::Array(items)) => items
|
||||
.iter()
|
||||
.map(|item| match item {
|
||||
Value::Null => String::new(),
|
||||
other => js_string(Some(other)),
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
.join(","),
|
||||
Some(Value::Object(_)) => "[object Object]".to_owned(),
|
||||
}
|
||||
}
|
||||
|
||||
fn same_number(a: &serde_json::Number, b: &serde_json::Number) -> bool {
|
||||
match (a.as_f64(), b.as_f64()) {
|
||||
(Some(a), Some(b)) => a.to_bits() == b.to_bits(),
|
||||
_ => a == b,
|
||||
}
|
||||
}
|
||||
|
||||
/// `isDeepStrictEqual` on JSON values: numbers by value (`Object.is`), objects by key set.
|
||||
pub fn js_deep_equal(a: &Value, b: &Value) -> bool {
|
||||
match (a, b) {
|
||||
(Value::Number(a), Value::Number(b)) => same_number(a, b),
|
||||
(Value::Array(a), Value::Array(b)) => {
|
||||
a.len() == b.len() && a.iter().zip(b).all(|(x, y)| js_deep_equal(x, y))
|
||||
}
|
||||
(Value::Object(a), Value::Object(b)) => same_entries(a, b),
|
||||
_ => a == b,
|
||||
}
|
||||
}
|
||||
|
||||
fn same_entries(a: &Map<String, Value>, b: &Map<String, Value>) -> bool {
|
||||
a.len() == b.len()
|
||||
&& a.iter()
|
||||
.all(|(key, value)| b.get(key).is_some_and(|other| js_deep_equal(value, other)))
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
//! Rename propagation (`packages/anatomy/src/curation/rename.ts`): every field pointing at the
|
||||
//! renamed id follows, and each rewrite is a record of its own so the CHANGELOG shows the
|
||||
//! cascade.
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
use super::records::{Recorder, js_string};
|
||||
use super::targets::{Step, TargetKind, all_findings, item_at_mut, list_at, top_collection_path};
|
||||
use crate::curation::PipelineError;
|
||||
|
||||
pub struct Rename {
|
||||
pub kind: TargetKind,
|
||||
pub from: String,
|
||||
pub to: String,
|
||||
}
|
||||
|
||||
/// A field of the annotations or the views that holds an id of the renamed kind.
|
||||
struct Link {
|
||||
kind: TargetKind,
|
||||
field: &'static str,
|
||||
}
|
||||
|
||||
fn links(kind: TargetKind) -> &'static [Link] {
|
||||
match kind {
|
||||
TargetKind::Zone => &[Link {
|
||||
kind: TargetKind::Annotation,
|
||||
field: "finding_zone_id",
|
||||
}],
|
||||
TargetKind::Region => &[Link {
|
||||
kind: TargetKind::Annotation,
|
||||
field: "vertebral_region_code",
|
||||
}],
|
||||
TargetKind::View => &[
|
||||
Link {
|
||||
kind: TargetKind::View,
|
||||
field: "parent_view_id",
|
||||
},
|
||||
Link {
|
||||
kind: TargetKind::Annotation,
|
||||
field: "view_id",
|
||||
},
|
||||
Link {
|
||||
kind: TargetKind::Annotation,
|
||||
field: "opens_view_id",
|
||||
},
|
||||
],
|
||||
TargetKind::Category => &[Link {
|
||||
kind: TargetKind::Annotation,
|
||||
field: "category_id",
|
||||
}],
|
||||
_ => &[],
|
||||
}
|
||||
}
|
||||
|
||||
fn follow_link(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
link: &Link,
|
||||
rename: &Rename,
|
||||
) -> Result<(), PipelineError> {
|
||||
let list = top_collection_path(link.kind);
|
||||
let from = Value::from(rename.from.as_str());
|
||||
for index in 0..list_at(doc, &list)?.len() {
|
||||
let mut path = list.clone();
|
||||
path.push(Step::Index(index));
|
||||
let item = item_at_mut(doc, &path)?;
|
||||
if item.get(link.field) != Some(&from) {
|
||||
continue;
|
||||
}
|
||||
item.insert(link.field.to_owned(), Value::from(rename.to.as_str()));
|
||||
let target = format!("{}:{}", link.kind.name(), js_string(item.get("id")));
|
||||
recorder.record(
|
||||
&target,
|
||||
Some(link.field),
|
||||
Some(from.clone()),
|
||||
Some(Value::from(rename.to.as_str())),
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn follow_citations(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
rename: &Rename,
|
||||
) -> Result<(), PipelineError> {
|
||||
let from = Value::from(rename.from.as_str());
|
||||
for entry in all_findings(doc)? {
|
||||
let item = item_at_mut(doc, &entry.path)?;
|
||||
let Some(ids) = item.get("reference_ids").and_then(Value::as_array) else {
|
||||
continue;
|
||||
};
|
||||
if !ids.contains(&from) {
|
||||
continue;
|
||||
}
|
||||
let before = Value::Array(ids.clone());
|
||||
let after: Vec<Value> = ids
|
||||
.iter()
|
||||
.map(|id| {
|
||||
if *id == from {
|
||||
Value::from(rename.to.as_str())
|
||||
} else {
|
||||
id.clone()
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
recorder.record(
|
||||
&entry.target,
|
||||
Some("reference_ids"),
|
||||
Some(before),
|
||||
Some(Value::Array(after.clone())),
|
||||
)?;
|
||||
item.insert("reference_ids".to_owned(), Value::Array(after));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn propagate_rename(
|
||||
doc: &mut Value,
|
||||
recorder: &mut Recorder<'_>,
|
||||
rename: &Rename,
|
||||
) -> Result<(), PipelineError> {
|
||||
if rename.kind == TargetKind::Reference {
|
||||
return follow_citations(doc, recorder, rename);
|
||||
}
|
||||
for link in links(rename.kind) {
|
||||
follow_link(doc, recorder, link, rename)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,444 @@
|
||||
//! French review report for the osteopath, ten fixed sections
|
||||
//! (`packages/anatomy/src/curation/review-report.ts`). It only formats data (changes,
|
||||
//! records, stats, references): no medical content is written here, every statement comes
|
||||
//! from the authored changeset. The text returned is the raw report, before the table
|
||||
//! alignment.
|
||||
|
||||
use std::cmp::Ordering;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
use super::changelog::{compact_value, record_subject};
|
||||
use super::labels::{kind_label, level_letter, priority_label};
|
||||
use super::records::{ChangeRecord, js_string};
|
||||
use super::schema::{ChangeKind, ChangeLevel, ChangePriority, ChangeSummary};
|
||||
use super::stats::{DocumentStats, stat_rows};
|
||||
use super::targets::{Step, TargetKind, all_findings, at, top_collection_path, zone_findings};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::legacy::types::Anomaly;
|
||||
use crate::curation::markdown::{Cell, code, table};
|
||||
use crate::curation::report_guard::assert_authored;
|
||||
use crate::curation::text::utf16_cmp;
|
||||
|
||||
pub struct ReviewReportInput<'a> {
|
||||
pub changes: &'a [ChangeSummary],
|
||||
pub records: &'a [ChangeRecord],
|
||||
pub document: &'a Value,
|
||||
pub stats_before: &'a DocumentStats,
|
||||
pub legacy_anomalies: &'a [Anomaly],
|
||||
}
|
||||
|
||||
const NONE: &str = "Aucun élément dans cette révision.";
|
||||
|
||||
/// `[...new Set(items)]`: first occurrences, in order.
|
||||
fn distinct(items: impl Iterator<Item = String>) -> Vec<String> {
|
||||
let mut seen = HashSet::new();
|
||||
items.filter(|item| seen.insert(item.clone())).collect()
|
||||
}
|
||||
|
||||
fn codes(items: &[String]) -> String {
|
||||
items
|
||||
.iter()
|
||||
.map(|item| code(Some(item)))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ")
|
||||
}
|
||||
|
||||
fn subjects(subjects: &HashMap<&str, Vec<String>>, change: &ChangeSummary) -> String {
|
||||
codes(
|
||||
subjects
|
||||
.get(change.id.as_str())
|
||||
.map_or(&[][..], Vec::as_slice),
|
||||
)
|
||||
}
|
||||
|
||||
/// The distinct subjects of each change's records, grouped in one pass.
|
||||
fn subjects_by_change(records: &[ChangeRecord]) -> HashMap<&str, Vec<String>> {
|
||||
let mut grouped: HashMap<&str, Vec<String>> = HashMap::new();
|
||||
let mut seen: HashSet<(&str, String)> = HashSet::new();
|
||||
for record in records {
|
||||
let subject = record_subject(record);
|
||||
if seen.insert((record.change.id.as_str(), subject.clone())) {
|
||||
grouped
|
||||
.entry(record.change.id.as_str())
|
||||
.or_default()
|
||||
.push(subject);
|
||||
}
|
||||
}
|
||||
grouped
|
||||
}
|
||||
|
||||
fn references(ids: &[String]) -> String {
|
||||
if ids.is_empty() {
|
||||
"aucune".to_owned()
|
||||
} else {
|
||||
codes(ids)
|
||||
}
|
||||
}
|
||||
|
||||
fn change_line(grouped: &HashMap<&str, Vec<String>>, change: &ChangeSummary) -> String {
|
||||
let confirm = if change.level.needs_confirmation() {
|
||||
"oui"
|
||||
} else {
|
||||
"non"
|
||||
};
|
||||
let question = change
|
||||
.question
|
||||
.as_ref()
|
||||
.map_or_else(String::new, |question| format!(" Question : {question}"));
|
||||
format!(
|
||||
"- {} {} (niveau {}, {}, confirmation requise : {confirm}). {} Objets : {}. \
|
||||
Références : {}.{question}",
|
||||
code(Some(&change.id)),
|
||||
change.title.fr,
|
||||
level_letter(change.level),
|
||||
kind_label(change.kind),
|
||||
change.rationale,
|
||||
subjects(grouped, change),
|
||||
references(&change.reference_ids),
|
||||
)
|
||||
}
|
||||
|
||||
fn change_list(input: &ReviewReportInput<'_>, keep: impl Fn(&ChangeSummary) -> bool) -> String {
|
||||
let grouped = subjects_by_change(input.records);
|
||||
let lines: Vec<String> = input
|
||||
.changes
|
||||
.iter()
|
||||
.filter(|change| keep(change))
|
||||
.map(|change| change_line(&grouped, change))
|
||||
.collect();
|
||||
if lines.is_empty() {
|
||||
NONE.to_owned()
|
||||
} else {
|
||||
lines.join("\n")
|
||||
}
|
||||
}
|
||||
|
||||
fn stat_table(head: &str, stats: &DocumentStats) -> String {
|
||||
let rows: Vec<Vec<Cell>> = stat_rows(stats)
|
||||
.into_iter()
|
||||
.map(|(label, value)| vec![Cell::from(label), Cell::from(value)])
|
||||
.collect();
|
||||
table(&[head, "Valeur"], &rows)
|
||||
}
|
||||
|
||||
fn initial_quality(input: &ReviewReportInput<'_>) -> String {
|
||||
let to_validate = input
|
||||
.legacy_anomalies
|
||||
.iter()
|
||||
.filter(|anomaly| anomaly.level == "a_valider")
|
||||
.count();
|
||||
[
|
||||
format!(
|
||||
"L'import historique signale {} anomalies, dont {to_validate} à valider (détail dans \
|
||||
`legacy-import/IMPORT-REPORT.md`).",
|
||||
input.legacy_anomalies.len()
|
||||
),
|
||||
String::new(),
|
||||
stat_table("Indicateur (import historique)", input.stats_before),
|
||||
]
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
fn translations(input: &ReviewReportInput<'_>) -> String {
|
||||
let changes: Vec<&ChangeSummary> = input
|
||||
.changes
|
||||
.iter()
|
||||
.filter(|change| change.kind == ChangeKind::Translation)
|
||||
.collect();
|
||||
let ids: HashSet<&str> = changes.iter().map(|change| change.id.as_str()).collect();
|
||||
let targets = distinct(
|
||||
input
|
||||
.records
|
||||
.iter()
|
||||
.filter(|record| ids.contains(record.change.id.as_str()))
|
||||
.map(record_subject),
|
||||
);
|
||||
let head = format!(
|
||||
"{} changement(s) de traduction, {} objet(s) traduit(s).",
|
||||
changes.len(),
|
||||
targets.len()
|
||||
);
|
||||
if targets.is_empty() {
|
||||
return head;
|
||||
}
|
||||
let lines: Vec<String> = targets
|
||||
.iter()
|
||||
.map(|target| format!("- {}", code(Some(target))))
|
||||
.collect();
|
||||
format!("{head}\n\n{}", lines.join("\n"))
|
||||
}
|
||||
|
||||
fn vet_flag_changes(input: &ReviewReportInput<'_>) -> Result<String, PipelineError> {
|
||||
let mut lines = Vec::new();
|
||||
for record in input
|
||||
.records
|
||||
.iter()
|
||||
.filter(|record| record.field.as_deref() == Some("requires_vet_referral"))
|
||||
{
|
||||
lines.push(format!(
|
||||
"- {} : {} → {} ({})",
|
||||
code(Some(&record.target)),
|
||||
compact_value(&record.before)?,
|
||||
compact_value(&record.after)?,
|
||||
code(Some(&record.change.id))
|
||||
));
|
||||
}
|
||||
Ok(if lines.is_empty() {
|
||||
"Aucun renvoi vétérinaire modifié dans cette révision.".to_owned()
|
||||
} else {
|
||||
lines.join("\n")
|
||||
})
|
||||
}
|
||||
|
||||
fn vet_flags_per_zone(document: &Value) -> Result<String, PipelineError> {
|
||||
let zones_path = top_collection_path(TargetKind::Zone);
|
||||
let count = document
|
||||
.get("finding_zones")
|
||||
.and_then(Value::as_array)
|
||||
.map_or(0, Vec::len);
|
||||
let mut rows = Vec::new();
|
||||
for index in 0..count {
|
||||
let mut zone_path = zones_path.clone();
|
||||
zone_path.push(Step::Index(index));
|
||||
let zone = at(document, &zone_path)?;
|
||||
let flags = zone_findings(document, &zone_path)?
|
||||
.iter()
|
||||
.filter(|entry| {
|
||||
at(document, &entry.path)
|
||||
.is_ok_and(|item| item.get("requires_vet_referral") == Some(&Value::Bool(true)))
|
||||
})
|
||||
.count();
|
||||
if flags > 0 {
|
||||
rows.push(vec![
|
||||
Cell::from(code(Some(&js_string(zone.get("id"))))),
|
||||
Cell::from(flags),
|
||||
]);
|
||||
}
|
||||
}
|
||||
Ok(if rows.is_empty() {
|
||||
"Aucun renvoi vétérinaire.".to_owned()
|
||||
} else {
|
||||
table(&["Zone", "Renvois vétérinaires"], &rows)
|
||||
})
|
||||
}
|
||||
|
||||
fn clinical_vigilance(input: &ReviewReportInput<'_>) -> Result<String, PipelineError> {
|
||||
Ok([
|
||||
"Renvois vétérinaires modifiés :".to_owned(),
|
||||
String::new(),
|
||||
vet_flag_changes(input)?,
|
||||
String::new(),
|
||||
"Renvois vétérinaires de la fiche, par zone :".to_owned(),
|
||||
String::new(),
|
||||
vet_flags_per_zone(input.document)?,
|
||||
]
|
||||
.join("\n"))
|
||||
}
|
||||
|
||||
/// Citing findings per reference id, counted in one walk: a finding citing an id twice
|
||||
/// counts once, as `ids.includes(id)` did.
|
||||
fn citation_counts(document: &Value) -> Result<HashMap<String, usize>, PipelineError> {
|
||||
let mut counts: HashMap<String, usize> = HashMap::new();
|
||||
for entry in all_findings(document)? {
|
||||
let ids = at(document, &entry.path)?
|
||||
.get("reference_ids")
|
||||
.and_then(Value::as_array);
|
||||
let distinct: HashSet<&str> = ids
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.filter_map(Value::as_str)
|
||||
.collect();
|
||||
for id in distinct {
|
||||
*counts.entry(id.to_owned()).or_default() += 1;
|
||||
}
|
||||
}
|
||||
Ok(counts)
|
||||
}
|
||||
|
||||
/// `Boolean(value)` of a reference field.
|
||||
fn truthy_text(value: Option<&Value>) -> Option<String> {
|
||||
match value {
|
||||
None | Some(Value::Null | Value::Bool(false)) => None,
|
||||
Some(Value::String(text)) if text.is_empty() => None,
|
||||
other => Some(js_string(other)),
|
||||
}
|
||||
}
|
||||
|
||||
fn bibliography(document: &Value) -> Result<String, PipelineError> {
|
||||
let references = document
|
||||
.get("references")
|
||||
.and_then(Value::as_array)
|
||||
.map_or(&[][..], Vec::as_slice);
|
||||
let counts = citation_counts(document)?;
|
||||
let mut lines = Vec::new();
|
||||
for (index, reference) in references.iter().enumerate() {
|
||||
let links: Vec<String> = [
|
||||
truthy_text(reference.get("url")).map(|url| format!("URL : {url}")),
|
||||
truthy_text(reference.get("doi")).map(|doi| format!("DOI : {doi}")),
|
||||
]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.collect();
|
||||
let place = if links.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!(" ({})", links.join(", "))
|
||||
};
|
||||
let id = js_string(reference.get("id"));
|
||||
lines.push(format!(
|
||||
"{}. {} {}{place}, cité par {} nœud(s).",
|
||||
index + 1,
|
||||
code(Some(&id)),
|
||||
js_string(reference.get("citation")),
|
||||
counts.get(&id).copied().unwrap_or(0)
|
||||
));
|
||||
}
|
||||
Ok(if lines.is_empty() {
|
||||
"Aucune référence déclarée.".to_owned()
|
||||
} else {
|
||||
lines.join("\n")
|
||||
})
|
||||
}
|
||||
|
||||
fn priority_rank(priority: ChangePriority) -> u8 {
|
||||
match priority {
|
||||
ChangePriority::High => 0,
|
||||
ChangePriority::Medium => 1,
|
||||
ChangePriority::Low => 2,
|
||||
}
|
||||
}
|
||||
|
||||
/// `byPriorityThenId`: priority order, then ids by UTF-16 code units (a stable sort).
|
||||
fn by_priority_then_id(a: &&ChangeSummary, b: &&ChangeSummary) -> Ordering {
|
||||
priority_rank(a.priority)
|
||||
.cmp(&priority_rank(b.priority))
|
||||
.then_with(|| utf16_cmp(&a.id, &b.id))
|
||||
}
|
||||
|
||||
fn professional_decisions(input: &ReviewReportInput<'_>) -> String {
|
||||
let mut questions: Vec<&ChangeSummary> = input
|
||||
.changes
|
||||
.iter()
|
||||
.filter(|change| change.question.is_some())
|
||||
.collect();
|
||||
questions.sort_by(by_priority_then_id);
|
||||
if questions.is_empty() {
|
||||
return "Aucune question ouverte.".to_owned();
|
||||
}
|
||||
questions
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, change)| {
|
||||
format!(
|
||||
"{}. {} (priorité {}) : {} Références : {}.",
|
||||
index + 1,
|
||||
code(Some(&change.id)),
|
||||
priority_label(change.priority),
|
||||
change.question.as_deref().unwrap_or_default(),
|
||||
references(&change.reference_ids)
|
||||
)
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
fn intro() -> String {
|
||||
[
|
||||
"# Rapport de revue : révision de travail",
|
||||
"",
|
||||
"Généré par `pnpm import:equine` à partir des changements rédigés \
|
||||
(`curation/changes.json`). Ne pas modifier à la main. Rien n’est publié : la révision \
|
||||
reste un brouillon jusqu’à la décision du professionnel.",
|
||||
]
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
/// A control character (a line break, a tab, a carriage return…) in an authored text the
|
||||
/// report writes in a list item would break the item, and Prettier re-indents what follows:
|
||||
/// refused (the TypeScript wrote it raw). A list-item field shares its line with the other fields of the change and the generated
|
||||
/// code spans: no control character, and no wiki-link or liquid bracket (`[[`, `]]`, `{%`, `%}`),
|
||||
/// which Prettier pairs across the whole line (a `[[` in a rationale closed by the question's
|
||||
/// `]]`, the code spans between them notwithstanding).
|
||||
fn assert_single_line(text: &str) -> Result<(), PipelineError> {
|
||||
let across = ["[[", "]]", "{%", "%}"]
|
||||
.iter()
|
||||
.any(|bracket| text.contains(bracket));
|
||||
if across || text.chars().any(char::is_control) {
|
||||
return Err(PipelineError::UnsupportedMarkdown(text.to_owned()));
|
||||
}
|
||||
assert_authored(text)
|
||||
}
|
||||
|
||||
fn assert_list_items_single_line(input: &ReviewReportInput<'_>) -> Result<(), PipelineError> {
|
||||
for change in input.changes {
|
||||
assert_single_line(&change.title.fr)?;
|
||||
assert_single_line(&change.rationale)?;
|
||||
if let Some(question) = &change.question {
|
||||
assert_single_line(question)?;
|
||||
}
|
||||
}
|
||||
let references = input.document.get("references").and_then(Value::as_array);
|
||||
for reference in references.into_iter().flatten() {
|
||||
for key in ["citation", "url", "doi"] {
|
||||
assert_single_line(&js_string(reference.get(key)))?;
|
||||
}
|
||||
}
|
||||
for record in input.records {
|
||||
if record.field.as_deref() == Some("requires_vet_referral") {
|
||||
assert_single_line(&compact_value(&record.before)?)?;
|
||||
assert_single_line(&compact_value(&record.after)?)?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `renderReviewReport`: the raw text, before the table alignment.
|
||||
pub fn render_review_report(input: &ReviewReportInput<'_>) -> Result<String, PipelineError> {
|
||||
assert_list_items_single_line(input)?;
|
||||
let is = |level: ChangeLevel| move |c: &ChangeSummary| c.level == level;
|
||||
let sections: Vec<(&str, String)> = vec![
|
||||
("1. Qualité des données initiales", initial_quality(input)),
|
||||
(
|
||||
"2. Principales erreurs anatomiques et ostéopathiques",
|
||||
change_list(input, |c| {
|
||||
c.level == ChangeLevel::A
|
||||
&& matches!(c.kind, ChangeKind::Correction | ChangeKind::Removal)
|
||||
}),
|
||||
),
|
||||
(
|
||||
"3. Corrections effectuées",
|
||||
change_list(input, is(ChangeLevel::A)),
|
||||
),
|
||||
(
|
||||
"4. Ajouts",
|
||||
change_list(input, |c| c.kind == ChangeKind::Enrichment),
|
||||
),
|
||||
(
|
||||
"5. Réorganisations du catalogue",
|
||||
change_list(input, |c| c.kind == ChangeKind::Restructure),
|
||||
),
|
||||
(
|
||||
"6. Éléments incertains",
|
||||
change_list(input, is(ChangeLevel::C)),
|
||||
),
|
||||
("7. Traductions", translations(input)),
|
||||
(
|
||||
"8. Points de vigilance clinique",
|
||||
clinical_vigilance(input)?,
|
||||
),
|
||||
("9. Bibliographie", bibliography(input.document)?),
|
||||
(
|
||||
"10. Décisions réservées au professionnel",
|
||||
professional_decisions(input),
|
||||
),
|
||||
];
|
||||
let mut parts = vec![intro()];
|
||||
parts.extend(
|
||||
sections
|
||||
.into_iter()
|
||||
.map(|(title, body)| format!("## {title}\n\n{body}")),
|
||||
);
|
||||
Ok(format!("{}\n", parts.join("\n\n")))
|
||||
}
|
||||
@@ -0,0 +1,456 @@
|
||||
//! The authored curation inputs (`packages/anatomy/src/curation/schema.ts`): a changeset of
|
||||
//! traceable changes applied on the legacy baseline, and the references registry they cite.
|
||||
//! Every object is strict, as the Zod schemas were, so a typo in a hand-written file fails
|
||||
//! loud instead of being ignored. Serialisation follows the Zod output: the shape order of the
|
||||
//! keys, `priority` always present, an absent option absent.
|
||||
//!
|
||||
//! An optional key is `.optional()` as in Zod: absent is fine, `null` is refused.
|
||||
//!
|
||||
//! Declared difference: a `set` without `value` is refused (`z.unknown()` accepted the missing
|
||||
//! key and `JSON.stringify` then dropped the field).
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use super::shape::assert_same_shape;
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::text::js_normalize;
|
||||
use crate::domain::js;
|
||||
use crate::domain::sheet::Reference;
|
||||
|
||||
/// Reserved for the asset step of the pipeline, never authored.
|
||||
pub const ASSET_CHANGE_ID: &str = "CHG-000";
|
||||
|
||||
/// A demonstrated error, B supported improvement, C uncertain (question).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum ChangeLevel {
|
||||
A,
|
||||
B,
|
||||
C,
|
||||
}
|
||||
|
||||
impl ChangeLevel {
|
||||
/// Derived, never stored: only a demonstrated error needs no sign-off.
|
||||
pub fn needs_confirmation(self) -> bool {
|
||||
self != Self::A
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ChangeKind {
|
||||
Correction,
|
||||
Terminology,
|
||||
Translation,
|
||||
Restructure,
|
||||
Enrichment,
|
||||
Reference,
|
||||
Geometry,
|
||||
Removal,
|
||||
Technical,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ChangePriority {
|
||||
High,
|
||||
Medium,
|
||||
Low,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum Scope {
|
||||
#[serde(rename = "self")]
|
||||
OwnItem,
|
||||
Subtree,
|
||||
}
|
||||
|
||||
/// Document-level collections an `add` on 'document' may extend.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum DocumentCollection {
|
||||
References,
|
||||
Categories,
|
||||
Views,
|
||||
Annotations,
|
||||
FindingZones,
|
||||
Regions,
|
||||
}
|
||||
|
||||
/// Where an added or moved item lands among its siblings.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(untagged, deny_unknown_fields)]
|
||||
pub enum Position {
|
||||
Index(#[serde(deserialize_with = "non_negative_int")] u64),
|
||||
After {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
after: String,
|
||||
},
|
||||
Before {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
before: String,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct SetOp {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub target: String,
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub field: String,
|
||||
pub value: Value,
|
||||
#[serde(
|
||||
default,
|
||||
deserialize_with = "js::present",
|
||||
skip_serializing_if = "Option::is_none"
|
||||
)]
|
||||
pub scope: Option<Scope>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct RenameOp {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub target: String,
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub id: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct AddOp {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub target: String,
|
||||
#[serde(
|
||||
default,
|
||||
deserialize_with = "js::present",
|
||||
skip_serializing_if = "Option::is_none"
|
||||
)]
|
||||
pub field: Option<DocumentCollection>,
|
||||
pub value: Value,
|
||||
#[serde(
|
||||
default,
|
||||
deserialize_with = "js::present",
|
||||
skip_serializing_if = "Option::is_none"
|
||||
)]
|
||||
pub position: Option<Position>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct RemoveOp {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub target: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct MoveOp {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub target: String,
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub to: String,
|
||||
#[serde(
|
||||
default,
|
||||
deserialize_with = "js::present",
|
||||
skip_serializing_if = "Option::is_none"
|
||||
)]
|
||||
pub position: Option<Position>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct CiteOp {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub target: String,
|
||||
#[serde(deserialize_with = "non_empty_strings_min_one")]
|
||||
pub reference_ids: Vec<String>,
|
||||
#[serde(
|
||||
default,
|
||||
deserialize_with = "js::present",
|
||||
skip_serializing_if = "Option::is_none"
|
||||
)]
|
||||
pub scope: Option<Scope>,
|
||||
}
|
||||
|
||||
/// The six op kinds of a change, tagged by `op`.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize)]
|
||||
#[serde(tag = "op", rename_all = "snake_case")]
|
||||
pub enum Op {
|
||||
Set(SetOp),
|
||||
Rename(RenameOp),
|
||||
Add(AddOp),
|
||||
Remove(RemoveOp),
|
||||
Move(MoveOp),
|
||||
Cite(CiteOp),
|
||||
}
|
||||
|
||||
impl Op {
|
||||
pub fn target(&self) -> &str {
|
||||
match self {
|
||||
Self::Set(op) => &op.target,
|
||||
Self::Rename(op) => &op.target,
|
||||
Self::Add(op) => &op.target,
|
||||
Self::Remove(op) => &op.target,
|
||||
Self::Move(op) => &op.target,
|
||||
Self::Cite(op) => &op.target,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The discriminated union: the `op` key picks the variant, the rest is parsed strictly.
|
||||
impl<'de> Deserialize<'de> for Op {
|
||||
fn deserialize<D: serde::Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
|
||||
use serde::de::Error;
|
||||
let mut object = Map::<String, Value>::deserialize(deserializer)?;
|
||||
let kind = match object.remove("op") {
|
||||
Some(Value::String(kind)) => kind,
|
||||
Some(_) => return Err(D::Error::custom("op must be a string")),
|
||||
None => return Err(D::Error::custom("missing field `op`")),
|
||||
};
|
||||
let rest = Value::Object(object);
|
||||
let parsed = match kind.as_str() {
|
||||
"set" => serde_json::from_value(rest).map(Self::Set),
|
||||
"rename" => serde_json::from_value(rest).map(Self::Rename),
|
||||
"add" => serde_json::from_value(rest).map(Self::Add),
|
||||
"remove" => serde_json::from_value(rest).map(Self::Remove),
|
||||
"move" => serde_json::from_value(rest).map(Self::Move),
|
||||
"cite" => serde_json::from_value(rest).map(Self::Cite),
|
||||
other => return Err(D::Error::custom(format!("unknown op \"{other}\""))),
|
||||
};
|
||||
parsed.map_err(|error| D::Error::custom(format!("{kind} op: {error}")))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct ChangeTitle {
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub fr: String,
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub en: String,
|
||||
}
|
||||
|
||||
/// A change as cited by its records: everything but the ops, in the Zod shape order.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||
pub struct ChangeSummary {
|
||||
pub id: String,
|
||||
pub level: ChangeLevel,
|
||||
pub kind: ChangeKind,
|
||||
pub title: ChangeTitle,
|
||||
pub rationale: String,
|
||||
pub reference_ids: Vec<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub question: Option<String>,
|
||||
pub priority: ChangePriority,
|
||||
}
|
||||
|
||||
/// One authored change: `ChangeSchema` (strict, `priority` defaulted, `ops` non-empty).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Change {
|
||||
#[serde(deserialize_with = "change_id")]
|
||||
pub id: String,
|
||||
pub level: ChangeLevel,
|
||||
pub kind: ChangeKind,
|
||||
pub title: ChangeTitle,
|
||||
#[serde(deserialize_with = "non_empty")]
|
||||
pub rationale: String,
|
||||
#[serde(deserialize_with = "non_empty_strings")]
|
||||
pub reference_ids: Vec<String>,
|
||||
#[serde(
|
||||
default,
|
||||
deserialize_with = "optional_non_empty",
|
||||
skip_serializing_if = "Option::is_none"
|
||||
)]
|
||||
pub question: Option<String>,
|
||||
#[serde(default = "default_priority")]
|
||||
pub priority: ChangePriority,
|
||||
#[serde(deserialize_with = "ops_min_one")]
|
||||
pub ops: Vec<Op>,
|
||||
}
|
||||
|
||||
fn default_priority() -> ChangePriority {
|
||||
ChangePriority::Medium
|
||||
}
|
||||
|
||||
impl Change {
|
||||
/// `changeSummary`: the change without its ops.
|
||||
pub fn summary(&self) -> ChangeSummary {
|
||||
ChangeSummary {
|
||||
id: self.id.clone(),
|
||||
level: self.level,
|
||||
kind: self.kind,
|
||||
title: self.title.clone(),
|
||||
rationale: self.rationale.clone(),
|
||||
reference_ids: self.reference_ids.clone(),
|
||||
question: self.question.clone(),
|
||||
priority: self.priority,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Changeset {
|
||||
pub changes: Vec<Change>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct ReferencesFile {
|
||||
references: Vec<Reference>,
|
||||
}
|
||||
|
||||
fn non_empty<'de, D: serde::Deserializer<'de>>(d: D) -> Result<String, D::Error> {
|
||||
let text = String::deserialize(d)?;
|
||||
if text.is_empty() {
|
||||
return Err(serde::de::Error::custom("must not be empty"));
|
||||
}
|
||||
Ok(text)
|
||||
}
|
||||
|
||||
fn optional_non_empty<'de, D: serde::Deserializer<'de>>(d: D) -> Result<Option<String>, D::Error> {
|
||||
non_empty(d).map(Some)
|
||||
}
|
||||
|
||||
fn non_empty_strings<'de, D: serde::Deserializer<'de>>(d: D) -> Result<Vec<String>, D::Error> {
|
||||
let items = Vec::<String>::deserialize(d)?;
|
||||
if items.iter().any(String::is_empty) {
|
||||
return Err(serde::de::Error::custom("ids must not be empty"));
|
||||
}
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
fn non_empty_strings_min_one<'de, D: serde::Deserializer<'de>>(
|
||||
d: D,
|
||||
) -> Result<Vec<String>, D::Error> {
|
||||
let items = non_empty_strings(d)?;
|
||||
if items.is_empty() {
|
||||
return Err(serde::de::Error::custom("at least one id"));
|
||||
}
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
fn ops_min_one<'de, D: serde::Deserializer<'de>>(d: D) -> Result<Vec<Op>, D::Error> {
|
||||
let ops = Vec::<Op>::deserialize(d)?;
|
||||
if ops.is_empty() {
|
||||
return Err(serde::de::Error::custom("a change needs at least one op"));
|
||||
}
|
||||
Ok(ops)
|
||||
}
|
||||
|
||||
/// `^CHG-\d{3,}$` (ASCII digits, as a JavaScript regex without the `u` flag).
|
||||
fn change_id<'de, D: serde::Deserializer<'de>>(d: D) -> Result<String, D::Error> {
|
||||
let id = String::deserialize(d)?;
|
||||
let digits = id.strip_prefix("CHG-").unwrap_or_default();
|
||||
if digits.len() < 3 || !digits.bytes().all(|byte| byte.is_ascii_digit()) {
|
||||
return Err(serde::de::Error::custom(format!(
|
||||
"invalid change id {id:?}"
|
||||
)));
|
||||
}
|
||||
Ok(id)
|
||||
}
|
||||
|
||||
/// `z.number().int().min(0)`: an integral, non-negative JavaScript number (−0 and `1.0`
|
||||
/// included); a value beyond `u64` saturates and fails the range check of the op.
|
||||
fn non_negative_int<'de, D: serde::Deserializer<'de>>(d: D) -> Result<u64, D::Error> {
|
||||
let value = f64::deserialize(d)?;
|
||||
if !value.is_finite() || value.fract() != 0.0 || value < 0.0 {
|
||||
return Err(serde::de::Error::custom(
|
||||
"position must be a non-negative integer",
|
||||
));
|
||||
}
|
||||
Ok(value as u64)
|
||||
}
|
||||
|
||||
fn duplicates<'a>(ids: impl Iterator<Item = &'a str>) -> Vec<&'a str> {
|
||||
let mut seen: std::collections::HashSet<&str> = std::collections::HashSet::new();
|
||||
ids.filter(|id| !seen.insert(id)).collect()
|
||||
}
|
||||
|
||||
fn refuse(file: &str, error: impl std::fmt::Display) -> PipelineError {
|
||||
PipelineError::Source(format!("{file}: {error}"))
|
||||
}
|
||||
|
||||
/// The size cap of an authored curation file (`changes.json`, `references.json`), checked
|
||||
/// before any parse allocation. The committed files are about 222 KB and 17 KB; the cap leaves
|
||||
/// a wide margin to grow and bounds the parse (RUST.md §11.16).
|
||||
pub const MAX_CURATION_FILE_BYTES: usize = 4 * 1024 * 1024;
|
||||
|
||||
/// `JSON.parse`: a repeated key keeps its first position and its last value (the parsed
|
||||
/// value is read into the typed shape afterwards, so duplicates never reach serde's derive),
|
||||
/// array index keys come first, integers beyond 2^53 are rounded (`js_normalize`).
|
||||
fn json_parse(file: &str, text: &str) -> Result<Value, PipelineError> {
|
||||
if text.len() > MAX_CURATION_FILE_BYTES {
|
||||
return Err(refuse(
|
||||
file,
|
||||
format!("larger than {MAX_CURATION_FILE_BYTES} bytes"),
|
||||
));
|
||||
}
|
||||
let mut value = js::parse_json(text).map_err(|error| refuse(file, error))?;
|
||||
js_normalize(&mut value);
|
||||
Ok(value)
|
||||
}
|
||||
|
||||
/// A key starting with NUL is reserved for the applier's transient touch marks
|
||||
/// (`records::TOUCH_MARK`); an authored value holding one is refused (the TypeScript refused
|
||||
/// such a key at the strict document gate).
|
||||
fn reserved_key(value: &Value) -> Option<&str> {
|
||||
match value {
|
||||
Value::Object(map) => map.iter().find_map(|(key, item)| {
|
||||
if key.starts_with('\0') {
|
||||
Some(key.as_str())
|
||||
} else {
|
||||
reserved_key(item)
|
||||
}
|
||||
}),
|
||||
Value::Array(items) => items.iter().find_map(reserved_key),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// `ChangesetSchema.parse(JSON.parse(text))`.
|
||||
pub fn parse_changeset(text: &str) -> Result<Changeset, PipelineError> {
|
||||
let value = json_parse("changes.json", text)?;
|
||||
if let Some(key) = reserved_key(&value) {
|
||||
return Err(refuse("changes.json", format!("reserved key {key:?}")));
|
||||
}
|
||||
let changeset: Changeset =
|
||||
serde_json::from_value(value.clone()).map_err(|e| refuse("changes.json", e))?;
|
||||
assert_same_shape("changes.json", &value, &changeset)?;
|
||||
for change in &changeset.changes {
|
||||
if change.level == ChangeLevel::C && change.question.is_none() {
|
||||
return Err(refuse(
|
||||
"changes.json",
|
||||
format!("{}: a level C change needs a question", change.id),
|
||||
));
|
||||
}
|
||||
}
|
||||
let ids = changeset.changes.iter().map(|change| change.id.as_str());
|
||||
if let Some(id) = duplicates(ids).first() {
|
||||
return Err(refuse("changes.json", format!("duplicate change id {id}")));
|
||||
}
|
||||
Ok(changeset)
|
||||
}
|
||||
|
||||
/// `ReferencesFileSchema.parse(JSON.parse(text)).references`.
|
||||
pub fn parse_references(text: &str) -> Result<Vec<Reference>, PipelineError> {
|
||||
let value = json_parse("references.json", text)?;
|
||||
let file: ReferencesFile =
|
||||
serde_json::from_value(value.clone()).map_err(|e| refuse("references.json", e))?;
|
||||
assert_same_shape("references.json", &value, &file)?;
|
||||
let ids = file
|
||||
.references
|
||||
.iter()
|
||||
.map(|reference| reference.id.as_str());
|
||||
if let Some(id) = duplicates(ids).first() {
|
||||
return Err(refuse(
|
||||
"references.json",
|
||||
format!("duplicate reference id {id}"),
|
||||
));
|
||||
}
|
||||
Ok(file.references)
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
//! Idempotent and converging PostgreSQL seed of an official sheet's working DRAFT
|
||||
//! (`packages/anatomy/src/curation/seed.ts`). Runs as the database owner in `supabase db
|
||||
//! reset` and in the PGlite harness. A fresh database gets the draft through
|
||||
//! `private.insert_sheet_draft` (audited, null actor); a database still holding an older draft
|
||||
//! as the sheet's only version gets its content replaced in place. Published rows are never
|
||||
//! touched and nothing is published: that stays a human act through `publish_sheet_version`.
|
||||
|
||||
use super::targets::fail;
|
||||
use crate::anatomy::svg::number::is_js_space;
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::legacy::types::AssetRecord;
|
||||
|
||||
pub const DOCUMENT_DOLLAR_TAG: &str = "$equine_doc$";
|
||||
|
||||
const SHEET_DESCRIPTION: &str =
|
||||
"Fiche anatomique officielle du cheval (import des données historiques, brouillon à valider)";
|
||||
|
||||
pub const DRAFT_CHANGE_NOTE: &str = "Révision de travail 4c (audit anatomique et ostéopathique) sur l’import des données historiques";
|
||||
|
||||
pub struct DraftSeedParams<'a> {
|
||||
pub sheet_id: &'a str,
|
||||
pub assets: &'a [AssetRecord],
|
||||
/// `document.json` text, embedded verbatim in a dollar-quoted literal.
|
||||
pub document_json: &'a str,
|
||||
pub change_note: &'a str,
|
||||
}
|
||||
|
||||
/// Only ever fed fixed values; doubled quotes keep it literal-safe.
|
||||
fn sql_text(value: &str) -> String {
|
||||
format!("'{}'", value.replace('\'', "''"))
|
||||
}
|
||||
|
||||
/// The second line still names the TypeScript generator: it is part of the frozen seed bytes
|
||||
/// (sha256 `c5a24312…ed1e`, the R5.1 ledger), so it stays as written; the generator is this
|
||||
/// module since R5.6 and `packages/anatomy` left at R5.7 (RUST.md §11.23).
|
||||
fn header() -> String {
|
||||
[
|
||||
"-- Official equine anatomical sheet: working revision, DRAFT.",
|
||||
"-- Generated by `pnpm import:equine` (packages/anatomy): do not edit by hand.",
|
||||
"-- Idempotent and converging. Never publishes: the version stays in status",
|
||||
"-- draft until a platform admin validates REVIEW-REPORT.md and publishes it.",
|
||||
]
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
fn sheet_insert(sheet_id: &str) -> String {
|
||||
[
|
||||
"insert into public.anatomical_sheets".to_owned(),
|
||||
" (id, species_id, name, slug, description, creator_user_id, is_official, visibility)"
|
||||
.to_owned(),
|
||||
"values (".to_owned(),
|
||||
format!(
|
||||
" {}, 'equine', 'Faunosteo / Cheval', 'faunosteo-cheval',",
|
||||
sql_text(sheet_id)
|
||||
),
|
||||
format!(" {},", sql_text(SHEET_DESCRIPTION)),
|
||||
" null, true, 'private'".to_owned(),
|
||||
")".to_owned(),
|
||||
"on conflict (id) do nothing;".to_owned(),
|
||||
]
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
fn asset_inserts(sheet_id: &str, assets: &[AssetRecord]) -> String {
|
||||
let rows: Vec<String> = assets
|
||||
.iter()
|
||||
.map(|a| {
|
||||
format!(
|
||||
" ({}, {}, {}, {}, 'image/svg+xml', {})",
|
||||
sql_text(&a.svg_asset_id),
|
||||
sql_text(sheet_id),
|
||||
sql_text(&a.sha256),
|
||||
a.byte_size,
|
||||
sql_text(&a.file)
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
[
|
||||
"insert into public.anatomical_sheet_assets".to_owned(),
|
||||
" (id, sheet_id, sha256, byte_size, content_type, file_name)".to_owned(),
|
||||
"values".to_owned(),
|
||||
rows.join(",\n"),
|
||||
"on conflict (sheet_id, sha256) do nothing;".to_owned(),
|
||||
]
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
/// `trimEnd`: the JavaScript white space and line terminators.
|
||||
fn trim_end_js(text: &str) -> &str {
|
||||
text.trim_end_matches(is_js_space)
|
||||
}
|
||||
|
||||
fn document_literal(document_json: &str) -> Result<String, PipelineError> {
|
||||
if document_json.contains(DOCUMENT_DOLLAR_TAG) {
|
||||
return Err(fail(format!(
|
||||
"document JSON contains the dollar-quote tag {DOCUMENT_DOLLAR_TAG}"
|
||||
)));
|
||||
}
|
||||
Ok(format!(
|
||||
"{DOCUMENT_DOLLAR_TAG}{}{DOCUMENT_DOLLAR_TAG}::jsonb",
|
||||
trim_end_js(document_json)
|
||||
))
|
||||
}
|
||||
|
||||
fn draft_insert(params: &DraftSeedParams<'_>, literal: &str) -> String {
|
||||
let sheet = sql_text(params.sheet_id);
|
||||
[
|
||||
"select private.insert_sheet_draft(".to_owned(),
|
||||
" null,".to_owned(),
|
||||
format!(" {sheet},"),
|
||||
format!(" {literal},"),
|
||||
format!(" {}", sql_text(params.change_note)),
|
||||
")".to_owned(),
|
||||
"where not exists (".to_owned(),
|
||||
format!(" select 1 from public.anatomical_sheet_versions where sheet_id = {sheet}"),
|
||||
");".to_owned(),
|
||||
]
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
/// Only when the sheet's single version is a draft whose content differs; the document
|
||||
/// literal is read once through the derived table.
|
||||
fn draft_converge(params: &DraftSeedParams<'_>, literal: &str) -> String {
|
||||
let sheet = sql_text(params.sheet_id);
|
||||
[
|
||||
"update public.anatomical_sheet_versions v".to_owned(),
|
||||
format!(
|
||||
"set content = d.content, change_note = {}",
|
||||
sql_text(params.change_note)
|
||||
),
|
||||
format!("from (select {literal} as content) d"),
|
||||
format!("where v.sheet_id = {sheet} and v.status = 'draft'"),
|
||||
" and v.content <> d.content".to_owned(),
|
||||
" and (select count(*) from public.anatomical_sheet_versions o".to_owned(),
|
||||
format!(" where o.sheet_id = {sheet}) = 1;"),
|
||||
]
|
||||
.join("\n")
|
||||
}
|
||||
|
||||
pub fn render_draft_seed_sql(params: &DraftSeedParams<'_>) -> Result<String, PipelineError> {
|
||||
let literal = document_literal(params.document_json)?;
|
||||
let parts = [
|
||||
header(),
|
||||
sheet_insert(params.sheet_id),
|
||||
asset_inserts(params.sheet_id, params.assets),
|
||||
draft_insert(params, &literal),
|
||||
draft_converge(params, &literal),
|
||||
];
|
||||
Ok(format!("{}\n", parts.join("\n\n")))
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
//! The JSON shapes Zod refused and serde's derives accept. serde reads a struct from a
|
||||
//! positional array (`"title": ["Titre", "Title"]`) and a unit enum variant from a one-key map
|
||||
//! (`"level": {"A": null}`); Zod refused both (`invalid_type`). The authored files and the
|
||||
//! curated document are therefore compared, by shape, with the typed value serialised back:
|
||||
//! an object must stay an object, an array an array of the same length, a scalar a scalar of
|
||||
//! the same kind. Keys the typed value adds (defaults filled in) are allowed.
|
||||
//!
|
||||
//! The R2 contracts carry the same derives; their readers are outside R5.3 (owner's go).
|
||||
|
||||
use serde::Serialize;
|
||||
use serde_json::Value;
|
||||
|
||||
use super::targets::fail;
|
||||
use crate::curation::PipelineError;
|
||||
|
||||
fn kind(value: &Value) -> &'static str {
|
||||
match value {
|
||||
Value::Null => "null",
|
||||
Value::Bool(_) => "a boolean",
|
||||
Value::Number(_) => "a number",
|
||||
Value::String(_) => "a string",
|
||||
Value::Array(_) => "an array",
|
||||
Value::Object(_) => "an object",
|
||||
}
|
||||
}
|
||||
|
||||
fn mismatch(path: &str, typed: &Value, written: &Value) -> String {
|
||||
let at = if path.is_empty() { "the root" } else { path };
|
||||
format!(
|
||||
"expected {} at {at}, received {}",
|
||||
kind(typed),
|
||||
kind(written)
|
||||
)
|
||||
}
|
||||
|
||||
/// The first place where `written` (as parsed) and `typed` (read and serialised back) differ
|
||||
/// in shape.
|
||||
fn first_mismatch(written: &Value, typed: &Value, path: &str) -> Option<String> {
|
||||
match (written, typed) {
|
||||
(Value::Object(w), Value::Object(t)) => w.iter().find_map(|(key, value)| {
|
||||
let here = format!("{path}.{key}");
|
||||
match t.get(key) {
|
||||
Some(typed) => first_mismatch(value, typed, &here),
|
||||
None => Some(format!("unexpected key at {here}")),
|
||||
}
|
||||
}),
|
||||
(Value::Array(w), Value::Array(t)) if w.len() != t.len() => Some(format!(
|
||||
"expected {} items at {path}, received {}",
|
||||
t.len(),
|
||||
w.len()
|
||||
)),
|
||||
(Value::Array(w), Value::Array(t)) => w
|
||||
.iter()
|
||||
.zip(t)
|
||||
.enumerate()
|
||||
.find_map(|(index, (w, t))| first_mismatch(w, t, &format!("{path}.{index}"))),
|
||||
_ if std::mem::discriminant(written) == std::mem::discriminant(typed) => None,
|
||||
_ => Some(mismatch(path, typed, written)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Refuses `written` when its typed reading `typed` does not keep its shape.
|
||||
pub fn assert_same_shape<T: Serialize>(
|
||||
what: &str,
|
||||
written: &Value,
|
||||
typed: &T,
|
||||
) -> Result<(), PipelineError> {
|
||||
let typed = serde_json::to_value(typed).map_err(|error| fail(format!("{what}: {error}")))?;
|
||||
match first_mismatch(written, &typed, "") {
|
||||
None => Ok(()),
|
||||
Some(why) => Err(fail(format!("{what}: {why}"))),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
//! Volumes of a sheet document (`packages/anatomy/src/curation/stats.ts`), compared before /
|
||||
//! after curation in the CHANGELOG and shown for the baseline in the review report. Nodes
|
||||
//! count finding categories too, as `countTree` does in the import report. Field order is the
|
||||
//! TypeScript object order, so the serialised stats compare with `oracles/r5/pipeline.json`.
|
||||
|
||||
use serde::Serialize;
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use super::records::js_string;
|
||||
use super::targets::{FindingEntry, all_findings, at};
|
||||
use crate::curation::PipelineError;
|
||||
|
||||
const ANNOTATION_KINDS: [&str; 3] = ["point", "zone", "detail_trigger"];
|
||||
const REVIEW_STATUSES: [&str; 3] = ["inherited", "reviewed", "to_confirm"];
|
||||
|
||||
/// `countBy`: the listed keys first (zero when absent), then any other value met.
|
||||
pub type Counts = Map<String, Value>;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct DocumentStats {
|
||||
pub views: usize,
|
||||
pub assets: usize,
|
||||
pub annotations: usize,
|
||||
pub annotations_by_kind: Counts,
|
||||
pub annotation_categories: usize,
|
||||
pub finding_zones: usize,
|
||||
pub finding_categories: usize,
|
||||
pub nodes: usize,
|
||||
pub leaves: usize,
|
||||
pub vet_referrals: usize,
|
||||
pub descriptions: usize,
|
||||
pub descriptions_fr_only: usize,
|
||||
pub cited_nodes: usize,
|
||||
pub nodes_by_review_status: Counts,
|
||||
pub references: usize,
|
||||
pub annotations_by_review_status: Counts,
|
||||
pub regions_with_count_range: usize,
|
||||
}
|
||||
|
||||
fn count_by<'a>(
|
||||
items: impl Iterator<Item = &'a Value>,
|
||||
keys: &[&str],
|
||||
key: impl Fn(&Value) -> String,
|
||||
) -> Counts {
|
||||
let mut counts = Map::new();
|
||||
for name in keys {
|
||||
counts.insert((*name).to_owned(), Value::from(0));
|
||||
}
|
||||
for item in items {
|
||||
let name = key(item);
|
||||
let current = counts.get(&name).and_then(Value::as_u64).unwrap_or(0);
|
||||
counts.insert(name, Value::from(current + 1));
|
||||
}
|
||||
counts
|
||||
}
|
||||
|
||||
/// `Boolean(text.fr)`: a non-empty string, a non-zero number, true, a non-empty object.
|
||||
fn truthy(value: Option<&Value>) -> bool {
|
||||
match value {
|
||||
None | Some(Value::Null) => false,
|
||||
Some(Value::Bool(flag)) => *flag,
|
||||
Some(Value::Number(number)) => number.as_f64().is_some_and(|n| n != 0.0),
|
||||
Some(Value::String(text)) => !text.is_empty(),
|
||||
Some(Value::Array(_) | Value::Object(_)) => true,
|
||||
}
|
||||
}
|
||||
|
||||
fn description(item: &Value) -> (bool, bool) {
|
||||
let text = item.get("description");
|
||||
(
|
||||
truthy(text.and_then(|t| t.get("fr"))),
|
||||
truthy(text.and_then(|t| t.get("en"))),
|
||||
)
|
||||
}
|
||||
|
||||
fn is_leaf(item: &Value) -> bool {
|
||||
item.get("children")
|
||||
.and_then(Value::as_array)
|
||||
.is_none_or(Vec::is_empty)
|
||||
}
|
||||
|
||||
fn is_cited(item: &Value) -> bool {
|
||||
item.get("reference_ids")
|
||||
.and_then(Value::as_array)
|
||||
.is_some_and(|ids| !ids.is_empty())
|
||||
}
|
||||
|
||||
/// `String(item.review_status ?? 'inherited')`.
|
||||
fn review_status(item: &Value) -> String {
|
||||
match item.get("review_status") {
|
||||
None | Some(Value::Null) => "inherited".to_owned(),
|
||||
Some(value) => js_string(Some(value)),
|
||||
}
|
||||
}
|
||||
|
||||
fn list<'a>(doc: &'a Value, key: &str) -> &'a [Value] {
|
||||
doc.get(key)
|
||||
.and_then(Value::as_array)
|
||||
.map_or(&[], Vec::as_slice)
|
||||
}
|
||||
|
||||
struct FindingStats {
|
||||
categories: usize,
|
||||
nodes: usize,
|
||||
leaves: usize,
|
||||
vet_referrals: usize,
|
||||
descriptions: usize,
|
||||
descriptions_fr_only: usize,
|
||||
cited: usize,
|
||||
by_review_status: Counts,
|
||||
}
|
||||
|
||||
fn finding_stats(doc: &Value, entries: &[FindingEntry]) -> Result<FindingStats, PipelineError> {
|
||||
let items: Vec<&Value> = entries
|
||||
.iter()
|
||||
.map(|entry| at(doc, &entry.path))
|
||||
.collect::<Result<_, _>>()?;
|
||||
let texts: Vec<(bool, bool)> = items.iter().map(|item| description(item)).collect();
|
||||
Ok(FindingStats {
|
||||
categories: entries.iter().filter(|entry| entry.is_category).count(),
|
||||
nodes: items.len(),
|
||||
leaves: items.iter().filter(|item| is_leaf(item)).count(),
|
||||
vet_referrals: items
|
||||
.iter()
|
||||
.filter(|item| item.get("requires_vet_referral") == Some(&Value::Bool(true)))
|
||||
.count(),
|
||||
descriptions: texts.iter().filter(|(fr, en)| *fr || *en).count(),
|
||||
descriptions_fr_only: texts.iter().filter(|(fr, en)| *fr && !*en).count(),
|
||||
cited: items.iter().filter(|item| is_cited(item)).count(),
|
||||
by_review_status: count_by(items.iter().copied(), &REVIEW_STATUSES, review_status),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn document_stats(doc: &Value) -> Result<DocumentStats, PipelineError> {
|
||||
let views = list(doc, "views");
|
||||
let mut assets: Vec<&Value> = Vec::new();
|
||||
for view in views {
|
||||
let asset = view.get("svg_asset_id");
|
||||
if truthy(asset) && !assets.contains(&asset.unwrap_or(&Value::Null)) {
|
||||
assets.push(asset.unwrap_or(&Value::Null));
|
||||
}
|
||||
}
|
||||
let annotations = list(doc, "annotations");
|
||||
let findings = finding_stats(doc, &all_findings(doc)?)?;
|
||||
let regions = doc
|
||||
.get("vertebral_config")
|
||||
.map(|config| list(config, "regions"))
|
||||
.unwrap_or(&[]);
|
||||
Ok(DocumentStats {
|
||||
views: views.len(),
|
||||
assets: assets.len(),
|
||||
annotations: annotations.len(),
|
||||
annotations_by_kind: count_by(annotations.iter(), &ANNOTATION_KINDS, |a| {
|
||||
js_string(a.get("kind"))
|
||||
}),
|
||||
annotation_categories: list(doc, "categories").len(),
|
||||
finding_zones: list(doc, "finding_zones").len(),
|
||||
finding_categories: findings.categories,
|
||||
nodes: findings.nodes,
|
||||
leaves: findings.leaves,
|
||||
vet_referrals: findings.vet_referrals,
|
||||
descriptions: findings.descriptions,
|
||||
descriptions_fr_only: findings.descriptions_fr_only,
|
||||
cited_nodes: findings.cited,
|
||||
nodes_by_review_status: findings.by_review_status,
|
||||
references: list(doc, "references").len(),
|
||||
annotations_by_review_status: count_by(annotations.iter(), &REVIEW_STATUSES, review_status),
|
||||
regions_with_count_range: regions
|
||||
.iter()
|
||||
.filter(|region| truthy(region.get("count_range")))
|
||||
.count(),
|
||||
})
|
||||
}
|
||||
|
||||
fn count_rows(label: &str, counts: &Counts) -> Vec<(String, usize)> {
|
||||
counts
|
||||
.iter()
|
||||
.map(|(key, value)| {
|
||||
let count = value
|
||||
.as_u64()
|
||||
.map_or(0, |n| usize::try_from(n).unwrap_or(usize::MAX));
|
||||
(format!("{label} ({key})"), count)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// `statRows`: labelled rows, fixed order, French labels.
|
||||
pub fn stat_rows(stats: &DocumentStats) -> Vec<(String, usize)> {
|
||||
let row = |label: &str, value: usize| vec![(label.to_owned(), value)];
|
||||
[
|
||||
row("Vues", stats.views),
|
||||
row("Dessins SVG distincts", stats.assets),
|
||||
row("Annotations", stats.annotations),
|
||||
count_rows("Annotations", &stats.annotations_by_kind),
|
||||
count_rows("Annotations, revue", &stats.annotations_by_review_status),
|
||||
row("Catégories d'annotation", stats.annotation_categories),
|
||||
row("Zones de constats", stats.finding_zones),
|
||||
row("Catégories de constats", stats.finding_categories),
|
||||
row("Nœuds (catégories incluses)", stats.nodes),
|
||||
row("Feuilles", stats.leaves),
|
||||
row("Renvois vétérinaires", stats.vet_referrals),
|
||||
count_rows("Nœuds, revue", &stats.nodes_by_review_status),
|
||||
row("Descriptions", stats.descriptions),
|
||||
row(
|
||||
"Descriptions en français seulement",
|
||||
stats.descriptions_fr_only,
|
||||
),
|
||||
row("Références bibliographiques", stats.references),
|
||||
row("Nœuds cités", stats.cited_nodes),
|
||||
row(
|
||||
"Régions vertébrales avec plage documentée",
|
||||
stats.regions_with_count_range,
|
||||
),
|
||||
]
|
||||
.concat()
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
//! How a produced review drawing is compared with the committed xmldom file: byte for byte,
|
||||
//! else byte for byte after a declared rule applied to the committed file's tree (strict: the
|
||||
//! committed file must rebuild to itself from its tree, so nothing outside the tree can hide). One rule exists: the R3 owner decision
|
||||
//! (2026-10-15) that a trigger without a name of its own carries the opened view's name in
|
||||
//! each language, where the TypeScript renderer wrote the French name twice. It is applied to
|
||||
//! the committed tree (the frozen file stays as it is): on an element that opens a view, a
|
||||
//! `data-anno-name-en` equal to the view's French name takes its English name, and a title
|
||||
//! `FR / FR` under it reads `FR / EN`. Any other difference is reported.
|
||||
|
||||
use crate::anatomy::svg::serialize::serialize;
|
||||
use crate::anatomy::svg::tree::parse_svg;
|
||||
use crate::anatomy::svg::{Element, Node};
|
||||
use crate::domain::sheet::AnatomicalSheetDocument;
|
||||
|
||||
/// (view id, French name, English name) of every view of the produced document.
|
||||
pub struct TriggerNames {
|
||||
views: Vec<(String, String, String)>,
|
||||
}
|
||||
|
||||
impl TriggerNames {
|
||||
pub fn of(document: &AnatomicalSheetDocument) -> Self {
|
||||
let views = document
|
||||
.views
|
||||
.iter()
|
||||
.map(|view| {
|
||||
(
|
||||
view.id().to_owned(),
|
||||
view.name().fr.clone(),
|
||||
view.name().en.clone(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
Self { views }
|
||||
}
|
||||
|
||||
fn apply(&self, element: &mut Element) {
|
||||
let opened = element.attr("data-anno-opens-view").map(str::to_owned);
|
||||
let view = opened.and_then(|id| self.views.iter().find(|(v, _, _)| *v == id));
|
||||
if let Some((_, fr, en)) = view {
|
||||
if element.attr("data-anno-name-en") == Some(fr.as_str()) {
|
||||
element.set_attr("data-anno-name-en", en.clone());
|
||||
}
|
||||
rename_titles(element, &format!("{fr} / {fr}"), &format!("{fr} / {en}"));
|
||||
}
|
||||
for child in &mut element.children {
|
||||
if let Node::Element(child) = child {
|
||||
self.apply(child);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn rename_titles(element: &mut Element, from: &str, to: &str) {
|
||||
for child in &mut element.children {
|
||||
let Node::Element(child) = child else {
|
||||
continue;
|
||||
};
|
||||
if child.local_name() == "title" {
|
||||
for text in &mut child.children {
|
||||
if let Node::Text(value) = text
|
||||
&& value == from
|
||||
{
|
||||
*value = to.to_owned();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// How two drawings were found equal.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum Match {
|
||||
Bytes,
|
||||
TreeAfterRule,
|
||||
}
|
||||
|
||||
/// The first place two trees differ, as a path of element names.
|
||||
fn first_difference(a: &Element, b: &Element, path: &str) -> Option<String> {
|
||||
let here = format!("{path}/{}", a.qname);
|
||||
if a.qname != b.qname || a.namespace != b.namespace {
|
||||
return Some(format!("{here}: element {} vs {}", a.qname, b.qname));
|
||||
}
|
||||
if a.attrs != b.attrs {
|
||||
return Some(format!("{here}: attributes differ"));
|
||||
}
|
||||
if a.children.len() != b.children.len() {
|
||||
return Some(format!(
|
||||
"{here}: {} vs {} children",
|
||||
a.children.len(),
|
||||
b.children.len()
|
||||
));
|
||||
}
|
||||
a.children
|
||||
.iter()
|
||||
.zip(&b.children)
|
||||
.enumerate()
|
||||
.find_map(|(index, pair)| match pair {
|
||||
(Node::Element(x), Node::Element(y)) => {
|
||||
first_difference(x, y, &format!("{here}[{index}]"))
|
||||
}
|
||||
(Node::Text(x), Node::Text(y)) if x == y => None,
|
||||
_ => Some(format!("{here}[{index}]: text or node kind differs")),
|
||||
})
|
||||
}
|
||||
|
||||
/// The prolog (the bytes before the root element) and the text after it.
|
||||
fn outside_root(text: &str) -> Result<(&str, &str), String> {
|
||||
let document = roxmltree::Document::parse(text).map_err(|error| error.to_string())?;
|
||||
let range = document.root_element().range();
|
||||
Ok((&text[..range.start], &text[range.end..]))
|
||||
}
|
||||
|
||||
/// Compares a produced drawing with the committed one: byte identity, else the committed file
|
||||
/// rebuilt from its tree must be byte-identical to itself (so the tree carries all of it: no
|
||||
/// comment, processing instruction, encoding or entity form the comparison could miss), the
|
||||
/// rule must change that tree, and the rebuilt text after the rule must be byte-identical to
|
||||
/// the produced one.
|
||||
pub fn compare_drawing(
|
||||
produced: &str,
|
||||
committed: &str,
|
||||
rule: Option<&TriggerNames>,
|
||||
) -> Result<Match, String> {
|
||||
if produced == committed {
|
||||
return Ok(Match::Bytes);
|
||||
}
|
||||
let Some(rule) = rule else {
|
||||
return Err("bytes differ and no tree rule is declared".to_owned());
|
||||
};
|
||||
// The core reader first: its prescan bounds the depth before any recursive parse.
|
||||
let tree = parse_svg(committed).map_err(|e| format!("committed: {}", e.reason()))?;
|
||||
let (prolog, epilog) = outside_root(committed).map_err(|e| format!("committed: {e}"))?;
|
||||
if format!("{prolog}{}{epilog}", serialize(&tree)) != committed {
|
||||
return Err("the committed file holds what its tree does not carry".to_owned());
|
||||
}
|
||||
let mut ruled = tree.clone();
|
||||
rule.apply(&mut ruled);
|
||||
if ruled == tree {
|
||||
return Err("bytes differ and the declared rule changes nothing".to_owned());
|
||||
}
|
||||
if format!("{prolog}{}{epilog}", serialize(&ruled)) == produced {
|
||||
return Ok(Match::TreeAfterRule);
|
||||
}
|
||||
let produced_tree = parse_svg(produced).map_err(|e| format!("produced: {}", e.reason()))?;
|
||||
Err(first_difference(&produced_tree, &ruled, "")
|
||||
.unwrap_or_else(|| "the texts differ outside the trees".to_owned()))
|
||||
}
|
||||
@@ -0,0 +1,424 @@
|
||||
//! Target grammar of the changeset (`packages/anatomy/src/curation/targets.ts`), one string
|
||||
//! per target:
|
||||
//!
|
||||
//! document | region:<code> | reference:<id> | category:<id> | view:<id>
|
||||
//! annotation:<id> | zone:<zoneId> | finding:<zoneId>/<catId>[/<nodeId>…]
|
||||
//!
|
||||
//! A finding path walks ids from the zone down; ids are unique among siblings, so the path
|
||||
//! names one category or node. Resolution works on the plain JSON form of the document
|
||||
//! (`serde_json::Value`, insertion order kept): the changeset reaches any field without one
|
||||
//! accessor per type, and the final strict parse checks the shape.
|
||||
//!
|
||||
//! The TypeScript held object references; here a located item is a path of keys and indices
|
||||
//! into the document, resolved again at each use (`item_mut`, `siblings_mut`).
|
||||
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use crate::curation::PipelineError;
|
||||
|
||||
pub type Item = Map<String, Value>;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum TargetKind {
|
||||
Document,
|
||||
Region,
|
||||
Reference,
|
||||
Category,
|
||||
View,
|
||||
Annotation,
|
||||
Zone,
|
||||
Finding,
|
||||
}
|
||||
|
||||
impl TargetKind {
|
||||
pub fn name(self) -> &'static str {
|
||||
match self {
|
||||
Self::Document => "document",
|
||||
Self::Region => "region",
|
||||
Self::Reference => "reference",
|
||||
Self::Category => "category",
|
||||
Self::View => "view",
|
||||
Self::Annotation => "annotation",
|
||||
Self::Zone => "zone",
|
||||
Self::Finding => "finding",
|
||||
}
|
||||
}
|
||||
|
||||
fn collection(name: &str) -> Option<Self> {
|
||||
Some(match name {
|
||||
"region" => Self::Region,
|
||||
"reference" => Self::Reference,
|
||||
"category" => Self::Category,
|
||||
"view" => Self::View,
|
||||
"annotation" => Self::Annotation,
|
||||
"zone" => Self::Zone,
|
||||
"finding" => Self::Finding,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
|
||||
/// Regions are keyed by their free code, everything else by a stable id.
|
||||
pub fn key_field(self) -> &'static str {
|
||||
if self == Self::Region { "code" } else { "id" }
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ParsedTarget {
|
||||
pub kind: TargetKind,
|
||||
pub ids: Vec<String>,
|
||||
}
|
||||
|
||||
/// One step into the document: an object key or an array index.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum Step {
|
||||
Key(&'static str),
|
||||
Index(usize),
|
||||
}
|
||||
|
||||
pub type Path = Vec<Step>;
|
||||
|
||||
/// A resolved target: the path of the list that holds the item (None: the document root) and
|
||||
/// the item's index in it.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Located {
|
||||
pub target: ParsedTarget,
|
||||
pub list: Option<Path>,
|
||||
pub index: usize,
|
||||
}
|
||||
|
||||
impl Located {
|
||||
pub fn item_path(&self) -> Path {
|
||||
let mut path = self.list.clone().unwrap_or_default();
|
||||
if self.list.is_some() {
|
||||
path.push(Step::Index(self.index));
|
||||
}
|
||||
path
|
||||
}
|
||||
}
|
||||
|
||||
/// One finding category or node met while walking a zone's tree.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct FindingEntry {
|
||||
pub target: String,
|
||||
pub path: Path,
|
||||
pub is_category: bool,
|
||||
}
|
||||
|
||||
pub fn fail(message: impl Into<String>) -> PipelineError {
|
||||
PipelineError::Guard(message.into())
|
||||
}
|
||||
|
||||
pub fn parse_target(target: &str) -> Result<ParsedTarget, PipelineError> {
|
||||
if target == "document" {
|
||||
return Ok(ParsedTarget {
|
||||
kind: TargetKind::Document,
|
||||
ids: Vec::new(),
|
||||
});
|
||||
}
|
||||
let unknown = || fail(format!("unknown target \"{target}\""));
|
||||
let colon = target.find(':').ok_or_else(unknown)?;
|
||||
let (kind, rest) = (&target[..colon], &target[colon + 1..]);
|
||||
if colon < 1 || rest.is_empty() {
|
||||
return Err(unknown());
|
||||
}
|
||||
let kind = TargetKind::collection(kind).ok_or_else(unknown)?;
|
||||
let ids: Vec<String> = if kind == TargetKind::Finding {
|
||||
rest.split('/').map(str::to_owned).collect()
|
||||
} else {
|
||||
vec![rest.to_owned()]
|
||||
};
|
||||
if ids.iter().any(String::is_empty) || (kind == TargetKind::Finding && ids.len() < 2) {
|
||||
return Err(fail(format!("malformed target \"{target}\"")));
|
||||
}
|
||||
Ok(ParsedTarget { kind, ids })
|
||||
}
|
||||
|
||||
pub fn format_target(target: &ParsedTarget) -> String {
|
||||
if target.kind == TargetKind::Document {
|
||||
return "document".to_owned();
|
||||
}
|
||||
format!("{}:{}", target.kind.name(), target.ids.join("/"))
|
||||
}
|
||||
|
||||
/// Path of the top-level list holding the items of a target kind.
|
||||
pub fn top_collection_path(kind: TargetKind) -> Path {
|
||||
match kind {
|
||||
TargetKind::Region => vec![Step::Key("vertebral_config"), Step::Key("regions")],
|
||||
TargetKind::Reference => vec![Step::Key("references")],
|
||||
TargetKind::Category => vec![Step::Key("categories")],
|
||||
TargetKind::View => vec![Step::Key("views")],
|
||||
TargetKind::Annotation => vec![Step::Key("annotations")],
|
||||
TargetKind::Zone | TargetKind::Finding | TargetKind::Document => {
|
||||
vec![Step::Key("finding_zones")]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn describe(step: &Step) -> String {
|
||||
match step {
|
||||
Step::Key(key) => (*key).to_owned(),
|
||||
Step::Index(index) => index.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The value at a path.
|
||||
pub fn at<'a>(doc: &'a Value, path: &[Step]) -> Result<&'a Value, PipelineError> {
|
||||
let mut value = doc;
|
||||
for step in path {
|
||||
value = match step {
|
||||
Step::Key(key) => value.get(key),
|
||||
Step::Index(index) => value.get(index),
|
||||
}
|
||||
.ok_or_else(|| fail(format!("document has no \"{}\"", describe(step))))?;
|
||||
}
|
||||
Ok(value)
|
||||
}
|
||||
|
||||
pub fn at_mut<'a>(doc: &'a mut Value, path: &[Step]) -> Result<&'a mut Value, PipelineError> {
|
||||
let mut value = doc;
|
||||
for step in path {
|
||||
value = match step {
|
||||
Step::Key(key) => value.get_mut(key),
|
||||
Step::Index(index) => value.get_mut(index),
|
||||
}
|
||||
.ok_or_else(|| fail(format!("document has no \"{}\"", describe(step))))?;
|
||||
}
|
||||
Ok(value)
|
||||
}
|
||||
|
||||
/// `listAt`: the array at a path, or "document has no list".
|
||||
pub fn list_at<'a>(doc: &'a Value, path: &[Step]) -> Result<&'a Vec<Value>, PipelineError> {
|
||||
let last = path.last().map(describe).unwrap_or_default();
|
||||
let no_list = || fail(format!("document has no list \"{last}\""));
|
||||
at(doc, path)
|
||||
.map_err(|_| no_list())?
|
||||
.as_array()
|
||||
.ok_or_else(no_list)
|
||||
}
|
||||
|
||||
pub fn list_at_mut<'a>(
|
||||
doc: &'a mut Value,
|
||||
path: &[Step],
|
||||
) -> Result<&'a mut Vec<Value>, PipelineError> {
|
||||
let last = path.last().map(describe).unwrap_or_default();
|
||||
let no_list = || fail(format!("document has no list \"{last}\""));
|
||||
at_mut(doc, path)
|
||||
.map_err(|_| no_list())?
|
||||
.as_array_mut()
|
||||
.ok_or_else(no_list)
|
||||
}
|
||||
|
||||
pub fn item_at<'a>(doc: &'a Value, path: &[Step]) -> Result<&'a Item, PipelineError> {
|
||||
at(doc, path)?
|
||||
.as_object()
|
||||
.ok_or_else(|| fail("target is not an object"))
|
||||
}
|
||||
|
||||
pub fn item_at_mut<'a>(doc: &'a mut Value, path: &[Step]) -> Result<&'a mut Item, PipelineError> {
|
||||
at_mut(doc, path)?
|
||||
.as_object_mut()
|
||||
.ok_or_else(|| fail("target is not an object"))
|
||||
}
|
||||
|
||||
/// `entry[key] === id` on an element of a list.
|
||||
pub fn has_key(entry: &Value, key: &str, id: &str) -> bool {
|
||||
entry.get(key).and_then(Value::as_str) == Some(id)
|
||||
}
|
||||
|
||||
/// `findIn`: the index of the sibling whose key is `id`.
|
||||
fn find_in(list: &[Value], key: &str, id: &str, raw: &str) -> Result<usize, PipelineError> {
|
||||
list.iter()
|
||||
.position(|entry| has_key(entry, key, id))
|
||||
.ok_or_else(|| fail(format!("unknown target \"{raw}\" (no \"{id}\")")))
|
||||
}
|
||||
|
||||
fn with(path: &[Step], step: Step) -> Path {
|
||||
let mut next = path.to_vec();
|
||||
next.push(step);
|
||||
next
|
||||
}
|
||||
|
||||
/// `childrenOf`: children of a finding item; a lean leaf gets an empty list on demand (the key
|
||||
/// is appended to the item, as the TypeScript did); a present value that is not a list is
|
||||
/// refused, never replaced.
|
||||
pub fn children_path(doc: &mut Value, item: &[Step]) -> Result<Path, PipelineError> {
|
||||
let holder = item_at_mut(doc, item)?;
|
||||
match holder.get("children") {
|
||||
None => {
|
||||
holder.insert("children".into(), Value::Array(Vec::new()));
|
||||
}
|
||||
Some(Value::Array(_)) => {}
|
||||
// Never replaced: the author's value would be lost (refused at the add already).
|
||||
Some(_) => return Err(fail("\"children\" is not a list")),
|
||||
}
|
||||
Ok(with(item, Step::Key("children")))
|
||||
}
|
||||
|
||||
fn locate_finding(doc: &Value, target: ParsedTarget, raw: &str) -> Result<Located, PipelineError> {
|
||||
let zones = top_collection_path(TargetKind::Zone);
|
||||
let zone = find_in(list_at(doc, &zones)?, "id", &target.ids[0], raw)?;
|
||||
let mut list = with(&with(&zones, Step::Index(zone)), Step::Key("categories"));
|
||||
let mut index = find_in(list_at(doc, &list)?, "id", &target.ids[1], raw)?;
|
||||
for id in &target.ids[2..] {
|
||||
list = with(&with(&list, Step::Index(index)), Step::Key("children"));
|
||||
let siblings = at(doc, &list).ok().and_then(Value::as_array);
|
||||
index = find_in(siblings.map_or(&[][..], Vec::as_slice), "id", id, raw)?;
|
||||
}
|
||||
Ok(Located {
|
||||
target,
|
||||
list: Some(list),
|
||||
index,
|
||||
})
|
||||
}
|
||||
|
||||
/// Resolves a target string or refuses with "unknown target".
|
||||
pub fn locate(doc: &Value, raw: &str) -> Result<Located, PipelineError> {
|
||||
let target = parse_target(raw)?;
|
||||
if target.kind == TargetKind::Document {
|
||||
return Ok(Located {
|
||||
target,
|
||||
list: None,
|
||||
index: 0,
|
||||
});
|
||||
}
|
||||
if target.kind == TargetKind::Finding {
|
||||
return locate_finding(doc, target, raw);
|
||||
}
|
||||
let list = top_collection_path(target.kind);
|
||||
let index = find_in(
|
||||
list_at(doc, &list)?,
|
||||
target.kind.key_field(),
|
||||
&target.ids[0],
|
||||
raw,
|
||||
)?;
|
||||
Ok(Located {
|
||||
target,
|
||||
list: Some(list),
|
||||
index,
|
||||
})
|
||||
}
|
||||
|
||||
/// `String(child.id)` of a finding item.
|
||||
pub fn id_text(item: &Value) -> String {
|
||||
super::records::js_string(item.get("id"))
|
||||
}
|
||||
|
||||
/// The item and every descendant node, depth first, in document order.
|
||||
fn walk(doc: &Value, path: &Path, target: &str, depth: usize, out: &mut Vec<FindingEntry>) {
|
||||
out.push(FindingEntry {
|
||||
target: target.to_owned(),
|
||||
path: path.clone(),
|
||||
is_category: depth == 0,
|
||||
});
|
||||
let children = at(doc, path)
|
||||
.ok()
|
||||
.and_then(|item| item.get("children"))
|
||||
.and_then(Value::as_array);
|
||||
let Some(children) = children else {
|
||||
return;
|
||||
};
|
||||
let list = with(path, Step::Key("children"));
|
||||
for (index, child) in children.iter().enumerate() {
|
||||
let next = with(&list, Step::Index(index));
|
||||
walk(
|
||||
doc,
|
||||
&next,
|
||||
&format!("{target}/{}", id_text(child)),
|
||||
depth + 1,
|
||||
out,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Every finding category and node of a zone, depth first.
|
||||
pub fn zone_findings(doc: &Value, zone: &Path) -> Result<Vec<FindingEntry>, PipelineError> {
|
||||
let zone_id = id_text(at(doc, zone)?);
|
||||
let categories = with(zone, Step::Key("categories"));
|
||||
let mut out = Vec::new();
|
||||
for (index, category) in list_at(doc, &categories)?.iter().enumerate() {
|
||||
let target = format!("finding:{zone_id}/{}", id_text(category));
|
||||
walk(
|
||||
doc,
|
||||
&with(&categories, Step::Index(index)),
|
||||
&target,
|
||||
0,
|
||||
&mut out,
|
||||
);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Every finding category and node of the document.
|
||||
pub fn all_findings(doc: &Value) -> Result<Vec<FindingEntry>, PipelineError> {
|
||||
let zones = top_collection_path(TargetKind::Zone);
|
||||
let mut out = Vec::new();
|
||||
for index in 0..list_at(doc, &zones)?.len() {
|
||||
out.extend(zone_findings(doc, &with(&zones, Step::Index(index)))?);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Finding items an op touches. A finding target alone (self) or with its descendants
|
||||
/// (subtree); a zone target stands for its categories alone (self) or its whole tree
|
||||
/// (subtree): a zone carries no citation.
|
||||
pub fn finding_scope(
|
||||
doc: &Value,
|
||||
located: &Located,
|
||||
subtree: bool,
|
||||
) -> Result<Vec<FindingEntry>, PipelineError> {
|
||||
let item = located.item_path();
|
||||
match located.target.kind {
|
||||
TargetKind::Zone => {
|
||||
let entries = zone_findings(doc, &item)?;
|
||||
Ok(if subtree {
|
||||
entries
|
||||
} else {
|
||||
entries.into_iter().filter(|e| e.is_category).collect()
|
||||
})
|
||||
}
|
||||
TargetKind::Finding => {
|
||||
let depth = usize::from(located.target.ids.len() != 2);
|
||||
let mut entries = Vec::new();
|
||||
walk(
|
||||
doc,
|
||||
&item,
|
||||
&format_target(&located.target),
|
||||
depth,
|
||||
&mut entries,
|
||||
);
|
||||
if !subtree {
|
||||
entries.truncate(1);
|
||||
}
|
||||
Ok(entries)
|
||||
}
|
||||
_ => Err(fail(format!(
|
||||
"\"{}\" is not a finding or a zone",
|
||||
format_target(&located.target)
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
/// The list an add or a move inserts into under a zone or a finding.
|
||||
pub fn child_list(doc: &mut Value, parent: &Located) -> Result<Path, PipelineError> {
|
||||
match parent.target.kind {
|
||||
TargetKind::Zone => {
|
||||
let path = with(&parent.item_path(), Step::Key("categories"));
|
||||
list_at(doc, &path)?;
|
||||
Ok(path)
|
||||
}
|
||||
TargetKind::Finding => children_path(doc, &parent.item_path()),
|
||||
_ => Err(fail(format!(
|
||||
"\"{}\" holds no findings",
|
||||
format_target(&parent.target)
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Target prefix of the items of `child_list(parent)`.
|
||||
pub fn child_prefix(parent: &Located) -> String {
|
||||
if parent.target.kind == TargetKind::Zone {
|
||||
return format!("finding:{}/", parent.target.ids[0]);
|
||||
}
|
||||
format!("{}/", format_target(&parent.target))
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
//! Deterministic identifiers of `packages/anatomy/src/legacy/ids.ts`: nothing random or
|
||||
//! time-based, so the importer emits the same bytes on every run.
|
||||
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
/// Lowercase hex sha256 of bytes (a string hashes as its UTF-8 bytes, as `createHash` does).
|
||||
pub fn sha256_hex(data: &[u8]) -> String {
|
||||
Sha256::digest(data)
|
||||
.iter()
|
||||
.map(|byte| format!("{byte:02x}"))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// RFC 4122 version 5 style with SHA-256 in place of SHA-1: the first 16 bytes of
|
||||
/// sha256(name), the version nibble forced to 5, the variant bits to 10xx.
|
||||
pub fn deterministic_uuid(name: &str) -> String {
|
||||
let digest = Sha256::digest(name.as_bytes());
|
||||
let mut bytes = [0u8; 16];
|
||||
bytes.copy_from_slice(&digest[..16]);
|
||||
bytes[6] = (bytes[6] & 0x0f) | 0x50;
|
||||
bytes[8] = (bytes[8] & 0x3f) | 0x80;
|
||||
let hex: String = bytes.iter().map(|byte| format!("{byte:02x}")).collect();
|
||||
format!(
|
||||
"{}-{}-{}-{}-{}",
|
||||
&hex[0..8],
|
||||
&hex[8..12],
|
||||
&hex[12..16],
|
||||
&hex[16..20],
|
||||
&hex[20..32]
|
||||
)
|
||||
}
|
||||
|
||||
/// Same bytes, same asset id.
|
||||
pub fn asset_id_from_sha256(file_sha256: &str) -> String {
|
||||
deterministic_uuid(&format!("faunosteo:asset:{file_sha256}"))
|
||||
}
|
||||
|
||||
pub fn sheet_id_for(species_id: &str, version: u32) -> String {
|
||||
deterministic_uuid(&format!("faunosteo:sheet:{species_id}:v{version}"))
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
//! Legacy SVG annotation → canonical annotation (`packages/anatomy/src/legacy/annotations.ts`).
|
||||
//! The finding zone link is resolved here, once, by exact FR + EN equality with the catalog;
|
||||
//! the document then carries `finding_zone_id` and nothing matches labels at runtime.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use super::categories::normalise_category_id;
|
||||
use super::regions::infer_region_code;
|
||||
use super::svg::{LegacyAnnotation, LegacyGeometry};
|
||||
use super::types::{AnnotationRow, Anomaly, RegionInference};
|
||||
use crate::curation::text::slugify;
|
||||
|
||||
pub struct AnnotationContext<'a> {
|
||||
pub view_id: &'a str,
|
||||
/// `${name_fr}\n${name_en}` → catalog zone key (a JavaScript `Map`: the last zone wins).
|
||||
pub zone_by_names: &'a HashMap<String, String>,
|
||||
}
|
||||
|
||||
pub struct MappedAnnotation {
|
||||
pub annotation: Value,
|
||||
pub row: AnnotationRow,
|
||||
pub region: Option<RegionInference>,
|
||||
pub anomalies: Vec<Anomaly>,
|
||||
}
|
||||
|
||||
pub fn zone_name_key(fr: &str, en: &str) -> String {
|
||||
format!("{fr}\n{en}")
|
||||
}
|
||||
|
||||
fn laterality(a: &LegacyAnnotation, out: &mut Vec<Anomaly>) -> &'static str {
|
||||
match a.lateral.as_deref() {
|
||||
Some("bilateral") => "bilateral",
|
||||
None | Some("unpaired") => "unpaired",
|
||||
Some(other) => {
|
||||
out.push(Anomaly::to_validate(
|
||||
"annotation.laterality.unknown",
|
||||
a.id.clone(),
|
||||
format!("Latéralité « {other} » inconnue, importée en « unpaired »."),
|
||||
));
|
||||
"unpaired"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn region_of(a: &LegacyAnnotation, out: &mut Vec<Anomaly>) -> Option<RegionInference> {
|
||||
let special = a.special.as_deref()?;
|
||||
if special != "vertebral-precision" {
|
||||
out.push(Anomaly::to_validate(
|
||||
"annotation.special.unknown",
|
||||
a.id.clone(),
|
||||
format!("special « {special} » ignoré."),
|
||||
));
|
||||
return None;
|
||||
}
|
||||
let found = infer_region_code(a.name_fr.as_deref(), a.name_en.as_deref());
|
||||
if found.is_none() {
|
||||
out.push(Anomaly::to_validate(
|
||||
"annotation.region.unknown",
|
||||
a.id.clone(),
|
||||
"Région vertébrale non déduite.",
|
||||
));
|
||||
}
|
||||
let fallback = a
|
||||
.name_fr
|
||||
.clone()
|
||||
.or_else(|| a.name_en.clone())
|
||||
.unwrap_or_else(|| a.id.clone());
|
||||
Some(RegionInference {
|
||||
annotation_id: a.id.clone(),
|
||||
label: found.as_ref().map_or(fallback, |m| m.label.clone()),
|
||||
root: found.as_ref().map(|m| m.root.clone()),
|
||||
code: found.map(|m| m.code),
|
||||
})
|
||||
}
|
||||
|
||||
fn geometry(a: &LegacyAnnotation) -> Value {
|
||||
a.geometry
|
||||
.as_ref()
|
||||
.map_or(Value::Null, LegacyGeometry::to_value)
|
||||
}
|
||||
|
||||
fn category_id(a: &LegacyAnnotation) -> String {
|
||||
normalise_category_id(a.category.as_deref().unwrap_or(""))
|
||||
}
|
||||
|
||||
fn trigger_name(a: &LegacyAnnotation, out: &mut Vec<Anomaly>) -> Value {
|
||||
match (a.name_fr.as_deref(), a.name_en.as_deref()) {
|
||||
(Some(fr), Some(en)) if !fr.is_empty() && !en.is_empty() => json!({ "fr": fr, "en": en }),
|
||||
_ => {
|
||||
out.push(Anomaly::informative(
|
||||
"annotation.trigger.unnamed",
|
||||
a.id.clone(),
|
||||
"Déclencheur de vue détail sans nom (name null), comme dans le SVG.",
|
||||
));
|
||||
Value::Null
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn map_trigger(a: &LegacyAnnotation, ctx: &AnnotationContext, out: &mut Vec<Anomaly>) -> Value {
|
||||
let opens = a.opens_detail.clone().unwrap_or_default();
|
||||
let laterality = laterality(a, out);
|
||||
let name = trigger_name(a, out);
|
||||
json!({
|
||||
"id": format!("{opens}-trigger"),
|
||||
"kind": "detail_trigger",
|
||||
"view_id": ctx.view_id,
|
||||
"category_id": category_id(a),
|
||||
"laterality": laterality,
|
||||
"geometry": geometry(a),
|
||||
"name": name,
|
||||
"opens_view_id": opens,
|
||||
})
|
||||
}
|
||||
|
||||
/// Missing labels fall back to the other language, never silently.
|
||||
fn observable_name(a: &LegacyAnnotation, out: &mut Vec<Anomaly>) -> (String, String) {
|
||||
let fr = a
|
||||
.name_fr
|
||||
.clone()
|
||||
.or_else(|| a.name_en.clone())
|
||||
.unwrap_or_else(|| a.id.clone());
|
||||
if a.name_fr.is_none() {
|
||||
out.push(Anomaly::to_validate(
|
||||
"annotation.name_fr.missing",
|
||||
a.id.clone(),
|
||||
format!("data-anno-name-fr absent : nom FR repris de « {fr} »."),
|
||||
));
|
||||
}
|
||||
if a.name_en.is_none() {
|
||||
out.push(Anomaly::to_validate(
|
||||
"annotation.name_en.missing",
|
||||
a.id.clone(),
|
||||
format!("data-anno-name-en absent : nom EN copié du FR « {fr} », à traduire."),
|
||||
));
|
||||
}
|
||||
let en = a.name_en.clone().unwrap_or_else(|| fr.clone());
|
||||
(fr, en)
|
||||
}
|
||||
|
||||
fn map_observable(
|
||||
a: &LegacyAnnotation,
|
||||
ctx: &AnnotationContext,
|
||||
out: &mut Vec<Anomaly>,
|
||||
) -> (Value, Option<RegionInference>) {
|
||||
let (fr, en) = observable_name(a, out);
|
||||
let zone_id = ctx.zone_by_names.get(&zone_name_key(&fr, &en)).cloned();
|
||||
if zone_id.is_none() {
|
||||
out.push(Anomaly::to_validate(
|
||||
"annotation.unmapped",
|
||||
a.id.clone(),
|
||||
"Annotation non reliée à une zone de constats.",
|
||||
));
|
||||
}
|
||||
let region = region_of(a, out);
|
||||
let laterality = laterality(a, out);
|
||||
let annotation = json!({
|
||||
"id": zone_id.clone().unwrap_or_else(|| slugify(&en)),
|
||||
"kind": if a.tag_name == "circle" { "point" } else { "zone" },
|
||||
"view_id": ctx.view_id,
|
||||
"category_id": category_id(a),
|
||||
"laterality": laterality,
|
||||
"geometry": geometry(a),
|
||||
"name": { "fr": fr, "en": en },
|
||||
"finding_zone_id": zone_id,
|
||||
"vertebral_region_code": region.as_ref().and_then(|r| r.code.clone()),
|
||||
});
|
||||
(annotation, region)
|
||||
}
|
||||
|
||||
fn text(value: &Value, key: &str) -> Option<String> {
|
||||
value.get(key).and_then(Value::as_str).map(str::to_owned)
|
||||
}
|
||||
|
||||
pub fn map_annotation(a: &LegacyAnnotation, ctx: &AnnotationContext) -> MappedAnnotation {
|
||||
let mut anomalies = Vec::new();
|
||||
let is_trigger = a
|
||||
.opens_detail
|
||||
.as_deref()
|
||||
.is_some_and(|opens| !opens.is_empty());
|
||||
let (annotation, region) = if is_trigger {
|
||||
(map_trigger(a, ctx, &mut anomalies), None)
|
||||
} else {
|
||||
map_observable(a, ctx, &mut anomalies)
|
||||
};
|
||||
let row = AnnotationRow {
|
||||
legacy_id: a.id.clone(),
|
||||
new_id: text(&annotation, "id").unwrap_or_default(),
|
||||
view_id: ctx.view_id.to_owned(),
|
||||
kind: text(&annotation, "kind").unwrap_or_default(),
|
||||
laterality: text(&annotation, "laterality").unwrap_or_default(),
|
||||
region_code: if is_trigger {
|
||||
None
|
||||
} else {
|
||||
text(&annotation, "vertebral_region_code")
|
||||
},
|
||||
finding_zone_id: if is_trigger {
|
||||
None
|
||||
} else {
|
||||
text(&annotation, "finding_zone_id")
|
||||
},
|
||||
geometry: LegacyGeometry::describe(a.geometry.as_ref()),
|
||||
};
|
||||
MappedAnnotation {
|
||||
annotation,
|
||||
row,
|
||||
region,
|
||||
anomalies,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
//! Every FR/EN text the importer writes itself, derived from the document so the list can
|
||||
//! never drift from what is written (`packages/anatomy/src/legacy/authored-labels.ts`).
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
use super::types::Anomaly;
|
||||
|
||||
pub const AUTHORED_LABEL_CODE: &str = "label.authored";
|
||||
|
||||
/// `${value}` of a JavaScript template literal for a JSON value read from the document.
|
||||
fn template(value: &Value) -> String {
|
||||
match value {
|
||||
Value::String(text) => text.clone(),
|
||||
other => other.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn authored(subject: String, text: Option<&Value>) -> Anomaly {
|
||||
let parts: Vec<String> = [("FR", "fr"), ("EN", "en")]
|
||||
.into_iter()
|
||||
.filter_map(|(label, key)| {
|
||||
text.and_then(|t| t.get(key))
|
||||
.map(|value| format!("{label} « {} »", template(value)))
|
||||
})
|
||||
.collect();
|
||||
Anomaly::to_validate(
|
||||
AUTHORED_LABEL_CODE,
|
||||
subject,
|
||||
format!("Libellé écrit par l'importateur : {}.", parts.join(", ")),
|
||||
)
|
||||
}
|
||||
|
||||
pub fn region_label_subject(code: &str) -> String {
|
||||
format!("vertebral_config.{code}")
|
||||
}
|
||||
|
||||
fn items<'a>(document: &'a Value, pointer: &str) -> &'a [Value] {
|
||||
document
|
||||
.pointer(pointer)
|
||||
.and_then(Value::as_array)
|
||||
.map_or(&[], Vec::as_slice)
|
||||
}
|
||||
|
||||
pub fn authored_label_anomalies(document: &Value) -> Vec<Anomaly> {
|
||||
let mut out = vec![
|
||||
authored("name".into(), document.get("name")),
|
||||
authored("description".into(), document.get("description")),
|
||||
];
|
||||
for view in items(document, "/views") {
|
||||
let id = view.get("id").map(template).unwrap_or_default();
|
||||
out.push(authored(format!("views.{id}"), view.get("name")));
|
||||
}
|
||||
for region in items(document, "/vertebral_config/regions") {
|
||||
let code = region.get("code").map(template).unwrap_or_default();
|
||||
out.push(authored(region_label_subject(&code), region.get("name")));
|
||||
}
|
||||
out
|
||||
}
|
||||
@@ -0,0 +1,353 @@
|
||||
//! The legacy findings catalog (`pathologies.json`, schema 2.0) → canonical finding zones in
|
||||
//! the lean authoring form (`packages/anatomy/src/legacy/catalog.ts`): optional keys are
|
||||
//! written only when the source carries them.
|
||||
//!
|
||||
//! The catalog is read as `JSON.parse` reads it (`domain::js::parse_json`, zones in
|
||||
//! `Object.entries` order) and its values are copied as they are. Declared difference: where
|
||||
//! JavaScript would coerce or crash on a wrong type (an id or a `children` that is not what
|
||||
//! the format says), the port refuses the catalog with a message naming the node.
|
||||
|
||||
use serde_json::{Map, Value, json};
|
||||
|
||||
use super::types::{Anomaly, ZoneStats};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::text::{js_keys, js_object, slugify};
|
||||
use crate::domain::js::{is_stable_id, parse_json};
|
||||
|
||||
/// A parsed catalog: the zones in `Object.entries` order, and the other top-level values.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct LegacyCatalog {
|
||||
pub root: Map<String, Value>,
|
||||
pub zones: Vec<(String, Map<String, Value>)>,
|
||||
}
|
||||
|
||||
impl LegacyCatalog {
|
||||
pub fn get(&self, key: &str) -> Option<&Value> {
|
||||
self.root.get(key)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_legacy_catalog(text: &str) -> Result<LegacyCatalog, PipelineError> {
|
||||
let source = |why: &str| PipelineError::Source(format!("pathologies.json: {why}"));
|
||||
let value = parse_json(text).map_err(|error| source(&error.to_string()))?;
|
||||
let Value::Object(root) = value else {
|
||||
return Err(source("\"zones\" object missing"));
|
||||
};
|
||||
let Some(Value::Object(zones)) = root.get("zones") else {
|
||||
return Err(source("\"zones\" object missing"));
|
||||
};
|
||||
let mut ordered = Vec::with_capacity(zones.len());
|
||||
for key in js_keys(zones) {
|
||||
let Some(Value::Object(zone)) = zones.get(key) else {
|
||||
return Err(source(&format!("zone {key} is not an object")));
|
||||
};
|
||||
ordered.push((key.clone(), zone.clone()));
|
||||
}
|
||||
Ok(LegacyCatalog {
|
||||
root,
|
||||
zones: ordered,
|
||||
})
|
||||
}
|
||||
|
||||
/// A node's string field, refused when absent or not a string.
|
||||
pub fn text_field<'a>(node: &'a Map<String, Value>, key: &str) -> Result<&'a str, PipelineError> {
|
||||
node.get(key).and_then(Value::as_str).ok_or_else(|| {
|
||||
PipelineError::Source(format!("pathologies.json: \"{key}\" is not a string"))
|
||||
})
|
||||
}
|
||||
|
||||
/// A node's child list (`node.children ?? []`), refused when it is not an array.
|
||||
pub fn children(node: &Map<String, Value>) -> Result<Vec<&Map<String, Value>>, PipelineError> {
|
||||
match node.get("children") {
|
||||
None | Some(Value::Null) => Ok(Vec::new()),
|
||||
Some(Value::Array(items)) => items.iter().map(as_node).collect(),
|
||||
Some(_) => Err(PipelineError::Source(
|
||||
"pathologies.json: \"children\" is not an array".into(),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// A zone's categories, refused when not an array of objects.
|
||||
pub fn categories(zone: &Map<String, Value>) -> Result<Vec<&Map<String, Value>>, PipelineError> {
|
||||
match zone.get("categories") {
|
||||
Some(Value::Array(items)) => items.iter().map(as_node).collect(),
|
||||
_ => Err(PipelineError::Source(
|
||||
"pathologies.json: \"categories\" is not an array".into(),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
fn as_node(value: &Value) -> Result<&Map<String, Value>, PipelineError> {
|
||||
value
|
||||
.as_object()
|
||||
.ok_or_else(|| PipelineError::Source("pathologies.json: a node is not an object".into()))
|
||||
}
|
||||
|
||||
pub fn category_kind(category_id: &str) -> &'static str {
|
||||
if category_id.ends_with("-dysfunctions") {
|
||||
"dysfunction"
|
||||
} else if category_id.ends_with("-observations") {
|
||||
"consultation_reason"
|
||||
} else if category_id == "anamnesis" {
|
||||
"anamnesis"
|
||||
} else {
|
||||
"custom"
|
||||
}
|
||||
}
|
||||
|
||||
/// JavaScript truthiness of a JSON value.
|
||||
pub fn truthy(value: Option<&Value>) -> bool {
|
||||
match value {
|
||||
None | Some(Value::Null) => false,
|
||||
Some(Value::Bool(flag)) => *flag,
|
||||
Some(Value::Number(number)) => number.as_f64().is_some_and(|n| n != 0.0 && !n.is_nan()),
|
||||
Some(Value::String(text)) => !text.is_empty(),
|
||||
Some(_) => true,
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
||||
pub struct TreeCounts {
|
||||
pub nodes: usize,
|
||||
pub leaves: usize,
|
||||
pub vet_referrals: usize,
|
||||
}
|
||||
|
||||
/// Counts categories as nodes, as the audit does (792 = 78 + 714). Works on the legacy and
|
||||
/// on the canonical form: both say `children` and `requires_vet_referral`.
|
||||
pub fn count_tree(nodes: &[&Map<String, Value>]) -> Result<TreeCounts, PipelineError> {
|
||||
let mut total = TreeCounts::default();
|
||||
for node in nodes {
|
||||
let kids = children(node)?;
|
||||
let sub = count_tree(&kids)?;
|
||||
total.nodes += 1 + sub.nodes;
|
||||
total.leaves += sub.leaves + usize::from(kids.is_empty());
|
||||
total.vet_referrals +=
|
||||
sub.vet_referrals + usize::from(truthy(node.get("requires_vet_referral")));
|
||||
}
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct MappedCatalog {
|
||||
pub zones: Vec<Value>,
|
||||
pub zone_stats: Vec<ZoneStats>,
|
||||
pub node_id_map: Map<String, Value>,
|
||||
pub missing_description_en: usize,
|
||||
pub sources: usize,
|
||||
pub anomalies: Vec<Anomaly>,
|
||||
}
|
||||
|
||||
/// Tallies of one catalog walk.
|
||||
#[derive(Default)]
|
||||
struct WalkState {
|
||||
node_ids: Vec<(String, Value)>,
|
||||
anomalies: Vec<Anomaly>,
|
||||
missing_description_en: usize,
|
||||
sources: usize,
|
||||
}
|
||||
|
||||
/// `{ fr?, en? }` from `description_fr` / `description_en`, when either is present.
|
||||
fn description(node: &Map<String, Value>) -> Option<Value> {
|
||||
let fr = node.get("description_fr");
|
||||
let en = node.get("description_en");
|
||||
if fr.is_none() && en.is_none() {
|
||||
return None;
|
||||
}
|
||||
let mut text = Map::new();
|
||||
if let Some(fr) = fr {
|
||||
text.insert("fr".into(), fr.clone());
|
||||
}
|
||||
if let Some(en) = en {
|
||||
text.insert("en".into(), en.clone());
|
||||
}
|
||||
Some(Value::Object(text))
|
||||
}
|
||||
|
||||
/// `{ fr: name_fr, en: name_en }`, an absent key left out as `JSON.stringify` leaves out
|
||||
/// `undefined`.
|
||||
fn name(node: &Map<String, Value>) -> Value {
|
||||
let mut text = Map::new();
|
||||
for (from, to) in [("name_fr", "fr"), ("name_en", "en")] {
|
||||
if let Some(value) = node.get(from) {
|
||||
text.insert(to.into(), value.clone());
|
||||
}
|
||||
}
|
||||
Value::Object(text)
|
||||
}
|
||||
|
||||
fn stable_node_id(id: &str, path: &[&str], state: &mut WalkState) -> String {
|
||||
if is_stable_id(id) {
|
||||
return id.to_owned();
|
||||
}
|
||||
let slug = slugify(id);
|
||||
let next = if slug.is_empty() {
|
||||
"node".to_owned()
|
||||
} else {
|
||||
slug
|
||||
};
|
||||
let key = [path, &[id]].concat().join("/");
|
||||
state
|
||||
.node_ids
|
||||
.push((key.clone(), Value::from(next.clone())));
|
||||
state.anomalies.push(Anomaly::to_validate(
|
||||
"finding.id.slugified",
|
||||
key,
|
||||
format!("Identifiant hors motif stable, remplacé par « {next} »."),
|
||||
));
|
||||
next
|
||||
}
|
||||
|
||||
fn tally_node(node: &Map<String, Value>, state: &mut WalkState) -> Result<(), PipelineError> {
|
||||
if node.contains_key("description_fr") && !node.contains_key("description_en") {
|
||||
state.missing_description_en += 1;
|
||||
}
|
||||
state.sources += match node.get("sources") {
|
||||
None | Some(Value::Null) => 0,
|
||||
Some(Value::Array(items)) => items.len(),
|
||||
Some(_) => {
|
||||
return Err(PipelineError::Source(
|
||||
"pathologies.json: \"sources\" is not an array".into(),
|
||||
));
|
||||
}
|
||||
};
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn map_node(
|
||||
node: &Map<String, Value>,
|
||||
path: &[&str],
|
||||
state: &mut WalkState,
|
||||
) -> Result<Value, PipelineError> {
|
||||
tally_node(node, state)?;
|
||||
let legacy_id = text_field(node, "id")?;
|
||||
let id = stable_node_id(legacy_id, path, state);
|
||||
let mut out = Map::new();
|
||||
out.insert("id".into(), Value::from(id));
|
||||
out.insert("name".into(), name(node));
|
||||
if let Some(text) = description(node) {
|
||||
out.insert("description".into(), text);
|
||||
}
|
||||
if node.get("requires_vet_referral") == Some(&Value::Bool(true)) {
|
||||
out.insert("requires_vet_referral".into(), Value::Bool(true));
|
||||
}
|
||||
if truthy(node.get("children")) {
|
||||
let child_path = [path, &[legacy_id]].concat();
|
||||
let mapped: Result<Vec<Value>, PipelineError> = children(node)?
|
||||
.into_iter()
|
||||
.map(|child| map_node(child, &child_path, state))
|
||||
.collect();
|
||||
out.insert("children".into(), Value::Array(mapped?));
|
||||
}
|
||||
Ok(Value::Object(out))
|
||||
}
|
||||
|
||||
fn map_category(
|
||||
zone_id: &str,
|
||||
category: &Map<String, Value>,
|
||||
state: &mut WalkState,
|
||||
) -> Result<Value, PipelineError> {
|
||||
tally_node(category, state)?;
|
||||
let category_id = text_field(category, "id")?;
|
||||
let kind = category_kind(category_id);
|
||||
if kind == "custom" {
|
||||
state.anomalies.push(Anomaly::to_validate(
|
||||
"finding.category.kind_unknown",
|
||||
format!("{zone_id}/{category_id}"),
|
||||
"Type de catégorie non reconnu, importé en « custom ».",
|
||||
));
|
||||
}
|
||||
let id = stable_node_id(category_id, &[zone_id], state);
|
||||
let path = [zone_id, category_id];
|
||||
let mapped: Result<Vec<Value>, PipelineError> = children(category)?
|
||||
.into_iter()
|
||||
.map(|child| map_node(child, &path, state))
|
||||
.collect();
|
||||
let mut out = json!({ "id": id, "kind": kind, "name": name(category), "children": mapped? });
|
||||
if let (Some(text), Some(map)) = (description(category), out.as_object_mut()) {
|
||||
map.insert("description".into(), text);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn zone_stats_of(key: &str, zone: &Map<String, Value>) -> Result<ZoneStats, PipelineError> {
|
||||
let cats = categories(zone)?;
|
||||
let counts = count_tree(&cats)?;
|
||||
let mut category_leaves = 0;
|
||||
for category in &cats {
|
||||
category_leaves += usize::from(children(category)?.is_empty());
|
||||
}
|
||||
let zone_id = zone.get("id").and_then(Value::as_str).unwrap_or(key);
|
||||
Ok(ZoneStats {
|
||||
zone_id: zone_id.to_owned(),
|
||||
nodes: counts.nodes,
|
||||
leaves: counts.leaves - category_leaves,
|
||||
vet_referrals: counts.vet_referrals,
|
||||
})
|
||||
}
|
||||
|
||||
/// Fails when the output tree does not hold exactly the input tree.
|
||||
pub fn assert_nothing_dropped(
|
||||
zone_id: &str,
|
||||
before: TreeCounts,
|
||||
after: TreeCounts,
|
||||
) -> Result<(), PipelineError> {
|
||||
let pairs = [
|
||||
("nodes", before.nodes, after.nodes),
|
||||
("leaves", before.leaves, after.leaves),
|
||||
("vetReferrals", before.vet_referrals, after.vet_referrals),
|
||||
];
|
||||
let detail: Vec<String> = pairs
|
||||
.iter()
|
||||
.filter(|(_, b, a)| b != a)
|
||||
.map(|(key, b, a)| format!("{key} {b} in, {a} out"))
|
||||
.collect();
|
||||
if detail.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
Err(PipelineError::Guard(format!(
|
||||
"zone {zone_id}: {}",
|
||||
detail.join("; ")
|
||||
)))
|
||||
}
|
||||
|
||||
fn map_zone(
|
||||
key: &str,
|
||||
zone: &Map<String, Value>,
|
||||
state: &mut WalkState,
|
||||
) -> Result<Value, PipelineError> {
|
||||
let zone_id = text_field(zone, "id")?;
|
||||
if zone_id != key {
|
||||
return Err(PipelineError::Guard(format!(
|
||||
"zone key {key} ≠ id {zone_id}"
|
||||
)));
|
||||
}
|
||||
let cats = categories(zone)?;
|
||||
let mut mapped = Vec::with_capacity(cats.len());
|
||||
for category in &cats {
|
||||
mapped.push(map_category(key, category, state)?);
|
||||
}
|
||||
let out_nodes: Vec<&Map<String, Value>> = mapped.iter().filter_map(Value::as_object).collect();
|
||||
assert_nothing_dropped(key, count_tree(&cats)?, count_tree(&out_nodes)?)?;
|
||||
Ok(json!({ "id": key, "name": name(zone), "categories": mapped }))
|
||||
}
|
||||
|
||||
pub fn map_catalog(catalog: &LegacyCatalog) -> Result<MappedCatalog, PipelineError> {
|
||||
let mut state = WalkState::default();
|
||||
let mut zones = Vec::with_capacity(catalog.zones.len());
|
||||
let mut zone_stats = Vec::with_capacity(catalog.zones.len());
|
||||
for (key, zone) in &catalog.zones {
|
||||
zones.push(map_zone(key, zone, &mut state)?);
|
||||
}
|
||||
for (key, zone) in &catalog.zones {
|
||||
zone_stats.push(zone_stats_of(key, zone)?);
|
||||
}
|
||||
Ok(MappedCatalog {
|
||||
zones,
|
||||
zone_stats,
|
||||
node_id_map: js_object(state.node_ids),
|
||||
missing_description_en: state.missing_description_en,
|
||||
sources: state.sources,
|
||||
anomalies: state.anomalies,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
//! Catalog-wide findings that need a business decision or an explanation
|
||||
//! (`packages/anatomy/src/legacy/catalog-checks.ts`). Nothing here alters the data.
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
use super::catalog::{LegacyCatalog, categories, children, text_field};
|
||||
use super::types::Anomaly;
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::text::utf16_cmp;
|
||||
|
||||
/// `/^(.+)-(dysfunctions|observations)$/`: the prefix and the suffix (`.` stops at a line
|
||||
/// terminator, as in JavaScript).
|
||||
fn prefixed(id: &str) -> Option<(&str, &str)> {
|
||||
["dysfunctions", "observations"]
|
||||
.into_iter()
|
||||
.find_map(|suffix| {
|
||||
let prefix = id.strip_suffix(suffix)?.strip_suffix('-')?;
|
||||
let terminators = ['\n', '\r', '\u{2028}', '\u{2029}'];
|
||||
(!prefix.is_empty() && !prefix.contains(terminators)).then_some((prefix, suffix))
|
||||
})
|
||||
}
|
||||
|
||||
/// A prefixed category id that names another zone (sphenoid → ssb-…).
|
||||
fn foreign_prefix_anomalies(catalog: &LegacyCatalog) -> Result<Vec<Anomaly>, PipelineError> {
|
||||
let zone_ids: Vec<&str> = catalog.zones.iter().map(|(key, _)| key.as_str()).collect();
|
||||
let mut out = Vec::new();
|
||||
for (zone_id, zone) in &catalog.zones {
|
||||
for category in categories(zone)? {
|
||||
let category_id = text_field(category, "id")?;
|
||||
let Some((prefix, suffix)) = prefixed(category_id) else {
|
||||
continue;
|
||||
};
|
||||
if prefix == zone_id || !zone_ids.contains(&prefix) {
|
||||
continue;
|
||||
}
|
||||
out.push(Anomaly::to_validate(
|
||||
"finding.category.foreign_prefix",
|
||||
format!("{zone_id}/{category_id}"),
|
||||
format!(
|
||||
"Identifiant de catégorie dupliqué entre {zone_id} et {prefix} ; \
|
||||
{zone_id}-{suffix} attendu ; décision ostéopathe requise. \
|
||||
Importé tel quel (non renommé)."
|
||||
),
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn child_ids(category: &Map<String, Value>) -> Result<String, PipelineError> {
|
||||
let ids: Result<Vec<&str>, PipelineError> = children(category)?
|
||||
.into_iter()
|
||||
.map(|child| text_field(child, "id"))
|
||||
.collect();
|
||||
Ok(ids?.join(", "))
|
||||
}
|
||||
|
||||
/// Same `anamnesis` id everywhere, different content per zone family.
|
||||
fn anamnesis_variants(catalog: &LegacyCatalog) -> Result<Vec<Anomaly>, PipelineError> {
|
||||
let mut variants: Vec<(String, Vec<String>)> = Vec::new();
|
||||
for (zone_id, zone) in &catalog.zones {
|
||||
let found = categories(zone)?
|
||||
.into_iter()
|
||||
.find(|c| c.get("id").and_then(Value::as_str) == Some("anamnesis"));
|
||||
let Some(category) = found else { continue };
|
||||
let key = child_ids(category)?;
|
||||
match variants.iter_mut().find(|(known, _)| *known == key) {
|
||||
Some((_, zones)) => zones.push(zone_id.clone()),
|
||||
None => variants.push((key, vec![zone_id.clone()])),
|
||||
}
|
||||
}
|
||||
let total = variants.len();
|
||||
Ok(variants
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
.map(|(index, (children, zones))| {
|
||||
Anomaly::informative(
|
||||
"finding.anamnesis.variant",
|
||||
format!("anamnesis variante {}/{total}", index + 1),
|
||||
format!("Zones {} : {children}.", zones.join(", ")),
|
||||
)
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
fn collect_ids(
|
||||
nodes: &[&Map<String, Value>],
|
||||
into: &mut Vec<String>,
|
||||
seen: &mut HashSet<String>,
|
||||
) -> Result<(), PipelineError> {
|
||||
for node in nodes {
|
||||
let id = text_field(node, "id")?;
|
||||
if seen.insert(id.to_owned()) {
|
||||
into.push(id.to_owned());
|
||||
}
|
||||
collect_ids(&children(node)?, into, seen)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Node ids are unique among siblings only; a finding is (zone, path).
|
||||
fn cross_zone_duplicates(catalog: &LegacyCatalog) -> Result<Vec<Anomaly>, PipelineError> {
|
||||
let mut zones_by_id: Vec<(String, usize)> = Vec::new();
|
||||
for (_, zone) in &catalog.zones {
|
||||
let (mut ids, mut seen) = (Vec::new(), HashSet::new());
|
||||
collect_ids(&categories(zone)?, &mut ids, &mut seen)?;
|
||||
for id in ids {
|
||||
match zones_by_id.iter_mut().find(|(known, _)| *known == id) {
|
||||
Some((_, count)) => *count += 1,
|
||||
None => zones_by_id.push((id, 1)),
|
||||
}
|
||||
}
|
||||
}
|
||||
let shared: Vec<(String, usize)> = zones_by_id.into_iter().filter(|(_, n)| *n > 1).collect();
|
||||
let mut top = shared.clone();
|
||||
top.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| utf16_cmp(&a.0, &b.0)));
|
||||
let listed: Vec<String> = top
|
||||
.iter()
|
||||
.take(8)
|
||||
.map(|(id, n)| format!("{id} ({n})"))
|
||||
.collect();
|
||||
Ok(vec![Anomaly::informative(
|
||||
"finding.id.shared_across_zones",
|
||||
format!("{} identifiants", shared.len()),
|
||||
format!(
|
||||
"Identifiants de nœuds réutilisés entre zones (unicité entre frères seulement, \
|
||||
adressage par zone + chemin). Les plus fréquents : {}.",
|
||||
listed.join(", ")
|
||||
),
|
||||
)])
|
||||
}
|
||||
|
||||
pub fn catalog_anomalies(catalog: &LegacyCatalog) -> Result<Vec<Anomaly>, PipelineError> {
|
||||
let mut out = foreign_prefix_anomalies(catalog)?;
|
||||
out.extend(anamnesis_variants(catalog)?);
|
||||
out.extend(cross_zone_duplicates(catalog)?);
|
||||
Ok(out)
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
//! SVG annotation categories → canonical categories (`packages/anatomy/src/legacy/categories.ts`).
|
||||
//! The legacy editor slugified labels byte-wise, so accented letters became `-`; those slugs
|
||||
//! are normalised here and every change is reported; labels the importer writes are reported
|
||||
//! as "to validate".
|
||||
|
||||
use serde_json::{Map, Value, json};
|
||||
|
||||
use super::svg::LegacyAnnotation;
|
||||
use super::types::Anomaly;
|
||||
use crate::curation::text::js_object;
|
||||
|
||||
/// Degraded legacy slug → real slug; other slugs are kept.
|
||||
pub fn normalise_category_id(legacy: &str) -> String {
|
||||
match legacy {
|
||||
"charni-res-vert-brales" => "charnieres-vertebrales",
|
||||
"cadrans-du-cr-ne" => "cadrans-du-crane",
|
||||
other => other,
|
||||
}
|
||||
.to_owned()
|
||||
}
|
||||
|
||||
/// Labels authored by the importer, keyed by normalised slug.
|
||||
fn label(id: &str) -> Option<(&'static str, &'static str)> {
|
||||
Some(match id {
|
||||
"articulations" => ("Articulations", "Joints"),
|
||||
"os" => ("Os", "Bones"),
|
||||
"zone-colonne" => ("Zone colonne", "Spine"),
|
||||
"charnieres-vertebrales" => ("Charnières vertébrales", "Vertebral hinges"),
|
||||
"cadrans-du-crane" => ("Cadrans du crâne", "Skull quadrants"),
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct MappedCategories {
|
||||
pub categories: Vec<Value>,
|
||||
pub id_map: Map<String, Value>,
|
||||
pub anomalies: Vec<Anomaly>,
|
||||
}
|
||||
|
||||
/// `data-anno-category-name` is used only when it is a real label (it repeats the slug in
|
||||
/// the equine SVGs).
|
||||
fn french_name(legacy: &str, category_name: Option<&str>) -> String {
|
||||
let id = normalise_category_id(legacy);
|
||||
match category_name {
|
||||
Some(name) if !name.is_empty() && name != legacy => name.to_owned(),
|
||||
_ => label(&id).map_or(id, |(fr, _)| fr.to_owned()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Category → its first non-null name, in order of first appearance (a JavaScript `Map`).
|
||||
fn first_seen(annotations: &[&LegacyAnnotation]) -> Vec<(String, Option<String>)> {
|
||||
let mut seen: Vec<(String, Option<String>)> = Vec::new();
|
||||
for a in annotations {
|
||||
let Some(category) = &a.category else {
|
||||
continue;
|
||||
};
|
||||
match seen.iter_mut().find(|(key, _)| key == category) {
|
||||
Some((_, name @ None)) => name.clone_from(&a.category_name),
|
||||
Some(_) => {}
|
||||
None => seen.push((category.clone(), a.category_name.clone())),
|
||||
}
|
||||
}
|
||||
seen
|
||||
}
|
||||
|
||||
/// Categories in order of first appearance across the SVGs.
|
||||
pub fn map_categories(annotations: &[&LegacyAnnotation]) -> MappedCategories {
|
||||
let mut categories = Vec::new();
|
||||
let mut ids = Vec::new();
|
||||
let mut anomalies = Vec::new();
|
||||
for (legacy, category_name) in first_seen(annotations) {
|
||||
let id = normalise_category_id(&legacy);
|
||||
let fr = french_name(&legacy, category_name.as_deref());
|
||||
let en = label(&id).map_or_else(|| id.clone(), |(_, en)| en.to_owned());
|
||||
categories.push(json!({ "id": id, "name": { "fr": fr, "en": en }, "color": null }));
|
||||
ids.push((legacy.clone(), Value::from(id.clone())));
|
||||
if id != legacy {
|
||||
anomalies.push(Anomaly::to_validate(
|
||||
"category.slug.normalised",
|
||||
legacy,
|
||||
format!("Slug technique dégradé (accents perdus) normalisé en « {id} »."),
|
||||
));
|
||||
}
|
||||
anomalies.push(Anomaly::to_validate(
|
||||
"category.label.authored",
|
||||
id,
|
||||
format!("Libellé écrit par l'importateur : FR « {fr} », EN « {en} »."),
|
||||
));
|
||||
}
|
||||
MappedCategories {
|
||||
categories,
|
||||
id_map: js_object(ids),
|
||||
anomalies,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,370 @@
|
||||
//! The one-off, deterministic import of the historical equine assets into the first
|
||||
//! canonical sheet document (`packages/anatomy/src/legacy/importer.ts`). The result is a
|
||||
//! DRAFT for business review: every correction is listed in `anomalies`, nothing is
|
||||
//! published.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use serde_json::{Map, Value, json};
|
||||
|
||||
use super::annotations::{AnnotationContext, MappedAnnotation, map_annotation, zone_name_key};
|
||||
use super::authored_labels::authored_label_anomalies;
|
||||
use super::catalog::{LegacyCatalog, MappedCatalog, map_catalog, parse_legacy_catalog};
|
||||
use super::catalog_checks::catalog_anomalies;
|
||||
use super::categories::{MappedCategories, map_categories};
|
||||
use super::svg::{ParsedSvg, parse_annotated_svg};
|
||||
use super::types::{
|
||||
AnnotationsByView, Anomaly, AssetRecord, EquineLegacyInput, ImportResult, ImportStats,
|
||||
SourceDigest, SvgSource,
|
||||
};
|
||||
use super::vertebrae::{VertebralFormula, parse_vertebral_formula, vertebral_regions};
|
||||
use crate::curation::PipelineError;
|
||||
use crate::curation::ids::{asset_id_from_sha256, sha256_hex, sheet_id_for};
|
||||
use crate::curation::text::js_object;
|
||||
use crate::domain::sheet::{
|
||||
AnatomicalSheetDocument, SHEET_DOCUMENT_SCHEMA_VERSION, validate_sheet_document,
|
||||
};
|
||||
|
||||
pub const EQUINE_SPECIES_ID: &str = "equine";
|
||||
pub const EQUINE_SHEET_VERSION: u32 = 1;
|
||||
pub const MAIN_VIEW_ID: &str = "main";
|
||||
pub const HEAD_VIEW_ID: &str = "head";
|
||||
|
||||
struct ParsedSources {
|
||||
assets: Vec<AssetRecord>,
|
||||
main: ParsedSvg,
|
||||
head: ParsedSvg,
|
||||
catalog: LegacyCatalog,
|
||||
formula: VertebralFormula,
|
||||
}
|
||||
|
||||
struct Mapped {
|
||||
annotations: Vec<MappedAnnotation>,
|
||||
categories: MappedCategories,
|
||||
catalog: MappedCatalog,
|
||||
}
|
||||
|
||||
fn asset_record(source: &SvgSource) -> AssetRecord {
|
||||
let sha256 = sha256_hex(&source.bytes);
|
||||
AssetRecord {
|
||||
file: source.file_name.clone(),
|
||||
svg_asset_id: asset_id_from_sha256(&sha256),
|
||||
sha256,
|
||||
byte_size: source.bytes.len(),
|
||||
}
|
||||
}
|
||||
|
||||
/// `Buffer.toString('utf8')`: invalid bytes become U+FFFD, a BOM is kept.
|
||||
fn lossy_text(bytes: &[u8]) -> String {
|
||||
String::from_utf8_lossy(bytes).into_owned()
|
||||
}
|
||||
|
||||
fn text_digest(file: &str, text: &str) -> SourceDigest {
|
||||
SourceDigest {
|
||||
file: file.to_owned(),
|
||||
sha256: sha256_hex(text.as_bytes()),
|
||||
byte_size: text.len(),
|
||||
}
|
||||
}
|
||||
|
||||
/// `new TextDecoder('utf-8', { fatal: true })`: invalid UTF-8 refused, one BOM removed.
|
||||
fn svg_text(source: &SvgSource) -> Result<&str, PipelineError> {
|
||||
let text = std::str::from_utf8(&source.bytes)
|
||||
.map_err(|_| PipelineError::Source(format!("{}: invalid UTF-8", source.file_name)))?;
|
||||
Ok(text.strip_prefix('\u{FEFF}').unwrap_or(text))
|
||||
}
|
||||
|
||||
fn parse_sources(input: &EquineLegacyInput) -> Result<ParsedSources, PipelineError> {
|
||||
Ok(ParsedSources {
|
||||
assets: vec![asset_record(&input.main_svg), asset_record(&input.head_svg)],
|
||||
main: parse_annotated_svg(svg_text(&input.main_svg)?, &input.main_svg.file_name)?,
|
||||
head: parse_annotated_svg(svg_text(&input.head_svg)?, &input.head_svg.file_name)?,
|
||||
catalog: parse_legacy_catalog(&lossy_text(&input.catalog))?,
|
||||
formula: parse_vertebral_formula(&lossy_text(&input.vertebrae))?,
|
||||
})
|
||||
}
|
||||
|
||||
/// `${value}` of a zone name part.
|
||||
fn template(value: Option<&Value>) -> String {
|
||||
match value {
|
||||
None => "undefined".to_owned(),
|
||||
Some(Value::String(text)) => text.clone(),
|
||||
Some(other) => other.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn zone_names(zones: &[Value]) -> HashMap<String, String> {
|
||||
let mut names = HashMap::new();
|
||||
for zone in zones {
|
||||
let fr = template(zone.pointer("/name/fr"));
|
||||
let en = template(zone.pointer("/name/en"));
|
||||
let id = zone.get("id").and_then(Value::as_str).unwrap_or_default();
|
||||
names.insert(zone_name_key(&fr, &en), id.to_owned());
|
||||
}
|
||||
names
|
||||
}
|
||||
|
||||
fn map_sources(sources: &ParsedSources) -> Result<Mapped, PipelineError> {
|
||||
let catalog = map_catalog(&sources.catalog)?;
|
||||
let zone_by_names = zone_names(&catalog.zones);
|
||||
let views = [(MAIN_VIEW_ID, &sources.main), (HEAD_VIEW_ID, &sources.head)];
|
||||
let mut annotations = Vec::new();
|
||||
for (view_id, svg) in views {
|
||||
let ctx = AnnotationContext {
|
||||
view_id,
|
||||
zone_by_names: &zone_by_names,
|
||||
};
|
||||
annotations.extend(svg.annotations.iter().map(|a| map_annotation(a, &ctx)));
|
||||
}
|
||||
let legacy: Vec<_> = sources
|
||||
.main
|
||||
.annotations
|
||||
.iter()
|
||||
.chain(&sources.head.annotations)
|
||||
.collect();
|
||||
Ok(Mapped {
|
||||
annotations,
|
||||
categories: map_categories(&legacy),
|
||||
catalog,
|
||||
})
|
||||
}
|
||||
|
||||
fn build_views(assets: &[AssetRecord]) -> Value {
|
||||
json!([
|
||||
{
|
||||
"id": MAIN_VIEW_ID, "kind": "main",
|
||||
"name": { "fr": "Vue générale", "en": "Overview" },
|
||||
"svg_asset_id": assets[0].svg_asset_id, "view_box": null,
|
||||
},
|
||||
{
|
||||
"id": HEAD_VIEW_ID, "kind": "detail", "parent_view_id": MAIN_VIEW_ID,
|
||||
"name": { "fr": "Tête", "en": "Head" },
|
||||
"svg_asset_id": assets[1].svg_asset_id, "view_box": null,
|
||||
},
|
||||
])
|
||||
}
|
||||
|
||||
fn build_document(sources: &ParsedSources, mapped: &Mapped) -> Value {
|
||||
let disclaimer = sources.catalog.get("scope_disclaimer");
|
||||
let notes = format!(
|
||||
"Import des données historiques: Horse.svg, Horse_head.svg, pathologies.json (schema {})",
|
||||
template(sources.catalog.get("schema_version"))
|
||||
);
|
||||
let annotations: Vec<&Value> = mapped.annotations.iter().map(|m| &m.annotation).collect();
|
||||
json!({
|
||||
"schema_version": SHEET_DOCUMENT_SCHEMA_VERSION,
|
||||
"species_id": EQUINE_SPECIES_ID,
|
||||
"name": { "fr": "Cheval", "en": "Horse" },
|
||||
"description": {
|
||||
"fr": "Fiche anatomique officielle Faunosteo",
|
||||
"en": "Official Faunosteo anatomical sheet",
|
||||
},
|
||||
"scope_disclaimer": disclaimer.map_or_else(|| json!({}), |text| json!({ "fr": text })),
|
||||
"attribution": { "authors": ["Faunosteo"], "license": null, "notes": notes },
|
||||
"vertebral_config": { "regions": vertebral_regions(&sources.formula) },
|
||||
"references": [],
|
||||
"categories": mapped.categories.categories,
|
||||
"views": build_views(&sources.assets),
|
||||
"annotations": annotations,
|
||||
"finding_zones": mapped.catalog.zones,
|
||||
})
|
||||
}
|
||||
|
||||
fn build_id_map(sources: &ParsedSources, mapped: &Mapped) -> Map<String, Value> {
|
||||
let annotations = mapped
|
||||
.annotations
|
||||
.iter()
|
||||
.map(|m| (m.row.legacy_id.clone(), Value::from(m.row.new_id.clone())));
|
||||
let assets = sources
|
||||
.assets
|
||||
.iter()
|
||||
.map(|a| (a.file.clone(), Value::from(a.svg_asset_id.clone())));
|
||||
let mut map = Map::new();
|
||||
map.insert("views".into(), json!({ "head": "head" }));
|
||||
map.insert("annotations".into(), Value::Object(js_object(annotations)));
|
||||
map.insert(
|
||||
"categories".into(),
|
||||
Value::Object(mapped.categories.id_map.clone()),
|
||||
);
|
||||
map.insert(
|
||||
"finding_nodes".into(),
|
||||
Value::Object(mapped.catalog.node_id_map.clone()),
|
||||
);
|
||||
map.insert("assets".into(), Value::Object(js_object(assets)));
|
||||
map
|
||||
}
|
||||
|
||||
fn count(annotations: &[Value], test: impl Fn(&Value) -> bool) -> usize {
|
||||
annotations.iter().filter(|a| test(a)).count()
|
||||
}
|
||||
|
||||
fn is(annotation: &Value, key: &str, value: &str) -> bool {
|
||||
annotation.get(key).and_then(Value::as_str) == Some(value)
|
||||
}
|
||||
|
||||
fn present(annotation: &Value, key: &str) -> bool {
|
||||
!is(annotation, "kind", "detail_trigger") && annotation.get(key).is_some_and(|v| !v.is_null())
|
||||
}
|
||||
|
||||
fn build_stats(sources: &ParsedSources, document: &Value, mapped: &Mapped) -> ImportStats {
|
||||
let annotations = document["annotations"]
|
||||
.as_array()
|
||||
.map_or(&[][..], Vec::as_slice);
|
||||
let totals = &mapped.catalog.zone_stats;
|
||||
ImportStats {
|
||||
legacy_annotations: sources.main.annotations.len() + sources.head.annotations.len(),
|
||||
annotations: annotations.len(),
|
||||
annotations_by_view: AnnotationsByView {
|
||||
main: count(annotations, |a| is(a, "view_id", MAIN_VIEW_ID)),
|
||||
head: count(annotations, |a| is(a, "view_id", HEAD_VIEW_ID)),
|
||||
},
|
||||
linked_annotations: count(annotations, |a| present(a, "finding_zone_id")),
|
||||
triggers: count(annotations, |a| is(a, "kind", "detail_trigger")),
|
||||
views: document["views"].as_array().map_or(0, Vec::len),
|
||||
vertebral_annotations: count(annotations, |a| present(a, "vertebral_region_code")),
|
||||
bilateral: count(annotations, |a| is(a, "laterality", "bilateral")),
|
||||
finding_zones: document["finding_zones"].as_array().map_or(0, Vec::len),
|
||||
nodes: totals.iter().map(|z| z.nodes).sum(),
|
||||
leaves: totals.iter().map(|z| z.leaves).sum(),
|
||||
vet_referrals: totals.iter().map(|z| z.vet_referrals).sum(),
|
||||
missing_description_en: mapped.catalog.missing_description_en,
|
||||
sources: mapped.catalog.sources,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every `data-anno-id` read must be written once; no inline legacy source may be dropped.
|
||||
fn assert_guards(stats: &ImportStats) -> Result<(), PipelineError> {
|
||||
if stats.legacy_annotations != stats.annotations {
|
||||
return Err(PipelineError::Guard(format!(
|
||||
"{} annotations read, {} written",
|
||||
stats.legacy_annotations, stats.annotations
|
||||
)));
|
||||
}
|
||||
if stats.sources != 0 {
|
||||
return Err(PipelineError::Guard(format!(
|
||||
"{} inline legacy sources found: map them to references",
|
||||
stats.sources
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn source_anomalies(sources: &ParsedSources, stats: &ImportStats) -> Vec<Anomaly> {
|
||||
let mut out = Vec::new();
|
||||
if stats.missing_description_en > 0 {
|
||||
out.push(Anomaly::to_validate(
|
||||
"finding.description_en.missing",
|
||||
"pathologies.json",
|
||||
format!(
|
||||
"{} descriptions sans traduction anglaise (clé « en » absente, rien d'inventé).",
|
||||
stats.missing_description_en
|
||||
),
|
||||
));
|
||||
}
|
||||
if sources.catalog.get("scope_disclaimer").is_some() {
|
||||
out.push(Anomaly::to_validate(
|
||||
"document.scope_disclaimer.no_en",
|
||||
"scope_disclaimer",
|
||||
"Texte de périmètre légal en français seulement : traduction anglaise à rédiger.",
|
||||
));
|
||||
}
|
||||
out.push(capped_anomaly(&sources.formula));
|
||||
if stats.sources == 0 {
|
||||
out.push(Anomaly::to_validate(
|
||||
"finding.sources.none",
|
||||
"pathologies.json",
|
||||
"Aucune source bibliographique sur les nœuds, alors que la règle projet en demande \
|
||||
au moins une par entrée.",
|
||||
));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn capped_anomaly(formula: &VertebralFormula) -> Anomaly {
|
||||
let range = formula.varies_range.as_deref().unwrap_or("non précisée");
|
||||
let caudal = formula.counts.get("Cd").copied().unwrap_or(f64::NAN);
|
||||
Anomaly::to_validate(
|
||||
"vertebral.cd.capped",
|
||||
"Cd",
|
||||
format!(
|
||||
"Caudales plafonnées à {} pour le sélecteur, alors que l'espèce varie ({range}) \
|
||||
selon le commentaire source.",
|
||||
super::svg::js_text(caudal)
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
fn informative_notes() -> Anomaly {
|
||||
Anomaly::informative(
|
||||
"annotation.geometry.hitbox",
|
||||
"géométrie",
|
||||
"Géométrie lue sur le premier élément de chaque annotation (zone de clic), le marqueur \
|
||||
visuel n'est pas importé.",
|
||||
)
|
||||
}
|
||||
|
||||
fn collect_anomalies(
|
||||
sources: &ParsedSources,
|
||||
mapped: &Mapped,
|
||||
stats: &ImportStats,
|
||||
document: &Value,
|
||||
) -> Result<Vec<Anomaly>, PipelineError> {
|
||||
let mut out = authored_label_anomalies(document);
|
||||
out.extend(sources.main.anomalies.iter().cloned());
|
||||
out.extend(sources.head.anomalies.iter().cloned());
|
||||
out.extend(
|
||||
mapped
|
||||
.annotations
|
||||
.iter()
|
||||
.flat_map(|m| m.anomalies.iter().cloned()),
|
||||
);
|
||||
out.extend(mapped.categories.anomalies.iter().cloned());
|
||||
out.extend(mapped.catalog.anomalies.iter().cloned());
|
||||
out.extend(catalog_anomalies(&sources.catalog)?);
|
||||
out.extend(source_anomalies(sources, stats));
|
||||
out.push(informative_notes());
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// `AnatomicalSheetDocumentSchema.parse`: the document must be a valid sheet document.
|
||||
fn parse_document(document: &Value) -> Result<AnatomicalSheetDocument, PipelineError> {
|
||||
serde_json::from_value(document.clone())
|
||||
.map_err(|error| PipelineError::Guard(format!("document schema: {error}")))
|
||||
}
|
||||
|
||||
pub fn import_equine_legacy(input: &EquineLegacyInput) -> Result<ImportResult, PipelineError> {
|
||||
let sources = parse_sources(input)?;
|
||||
let mapped = map_sources(&sources)?;
|
||||
let document = build_document(&sources, &mapped);
|
||||
let parsed = parse_document(&document)?;
|
||||
let stats = build_stats(&sources, &document, &mapped);
|
||||
assert_guards(&stats)?;
|
||||
Ok(ImportResult {
|
||||
legacy_id_map: build_id_map(&sources, &mapped),
|
||||
source_digests: vec![
|
||||
text_digest("pathologies.json", &lossy_text(&input.catalog)),
|
||||
text_digest("vertebrae.ts", &lossy_text(&input.vertebrae)),
|
||||
],
|
||||
sheet_id: sheet_id_for(EQUINE_SPECIES_ID, EQUINE_SHEET_VERSION),
|
||||
view_boxes: view_boxes(&sources),
|
||||
annotation_rows: mapped.annotations.iter().map(|m| m.row.clone()).collect(),
|
||||
region_inferences: mapped
|
||||
.annotations
|
||||
.iter()
|
||||
.filter_map(|m| m.region.clone())
|
||||
.collect(),
|
||||
zone_stats: mapped.catalog.zone_stats.clone(),
|
||||
anomalies: collect_anomalies(&sources, &mapped, &stats, &document)?,
|
||||
issues: validate_sheet_document(&parsed),
|
||||
assets: sources.assets,
|
||||
stats,
|
||||
document,
|
||||
})
|
||||
}
|
||||
|
||||
fn view_boxes(sources: &ParsedSources) -> Map<String, Value> {
|
||||
let mut map = Map::new();
|
||||
map.insert(MAIN_VIEW_ID.into(), json!(sources.main.view_box));
|
||||
map.insert(HEAD_VIEW_ID.into(), json!(sources.head.view_box));
|
||||
map
|
||||
}
|
||||
Loaded 100 of 291 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user