Files
faunosteo/.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md

148 lines
24 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# PLAN: Rust R5, the anatomical pipeline (legacy importer, curation, migration map, seed, upload tool), `packages/anatomy` and `packages/domain` deleted
- date: 2026-10-22 | rev 1 (R5.1 only; the later slices are a proposal for the R5.1 checkpoint) | branch: `feature/rust-r5-pipeline-core` (cut from develop 9c0117a, R4 merged)
- spec: owner brief 2026-10-22 « Clôture R4 validée et autorisation R5.1 » (§A.0); `docs/development/RUST_ARCHITECTURE_REVIEW.md` II.2 rows "anatomy curation pipeline" and "upload-sheet-assets" (lines 299, 300), VI.1 row R5 (636), VI.3 (655), VI.4 (662 to 668), D-R5 (837, 873), Annex C D-R5 amendment (974); BDR-043, BDR-045; CLAUDE.md "Anatomical sheet model", "Schemas", archive-reader gate
- facts: `.claude/audits/R5_PIPELINE_INVENTORY.md` (the TypeScript pipeline), `.claude/audits/R5_DOCUMENT_SHA256_CENSUS.md` (the fingerprint), the oracles under `crates/core/tests/oracles/r5/`, the study scripts `scripts/r5-fingerprints.mjs` and `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`
- status: R5.1 validated 2026-10-22; D-R5 decided (A.5); R5.1b and R5.2 validated 2026-10-22; R5.3 done (contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`, RUST.md §11.12), checkpoint R5.3 pending; R5.4 on the owner's go.
- not in scope of R5.1: any Rust implementation of the pipeline, any change to `documentSha256` or to the readers, the deletion of `packages/anatomy` or `packages/domain`, any golden or historical reference regeneration, release 0.2.0, deployment, SaaS features.
# Part A. For the owner
## A.0 Your brief (2026-10-22), applied as written
Branch from develop by gitflow (done). Existing plan and decisions read, no new architecture study. R5.1 = characterise the TypeScript pipeline and freeze its results (legacy import, draft, changesets, migration maps, CHANGELOG and REVIEW-REPORT, review drawings and overlays, SVG assets and upload, seed SQL, current fingerprints, determinism, compatibility with older versions and archives); reuse the existing oracles, fixtures and goldens where they suffice; keep the N-1 readers; never bend a historical reference to a new implementation. D-R5: a targeted, reproducible study of option A (sha256 of the bytes the Rust pipeline emits) against option B (sha256 of RFC 8785 canonical JSON) on the project's real data, a matrix of old and new fingerprints, an encoding change told apart from a content change, a versioned convention and a compatibility mechanism for the old fingerprints, no silent loss, no visible migration just to hide an algorithm change when a transparent and safe compatibility exists. Business invariants of §4 preserved by every later migration; no sheet declared validated by a technical migration. Stop before any Rust pipeline code and before any fingerprint change.
## A.1 What R5 delivers (the whole phase, for orientation)
The curation pipeline (legacy import, changeset application, migration map, CHANGELOG, REVIEW-REPORT, review drawings, cleaned asset, seed SQL) and the asset upload tool run in Rust (`crates/core` curation modules behind a feature, `cli` binaries `import-equine` and `upload-sheet-assets`), with the same inputs (legacy sources, `curation/changes.json`, `curation/references.json`) and the same semantics (ids, counts, 612-leaf migration, integrity rules). `packages/anatomy` and `packages/domain` leave the repository once nothing imports them; the N-1 Zod reader under `packages/core-wasm/tests/n-1` stays. The sheet fingerprint follows the D-R5 decision.
## A.2 R5.1 deliverables (this slice)
1. Inventory of the TypeScript pipeline: `.claude/audits/R5_PIPELINE_INVENTORY.md`.
2. Oracles frozen with the TypeScript code (§B.2), generated by `MAKE_R5_ORACLES=1` from `packages/anatomy/tests/r5.pipeline-oracles.test.ts`, which otherwise checks the pipeline still matches its freeze.
3. The D-R5 study (§B.3): facts, the fingerprint matrix, the compatibility mechanism, the versioned convention, costs, platform determinism, I-JSON and Unicode checks, the incidence on the archives already exported.
4. The checkpoint report (the owner's §5 list), written in RUST.md §11 and delivered in the conversation.
## A.3 Findings that need your decision at the checkpoint
- D-R5 itself (§B.3.6): A, B or C (reproduce Prettier's bytes), the canonical form (input or validated), the compatibility mechanism, the convention's name.
- F1, `js_number` tie rounding (§B.3.4): the core prints an exact decimal tie upward (`840.1692504882813`) where JavaScript, Python and `zmij` print to even (`…812`); 1 value of the legacy sources, 7.4 % of float32-exact coordinates, 0.8 % of random doubles. Invisible today (the curated draft holds no tie; the R2 contract test re-prints its expected text through the same function). The fix is small (digits from `zmij`, already locked through serde_json) and changes core bytes only for tie values; it precedes any Rust importer. Your go is needed because it touches a core serialiser (D-R11).
- The documentation contradiction: BDR-045 and Annex C say "existing fingerprints untouched", RUST.md §10 and the TODO still say "will change once". The plan follows BDR-045 (the later amendment) until you say otherwise.
- Byte-preserving port or one-time regeneration of the generated files (§B.5): the Rust pipeline can reproduce the JSON bytes only by emitting insertion order and Prettier's short-array rule, and the Markdown bytes by padding tables (the only Prettier effect measured); the review SVGs cannot match xmldom's serialiser and are compared as parsed trees. The recommendation is: JSON files byte-exact where cheap (`to_js_pretty` plus the one array rule) or regenerated once with a JCS-equality proof, Markdown byte-exact by padding, SVGs regenerated once with a tree proof, and `supabase/seed/**` added to `.prettierignore` at the switch (today a root `pnpm format` would reformat Rust-written files).
- `draft/assets/horse.svg`: keep the xmldom-cleaned bytes as the frozen asset (sha `0ea5202f…`, id `7f767f74…`, already the Storage object) and have the Rust tool verify rather than regenerate it, or re-clean it with the R3 reader and accept a new asset sha, id, document, SQL and Storage upload. Recommendation: keep the frozen bytes (the cleaning was a one-time legacy transformation; the R3 reader already proves the file).
## A.4 Stop rules of R5.1
Stop and report when: a generated file does not reproduce from a fresh run (the golden test fails); an oracle generator would have to alter a committed generated file or golden; a study step needs a change to a gated reader; the branch would need a develop merge.
## A.5 Owner decisions (2026-10-22, BDR-068)
R5.1 validated. D-R5 = option B, convention `sha256-rfc8785-validated-v1`, applied at R5.6 only, proven by the RFC vectors, edge cases and an independent differential; `v1` freezes the construction and validation rules. Aliases of the historical fingerprints from trusted frozen documents only (never from an archive or a bundle); Report v3 `sheet.documentSha256` keeps 64 hex; an unknown fingerprint is never an official sheet; a real anatomical change still migrates. F1 fixed first (R5.1b), with an independent reference. Byte-exact reproduction by default, every difference declared. Historical SVG bytes kept (never re-serialised). `.prettierignore` on the Rust-written paths, narrow, with a reproducible check. `ureq` preferred for the upload tool (R5.5). Slicing B.5 validated; stop after R5.2 (checkpoint: F1 result and impact on R2-R4, importer parity, byte differences, tests and audits, residual risks, conditions for R5.3).
# Part B. The plan
## B.1 Inventory
See `.claude/audits/R5_PIPELINE_INVENTORY.md`: 43 source files, 4,897 lines (legacy 1,842, curation 2,329, render 553, cli 119), 2,553 lines of tests (131 cases, 16 files), runtime libraries Prettier 3.8.3, @xmldom/xmldom 0.9.12, Zod 3.25.76 (the first two declared as devDependencies). 18 generated files: 7 JSON and 3 Markdown through Prettier, 5 SVGs through xmldom's XMLSerializer, 1 raw SQL template, 2 byte copies. The real changeset: 45 changes (14 A, 22 B, 9 C), 687 ops (587 set, 45 cite, 33 rename, 18 add, 4 remove, 0 move), 48 references. Measured Prettier effect on today's data: Markdown table padding only (134, 863 and 52 lines), JSON short scalar arrays collapsed plus a trailing LF; the data is identical after re-parsing. The JSON key order is the working form's insertion order (`review_status` appended last), not the schema order a typed serde struct would emit. `@faunosteo/anatomy` has no consumer outside itself; `@faunosteo/domain` is imported by `packages/anatomy` only (5 runtime symbols, all with R2 counterparts); the crate tests read `packages/domain/tests/fixtures/report-v2.*.json` and `r4.census.test.ts` pins xmldom under `packages/anatomy`. Entry points: `importEquineLegacy` (legacy sources → ImportResult), `buildEquineDraft` (ImportResult + curation → DraftResult), `renderArtifacts` (4 legacy files), `renderDraftArtifacts` (document, assets, sheet-ref, migration map, 4 review SVGs, CHANGELOG, REVIEW-REPORT, seed SQL), `cli/import-equine.ts` (writes both folders, removes stale files), `packages/supabase/scripts/upload-sheet-assets.ts` (bucket `anatomical-sheet-assets`, key `<sha256>.svg`, `upsert: false`, bytes verified against `assets.json`, RPC `confirm_sheet_asset_upload`). JavaScript semantics to pin: `deterministicUuid` (sha256 of the name, version nibble 5, variant 10xx), `slugify` (NFD, combining marks U+0300 to U+036F stripped, `toLowerCase`, `[^a-z0-9]+` → `-`, trimmed, 64 chars), `sha256Hex` over UTF-8, `JSON.stringify(v, null, 2)` then Prettier json (short scalar arrays collapsed when they fit in 80 columns, objects always expanded), Prettier markdown, xmldom `XMLSerializer` for the review SVGs, `TextDecoder('utf-8', { fatal: true })`.
## B.2 Oracles frozen at R5.1 (`crates/core/tests/oracles/r5/`, generator `MAKE_R5_ORACLES=1`, never regenerated without the owner)
| Id | Content | Location | Used by |
|---|---|---|---|
| P1 | the committed generated folders `legacy-import/` (5 files) and `draft/` (14 files): the byte oracle, already pinned to a fresh run by `seed.golden.test.ts` | `supabase/seed/anatomical-sheets/equine/{legacy-import,draft}` | every R5 slice |
| P2 | `pipeline.json`: legacy ImportResult without the document (sheetId, viewBoxes, stats, source digests, assets, 27 anomalies, issues, region inferences, zone stats, annotation rows); DraftResult without the documents (46 changes, assets, stats before and after, issues, migration stats, 825 change records with before and after); the ledger of the 16 text artifacts (folder, file, bytes, sha256; for the 3 markdown files the sha256 before Prettier and whether Prettier changed the text) | `oracles/r5/pipeline.json` (1.3 MB) | R5.2 to R5.5 |
| P3 | `ids.json`: `deterministicUuid` over the 3 names the pipeline derives ids from plus 3 edges; `slugify` over 17 edge labels and every `name.fr` / `name.en` of both documents | `oracles/r5/ids.json` | R5.2 |
| P4 | `fingerprints.json`: for the 7 JSON artifacts the fingerprint under each rule (today's file bytes, A pretty with and without LF, A compact, B RFC 8785; for the two documents also the validated form under A and B); the ES6 text of the 65 non-integer numbers of the two documents; the two RFC 8785 vectors | `oracles/r5/fingerprints.json` | D-R5, R5.2 |
| P5 | migration semantics: `draft/report-migration-map.json` (P1) and the 612-leaf suite `curation.migration-leaves.test.ts`, plus the R2 oracle `oracles/migration-map.draft.json` and the Rust `archive_support` walk | existing | R5.3 |
| P6 | the review drawings `draft/review/{annotations,app}-{main,head}.svg` (P1, xmldom bytes) and the R3 layer and highlight oracles | existing | R5.4 (parsed-tree comparison) |
| P7 | the upload tool's contract (B.1) and the operator check SUPABASE.md §3.1 | documented | R5.5 |
| P8 | determinism: two in-process runs byte-equal (test in the generator file); `pnpm import:equine` not run (it rewrites the folders; the golden proves the same bytes) | test | every slice |
## B.3 D-R5 study
### B.3.1 Today's rule
`documentSha256 = sha256Hex(formatJson(document))`: the Prettier-formatted JSON text of the curated document in its input form (defaults omitted), the exact bytes of `draft/document.json`, embedded verbatim in `equine_draft.sql` and copied into `sheet-ref.json` and `report-migration-map.json`. One value has ever existed: `27c7383c…2660`, unchanged since 0a17e39 (2026-10-10), inside 99 committed PDFs, 8 JSON fixtures and 6 Rust oracle files. Compared, never recomputed, by the rebase (fingerprint only), the envelope invariant and the export (sheetId and fingerprint). The database stores the document as `jsonb` and no fingerprint. Full census: `R5_DOCUMENT_SHA256_CENSUS.md`.
### B.3.2 The matrix (real data, `node scripts/r5-fingerprints.mjs`, cross-checked in Rust)
| Artifact | Today (file bytes) | A: `to_js_pretty` + LF | A without LF | B: RFC 8785 | B validated form |
|---|---|---|---|---|---|
| draft/document.json | 27c7383c…2660 | 90f668a3…aa1d | 79f143c0…eb51 | a56e4a24…88a1 | 12d78872…74e2 |
| legacy-import/document.json | 91e459f4…14c2 | 8c94a464…adc2 (Node) | 6fb31371…7f66 (Node) | d8b5f2cf…81eb (Node) | 1217637e…c024 (Node) |
| draft/report-migration-map.json | 4139c51d…fce9 | 4139c51d…fce9 (same) | 01fac788…3471 | f082b66f…2ca6 | n/a |
| draft/assets.json, sheet-ref.json, legacy-id-map.json, legacy assets.json | file = A with LF (no short array to collapse) | | | | |
Rust reproduction (throwaway experiment on the core crate, `to_js_pretty`, `to_js_compact`, a 40-line JCS on `js_number`): every draft digest identical to Node, byte for byte; the legacy-import digests differ through the single tie value (F1, §B.3.4); the RFC vectors pass; JCS idempotent. Node vs Rust `JSON.stringify(v, null, 2)` over the 134 JSON fixtures of the repository: 2 differ, both the legacy baseline, both on that value; `oracles/characterisation.json` is not parseable by serde (lone surrogates, R2 L1 rule).
Encoding change vs content change: the today / A / B columns of one row are three encodings of one value (parse → deep equal; `validated_equals_input` is false only because the validated form fills defaults, not because content differs). A change of the draft content changes every column.
### B.3.3 Facts the brief asked for
- UTF-16 key order: every key of every artifact is ASCII; UTF-16 order equals code-point and byte order on this data. The Rust canonicaliser sorts by `encode_utf16()` anyway (the RFC vector with U+0080, ö, €, 😀 and U+FB33 passes).
- Numbers: 208 (draft) and 195 (legacy) tokens, 29 and 36 non-integer, none negative, no exponent, no `-0`, no integer beyond 2^53; all already in ECMAScript `Number::toString` form. JCS mandates exactly that form; the core's `js_number` implements it with one tie defect (F1).
- Unicode: 788 non-ASCII strings (precomposed French accents, « », –, °, —), all NFC, no control character, no escape, no astral character, no lone surrogate. JCS does not normalise; a lone surrogate (I-JSON forbids it) is refused by the core (L1) and never produced by the pipeline.
- I-JSON: satisfied by both documents (no duplicate key, numbers within f64, valid Unicode).
- Cost and dependencies: option A costs nothing (the core already has `to_js_pretty`, D-R11); option B costs about 40 lines in `crates/core/src/json.rs` over `js_number` and serde_json's escaper, no new crate (`serde_jcs`, listed as rejected in the review, is not needed; `zmij` 1.0.23 is already locked through serde_json 1.0.154); a 15-line Node twin for tests and tooling. Hashing: `sha2` becomes a normal dependency of the core (today dev-only) or the hash stays outside the library (the tool computes it); the wasm bundle grows by the sha256 code only if the browser must verify fingerprints, which no current path does.
- Platform determinism: B is a pure function of the parsed value (sort, ES6 numbers, fixed escapes); Node 24 and Rust 1.99 agree on every draft artifact. A depends on the writer's layout rules; today's value depends on Prettier 3.8.3's line fitting.
- Database: a PostgreSQL 17 `jsonb` round trip (PGlite, `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`) keeps B for both documents and breaks today's value and A (keys reordered, scale kept). B is the only candidate a service could recompute from `anatomical_sheet_versions.content` at Step 13.
### B.3.4 F1: `js_number` prints exact ties upward
`crates/core/src/json.rs` takes the shortest digits from `format!("{:e}")` (Grisu); on an exact decimal tie between two shortest candidates ECMAScript picks the even one, Grisu the upper one. Measured: `840.1692504882812` (legacy annotation `cx`, from Horse.svg) prints `…813`; 7,401 of 100,000 float32-exact values in 0..2000 and 769 of 100,000 random doubles differ; `zmij` (serde_json's formatter) differs on none. Reached by the canvas layer (R3), the PDF writer, the archive attachment, the R2 contract tests. Invisible today: the curated draft carries no tie value, and `sheet_contract.rs` re-prints its expected text through `to_js_pretty`, so the oracle cannot see it. Proposed fix (after your go, before R5.2): digits from `zmij` (or `serde_json::to_string(&f64)`), the two tie values and the 8 long decimals of the legacy SVG as unit tests, `zod_oracle` comparing against the JS text recorded in the oracle rather than a re-print.
### B.3.5 Compatibility mechanism (approved at A.5, applied at R5.6)
The sheet side carries the convention and the aliases; the report contract does not change.
- `sheet-ref.json` (and the bundle payload `ref`, and `anatomical_sheet_versions` when Step 13 stores a fingerprint): `document_sha256` holds the current convention's value; a new `fingerprint` block names the convention (`"algorithm": "rfc8785-sha256"` for B, `"js-pretty-sha256"` for A) and lists `aliases`: the fingerprints of the same content under earlier conventions, computed once at the switch from the frozen P4 oracle (`27c7383c…2660` for the current draft). The alias is bound to identical content by construction (the oracle records old and new over the same bytes), so a report stamped `27c7…` is the same sheet, not a rebase.
- The rebase and the envelope invariant treat a fingerprint equal to the current value or to an alias as "same document". Gated readers: `report/rebase.rs`, `archive/read.rs`, `session.rs`, `pdf/write/input.rs` (your ask-before-touch rule; the change is one membership test).
- A new report is stamped with the current value; an old archive opens with no message and no rebase; an archive on an older content revision rebases as today. No archive is rewritten; the exported envelope keeps what it holds.
- Report v3 keeps `sheet.documentSha256` (64 hex) so the N-1 reader and every fixture stay valid; the convention's name lives with the sheet, which is the one place that knows which rule produced its value.
- Fixtures: 99 PDFs, 8 JSON fixtures and 6 oracles keep `27c7…` untouched (they remain valid archives of the same content through the alias).
### B.3.6 Options compared
| | A: bytes the Rust pipeline emits (`to_js_pretty` + LF) | B: RFC 8785 of the validated document | C: Rust reproduces Prettier's bytes |
|---|---|---|---|
| Value at the switch | 90f668a3… (changes once) | 12d78872… (changes once) | 27c7383c… (unchanged) |
| Needs the alias mechanism to avoid a visible migration | yes | yes | no |
| Survives a reformat, a writer change, a jsonb round trip | no | yes | no |
| Code | 0 lines (exists) | ~40 lines Rust + 15 Node, no crate | a Prettier-json subset printer (fill rule for scalar arrays, 80 columns), bound forever to a formatter's layout |
| Risk | identity re-tied to a serialiser (the Prettier problem moved to Rust) | one more concept; the canonical form must be named (input or validated) | fragile: any Prettier rule the data exercises later changes the value |
| Review's row | D-R5 rev 2 | Annex C preference, BDR-045 | rejected rev 1 (in Node form) |
Recommendation: B over the validated form (what every reader holds, recomputable from the database after validation), the alias mechanism of §B.3.5, the convention named in `sheet-ref.json`, F1 fixed first. The old value `27c7…` stays valid forever as an alias; no archive sees a message for the switch. If you prefer no new concept at all, C is possible on this data (the only Prettier effect measured is the collapse of short scalar arrays), but it keeps the fingerprint chained to a formatter's layout, which Annex C refused.
## B.4 Business invariants (brief §4) and where they are proven
| Invariant | Oracle or test |
|---|---|
| stable ids | P3 (uuid, slug), P1 documents, `legacy-id-map.json` |
| zones and anatomical points, main and detail views | P1 documents (annotations, views), P2 annotation rows and records |
| annotation ↔ observation relations (`finding_zone_id`, `opens_view_id`) | P1, integrity rules (R2 core, 21 codes) |
| vertebral configurations | P1 `vertebral_config`, P2 region inferences |
| bilingual data | P1, P3 slug over every `name.fr` / `name.en` |
| historical references | `curation/references.json` (authored, never generated), P1 |
| existing reports and bilans | P5 (migration map, 612 leaves), the R2 rebase oracles, the R4 PDF fixtures, the alias mechanism |
| no sheet validated by a migration | the seed inserts a draft (`insert_sheet_draft`, status draft); publication stays a platform-admin action |
## B.5 Slices after R5.1 (proposal for the checkpoint; nothing started)
| Slice | Deliverable | Proof |
|---|---|---|
| R5.1 | this freeze and study | oracles committed, generator test green, checkpoint report |
| R5.1b | F1 fix in `json.rs`; `zod_oracle` compares the recorded JS text | tie unit tests; every R2/R3/R4 suite unchanged (no tie in the draft) |
| R5.2 | legacy importer in Rust (`crates/core/src/curation/legacy`): SVG sources through the R3 reader, catalog, regions, ids, anomalies, stats; `import-equine` emits `legacy-import/` | P1 bytes of document.json, legacy-id-map.json, assets.json reproduced exactly; IMPORT-REPORT compared as parsed tables (P2 pre-Prettier digest) or byte-exact if the Prettier markdown effect is nil |
| R5.3 | changeset schema, targets, ops, applier, migration map, stats, seed SQL | P1 document.json, sheet-ref.json, report-migration-map.json, equine_draft.sql byte-exact under the decided fingerprint; P2 825 records equal; 612-leaf suite ported |
| R5.4 | review overlays and app drawings on the R3 layer and compose; CHANGELOG and REVIEW-REPORT | parsed-tree comparison against the xmldom files (declaration, prefixes, whitespace declared); markdown as parsed tables |
| R5.5 | `upload-sheet-assets` binary (reqwest or ureq, service key from the environment), operator runbook | local stack run, SUPABASE.md §3.1 |
| R5.6 | D-R5 switch: fingerprint convention, aliases, the one membership test in the gated readers (your go per file), seed regenerated once by the Rust tool, CHANGELOG | old archives open with no message; a report on an older content revision still rebases; N-1 reader green |
| R5.7 | consumer census, `packages/anatomy` and `packages/domain` deleted (`packages/domain/tests/fixtures/report-v2.*.json` moved under `crates/core/tests/fixtures` first, `r4.census.test.ts` xmldom assertion inverted), `@xmldom/xmldom` and `prettier` leave the anatomy lockfile, docs (CLAUDE.md map, WORKSPACE, packages/README, RUST.md §11), ledger of the deleted tests | grep census, `pnpm check:all`, test counts reconciled |
Order constraint: R5.6 needs the owner's D-R5 decision; R5.2 needs R5.1b.
## B.6 Test ledger
`crates/core/tests/curation/LEDGER.tsv` at R5.7: one row per TS case of `packages/anatomy/tests` and `packages/domain/tests` at the freeze commit (file, title, disposition ported / replaced / kept / deleted, target). Counts at the freeze: anatomy 131 cases in 16 files (the R5.1 generator adds 6), domain 262 in 13 files.