Files
faunosteo/.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md

24 KiB
Raw Permalink Blame History

PLAN: Rust R5, the anatomical pipeline (legacy importer, curation, migration map, seed, upload tool), packages/anatomy and packages/domain deleted

  • date: 2026-10-22 | rev 1 (R5.1 only; the later slices are a proposal for the R5.1 checkpoint) | branch: feature/rust-r5-pipeline-core (cut from develop 9c0117a, R4 merged)
  • spec: owner brief 2026-10-22 « Clôture R4 validée et autorisation R5.1 » (§A.0); docs/development/RUST_ARCHITECTURE_REVIEW.md II.2 rows "anatomy curation pipeline" and "upload-sheet-assets" (lines 299, 300), VI.1 row R5 (636), VI.3 (655), VI.4 (662 to 668), D-R5 (837, 873), Annex C D-R5 amendment (974); BDR-043, BDR-045; CLAUDE.md "Anatomical sheet model", "Schemas", archive-reader gate
  • facts: .claude/audits/R5_PIPELINE_INVENTORY.md (the TypeScript pipeline), .claude/audits/R5_DOCUMENT_SHA256_CENSUS.md (the fingerprint), the oracles under crates/core/tests/oracles/r5/, the study scripts scripts/r5-fingerprints.mjs and packages/supabase/scripts/r5-jsonb-fingerprint.mjs
  • status: R5.1 validated 2026-10-22; D-R5 decided (A.5); R5.1b and R5.2 validated 2026-10-22; R5.3 done (contract .claude/tasks/contracts/2026-10-22-rust-r5-draft.md, RUST.md §11.12), checkpoint R5.3 pending; R5.4 on the owner's go.
  • not in scope of R5.1: any Rust implementation of the pipeline, any change to documentSha256 or to the readers, the deletion of packages/anatomy or packages/domain, any golden or historical reference regeneration, release 0.2.0, deployment, SaaS features.

Part A. For the owner

A.0 Your brief (2026-10-22), applied as written

Branch from develop by gitflow (done). Existing plan and decisions read, no new architecture study. R5.1 = characterise the TypeScript pipeline and freeze its results (legacy import, draft, changesets, migration maps, CHANGELOG and REVIEW-REPORT, review drawings and overlays, SVG assets and upload, seed SQL, current fingerprints, determinism, compatibility with older versions and archives); reuse the existing oracles, fixtures and goldens where they suffice; keep the N-1 readers; never bend a historical reference to a new implementation. D-R5: a targeted, reproducible study of option A (sha256 of the bytes the Rust pipeline emits) against option B (sha256 of RFC 8785 canonical JSON) on the project's real data, a matrix of old and new fingerprints, an encoding change told apart from a content change, a versioned convention and a compatibility mechanism for the old fingerprints, no silent loss, no visible migration just to hide an algorithm change when a transparent and safe compatibility exists. Business invariants of §4 preserved by every later migration; no sheet declared validated by a technical migration. Stop before any Rust pipeline code and before any fingerprint change.

A.1 What R5 delivers (the whole phase, for orientation)

The curation pipeline (legacy import, changeset application, migration map, CHANGELOG, REVIEW-REPORT, review drawings, cleaned asset, seed SQL) and the asset upload tool run in Rust (crates/core curation modules behind a feature, cli binaries import-equine and upload-sheet-assets), with the same inputs (legacy sources, curation/changes.json, curation/references.json) and the same semantics (ids, counts, 612-leaf migration, integrity rules). packages/anatomy and packages/domain leave the repository once nothing imports them; the N-1 Zod reader under packages/core-wasm/tests/n-1 stays. The sheet fingerprint follows the D-R5 decision.

A.2 R5.1 deliverables (this slice)

  1. Inventory of the TypeScript pipeline: .claude/audits/R5_PIPELINE_INVENTORY.md.
  2. Oracles frozen with the TypeScript code (§B.2), generated by MAKE_R5_ORACLES=1 from packages/anatomy/tests/r5.pipeline-oracles.test.ts, which otherwise checks the pipeline still matches its freeze.
  3. The D-R5 study (§B.3): facts, the fingerprint matrix, the compatibility mechanism, the versioned convention, costs, platform determinism, I-JSON and Unicode checks, the incidence on the archives already exported.
  4. The checkpoint report (the owner's §5 list), written in RUST.md §11 and delivered in the conversation.

A.3 Findings that need your decision at the checkpoint

  • D-R5 itself (§B.3.6): A, B or C (reproduce Prettier's bytes), the canonical form (input or validated), the compatibility mechanism, the convention's name.
  • F1, js_number tie rounding (§B.3.4): the core prints an exact decimal tie upward (840.1692504882813) where JavaScript, Python and zmij print to even (…812); 1 value of the legacy sources, 7.4 % of float32-exact coordinates, 0.8 % of random doubles. Invisible today (the curated draft holds no tie; the R2 contract test re-prints its expected text through the same function). The fix is small (digits from zmij, already locked through serde_json) and changes core bytes only for tie values; it precedes any Rust importer. Your go is needed because it touches a core serialiser (D-R11).
  • The documentation contradiction: BDR-045 and Annex C say "existing fingerprints untouched", RUST.md §10 and the TODO still say "will change once". The plan follows BDR-045 (the later amendment) until you say otherwise.
  • Byte-preserving port or one-time regeneration of the generated files (§B.5): the Rust pipeline can reproduce the JSON bytes only by emitting insertion order and Prettier's short-array rule, and the Markdown bytes by padding tables (the only Prettier effect measured); the review SVGs cannot match xmldom's serialiser and are compared as parsed trees. The recommendation is: JSON files byte-exact where cheap (to_js_pretty plus the one array rule) or regenerated once with a JCS-equality proof, Markdown byte-exact by padding, SVGs regenerated once with a tree proof, and supabase/seed/** added to .prettierignore at the switch (today a root pnpm format would reformat Rust-written files).
  • draft/assets/horse.svg: keep the xmldom-cleaned bytes as the frozen asset (sha 0ea5202f…, id 7f767f74…, already the Storage object) and have the Rust tool verify rather than regenerate it, or re-clean it with the R3 reader and accept a new asset sha, id, document, SQL and Storage upload. Recommendation: keep the frozen bytes (the cleaning was a one-time legacy transformation; the R3 reader already proves the file).

A.4 Stop rules of R5.1

Stop and report when: a generated file does not reproduce from a fresh run (the golden test fails); an oracle generator would have to alter a committed generated file or golden; a study step needs a change to a gated reader; the branch would need a develop merge.

A.5 Owner decisions (2026-10-22, BDR-068)

R5.1 validated. D-R5 = option B, convention sha256-rfc8785-validated-v1, applied at R5.6 only, proven by the RFC vectors, edge cases and an independent differential; v1 freezes the construction and validation rules. Aliases of the historical fingerprints from trusted frozen documents only (never from an archive or a bundle); Report v3 sheet.documentSha256 keeps 64 hex; an unknown fingerprint is never an official sheet; a real anatomical change still migrates. F1 fixed first (R5.1b), with an independent reference. Byte-exact reproduction by default, every difference declared. Historical SVG bytes kept (never re-serialised). .prettierignore on the Rust-written paths, narrow, with a reproducible check. ureq preferred for the upload tool (R5.5). Slicing B.5 validated; stop after R5.2 (checkpoint: F1 result and impact on R2-R4, importer parity, byte differences, tests and audits, residual risks, conditions for R5.3).

Part B. The plan

B.1 Inventory

See .claude/audits/R5_PIPELINE_INVENTORY.md: 43 source files, 4,897 lines (legacy 1,842, curation 2,329, render 553, cli 119), 2,553 lines of tests (131 cases, 16 files), runtime libraries Prettier 3.8.3, @xmldom/xmldom 0.9.12, Zod 3.25.76 (the first two declared as devDependencies). 18 generated files: 7 JSON and 3 Markdown through Prettier, 5 SVGs through xmldom's XMLSerializer, 1 raw SQL template, 2 byte copies. The real changeset: 45 changes (14 A, 22 B, 9 C), 687 ops (587 set, 45 cite, 33 rename, 18 add, 4 remove, 0 move), 48 references. Measured Prettier effect on today's data: Markdown table padding only (134, 863 and 52 lines), JSON short scalar arrays collapsed plus a trailing LF; the data is identical after re-parsing. The JSON key order is the working form's insertion order (review_status appended last), not the schema order a typed serde struct would emit. @faunosteo/anatomy has no consumer outside itself; @faunosteo/domain is imported by packages/anatomy only (5 runtime symbols, all with R2 counterparts); the crate tests read packages/domain/tests/fixtures/report-v2.*.json and r4.census.test.ts pins xmldom under packages/anatomy. Entry points: importEquineLegacy (legacy sources → ImportResult), buildEquineDraft (ImportResult + curation → DraftResult), renderArtifacts (4 legacy files), renderDraftArtifacts (document, assets, sheet-ref, migration map, 4 review SVGs, CHANGELOG, REVIEW-REPORT, seed SQL), cli/import-equine.ts (writes both folders, removes stale files), packages/supabase/scripts/upload-sheet-assets.ts (bucket anatomical-sheet-assets, key <sha256>.svg, upsert: false, bytes verified against assets.json, RPC confirm_sheet_asset_upload). JavaScript semantics to pin: deterministicUuid (sha256 of the name, version nibble 5, variant 10xx), slugify (NFD, combining marks U+0300 to U+036F stripped, toLowerCase, [^a-z0-9]+ → -, trimmed, 64 chars), sha256Hex over UTF-8, JSON.stringify(v, null, 2) then Prettier json (short scalar arrays collapsed when they fit in 80 columns, objects always expanded), Prettier markdown, xmldom XMLSerializer for the review SVGs, TextDecoder('utf-8', { fatal: true }).

B.2 Oracles frozen at R5.1 (crates/core/tests/oracles/r5/, generator MAKE_R5_ORACLES=1, never regenerated without the owner)

Id Content Location Used by
P1 the committed generated folders legacy-import/ (5 files) and draft/ (14 files): the byte oracle, already pinned to a fresh run by seed.golden.test.ts supabase/seed/anatomical-sheets/equine/{legacy-import,draft} every R5 slice
P2 pipeline.json: legacy ImportResult without the document (sheetId, viewBoxes, stats, source digests, assets, 27 anomalies, issues, region inferences, zone stats, annotation rows); DraftResult without the documents (46 changes, assets, stats before and after, issues, migration stats, 825 change records with before and after); the ledger of the 16 text artifacts (folder, file, bytes, sha256; for the 3 markdown files the sha256 before Prettier and whether Prettier changed the text) oracles/r5/pipeline.json (1.3 MB) R5.2 to R5.5
P3 ids.json: deterministicUuid over the 3 names the pipeline derives ids from plus 3 edges; slugify over 17 edge labels and every name.fr / name.en of both documents oracles/r5/ids.json R5.2
P4 fingerprints.json: for the 7 JSON artifacts the fingerprint under each rule (today's file bytes, A pretty with and without LF, A compact, B RFC 8785; for the two documents also the validated form under A and B); the ES6 text of the 65 non-integer numbers of the two documents; the two RFC 8785 vectors oracles/r5/fingerprints.json D-R5, R5.2
P5 migration semantics: draft/report-migration-map.json (P1) and the 612-leaf suite curation.migration-leaves.test.ts, plus the R2 oracle oracles/migration-map.draft.json and the Rust archive_support walk existing R5.3
P6 the review drawings draft/review/{annotations,app}-{main,head}.svg (P1, xmldom bytes) and the R3 layer and highlight oracles existing R5.4 (parsed-tree comparison)
P7 the upload tool's contract (B.1) and the operator check SUPABASE.md §3.1 documented R5.5
P8 determinism: two in-process runs byte-equal (test in the generator file); pnpm import:equine not run (it rewrites the folders; the golden proves the same bytes) test every slice

B.3 D-R5 study

B.3.1 Today's rule

documentSha256 = sha256Hex(formatJson(document)): the Prettier-formatted JSON text of the curated document in its input form (defaults omitted), the exact bytes of draft/document.json, embedded verbatim in equine_draft.sql and copied into sheet-ref.json and report-migration-map.json. One value has ever existed: 27c7383c…2660, unchanged since 0a17e39 (2026-10-10), inside 99 committed PDFs, 8 JSON fixtures and 6 Rust oracle files. Compared, never recomputed, by the rebase (fingerprint only), the envelope invariant and the export (sheetId and fingerprint). The database stores the document as jsonb and no fingerprint. Full census: R5_DOCUMENT_SHA256_CENSUS.md.

B.3.2 The matrix (real data, node scripts/r5-fingerprints.mjs, cross-checked in Rust)

Artifact Today (file bytes) A: to_js_pretty + LF A without LF B: RFC 8785 B validated form
draft/document.json 27c7383c…2660 90f668a3…aa1d 79f143c0…eb51 a56e4a24…88a1 12d78872…74e2
legacy-import/document.json 91e459f4…14c2 8c94a464…adc2 (Node) 6fb31371…7f66 (Node) d8b5f2cf…81eb (Node) 1217637e…c024 (Node)
draft/report-migration-map.json 4139c51d…fce9 4139c51d…fce9 (same) 01fac788…3471 f082b66f…2ca6 n/a
draft/assets.json, sheet-ref.json, legacy-id-map.json, legacy assets.json file = A with LF (no short array to collapse)

Rust reproduction (throwaway experiment on the core crate, to_js_pretty, to_js_compact, a 40-line JCS on js_number): every draft digest identical to Node, byte for byte; the legacy-import digests differ through the single tie value (F1, §B.3.4); the RFC vectors pass; JCS idempotent. Node vs Rust JSON.stringify(v, null, 2) over the 134 JSON fixtures of the repository: 2 differ, both the legacy baseline, both on that value; oracles/characterisation.json is not parseable by serde (lone surrogates, R2 L1 rule).

Encoding change vs content change: the today / A / B columns of one row are three encodings of one value (parse → deep equal; validated_equals_input is false only because the validated form fills defaults, not because content differs). A change of the draft content changes every column.

B.3.3 Facts the brief asked for

  • UTF-16 key order: every key of every artifact is ASCII; UTF-16 order equals code-point and byte order on this data. The Rust canonicaliser sorts by encode_utf16() anyway (the RFC vector with U+0080, ö, €, 😀 and U+FB33 passes).
  • Numbers: 208 (draft) and 195 (legacy) tokens, 29 and 36 non-integer, none negative, no exponent, no -0, no integer beyond 2^53; all already in ECMAScript Number::toString form. JCS mandates exactly that form; the core's js_number implements it with one tie defect (F1).
  • Unicode: 788 non-ASCII strings (precomposed French accents, « », –, °, —), all NFC, no control character, no escape, no astral character, no lone surrogate. JCS does not normalise; a lone surrogate (I-JSON forbids it) is refused by the core (L1) and never produced by the pipeline.
  • I-JSON: satisfied by both documents (no duplicate key, numbers within f64, valid Unicode).
  • Cost and dependencies: option A costs nothing (the core already has to_js_pretty, D-R11); option B costs about 40 lines in crates/core/src/json.rs over js_number and serde_json's escaper, no new crate (serde_jcs, listed as rejected in the review, is not needed; zmij 1.0.23 is already locked through serde_json 1.0.154); a 15-line Node twin for tests and tooling. Hashing: sha2 becomes a normal dependency of the core (today dev-only) or the hash stays outside the library (the tool computes it); the wasm bundle grows by the sha256 code only if the browser must verify fingerprints, which no current path does.
  • Platform determinism: B is a pure function of the parsed value (sort, ES6 numbers, fixed escapes); Node 24 and Rust 1.99 agree on every draft artifact. A depends on the writer's layout rules; today's value depends on Prettier 3.8.3's line fitting.
  • Database: a PostgreSQL 17 jsonb round trip (PGlite, packages/supabase/scripts/r5-jsonb-fingerprint.mjs) keeps B for both documents and breaks today's value and A (keys reordered, scale kept). B is the only candidate a service could recompute from anatomical_sheet_versions.content at Step 13.

B.3.4 F1: js_number prints exact ties upward

crates/core/src/json.rs takes the shortest digits from format!("{:e}") (Grisu); on an exact decimal tie between two shortest candidates ECMAScript picks the even one, Grisu the upper one. Measured: 840.1692504882812 (legacy annotation cx, from Horse.svg) prints …813; 7,401 of 100,000 float32-exact values in 0..2000 and 769 of 100,000 random doubles differ; zmij (serde_json's formatter) differs on none. Reached by the canvas layer (R3), the PDF writer, the archive attachment, the R2 contract tests. Invisible today: the curated draft carries no tie value, and sheet_contract.rs re-prints its expected text through to_js_pretty, so the oracle cannot see it. Proposed fix (after your go, before R5.2): digits from zmij (or serde_json::to_string(&f64)), the two tie values and the 8 long decimals of the legacy SVG as unit tests, zod_oracle comparing against the JS text recorded in the oracle rather than a re-print.

B.3.5 Compatibility mechanism (approved at A.5, applied at R5.6)

The sheet side carries the convention and the aliases; the report contract does not change.

  • sheet-ref.json (and the bundle payload ref, and anatomical_sheet_versions when Step 13 stores a fingerprint): document_sha256 holds the current convention's value; a new fingerprint block names the convention ("algorithm": "rfc8785-sha256" for B, "js-pretty-sha256" for A) and lists aliases: the fingerprints of the same content under earlier conventions, computed once at the switch from the frozen P4 oracle (27c7383c…2660 for the current draft). The alias is bound to identical content by construction (the oracle records old and new over the same bytes), so a report stamped 27c7… is the same sheet, not a rebase.
  • The rebase and the envelope invariant treat a fingerprint equal to the current value or to an alias as "same document". Gated readers: report/rebase.rs, archive/read.rs, session.rs, pdf/write/input.rs (your ask-before-touch rule; the change is one membership test).
  • A new report is stamped with the current value; an old archive opens with no message and no rebase; an archive on an older content revision rebases as today. No archive is rewritten; the exported envelope keeps what it holds.
  • Report v3 keeps sheet.documentSha256 (64 hex) so the N-1 reader and every fixture stay valid; the convention's name lives with the sheet, which is the one place that knows which rule produced its value.
  • Fixtures: 99 PDFs, 8 JSON fixtures and 6 oracles keep 27c7… untouched (they remain valid archives of the same content through the alias).

B.3.6 Options compared

A: bytes the Rust pipeline emits (to_js_pretty + LF) B: RFC 8785 of the validated document C: Rust reproduces Prettier's bytes
Value at the switch 90f668a3… (changes once) 12d78872… (changes once) 27c7383c… (unchanged)
Needs the alias mechanism to avoid a visible migration yes yes no
Survives a reformat, a writer change, a jsonb round trip no yes no
Code 0 lines (exists) ~40 lines Rust + 15 Node, no crate a Prettier-json subset printer (fill rule for scalar arrays, 80 columns), bound forever to a formatter's layout
Risk identity re-tied to a serialiser (the Prettier problem moved to Rust) one more concept; the canonical form must be named (input or validated) fragile: any Prettier rule the data exercises later changes the value
Review's row D-R5 rev 2 Annex C preference, BDR-045 rejected rev 1 (in Node form)

Recommendation: B over the validated form (what every reader holds, recomputable from the database after validation), the alias mechanism of §B.3.5, the convention named in sheet-ref.json, F1 fixed first. The old value 27c7… stays valid forever as an alias; no archive sees a message for the switch. If you prefer no new concept at all, C is possible on this data (the only Prettier effect measured is the collapse of short scalar arrays), but it keeps the fingerprint chained to a formatter's layout, which Annex C refused.

B.4 Business invariants (brief §4) and where they are proven

Invariant Oracle or test
stable ids P3 (uuid, slug), P1 documents, legacy-id-map.json
zones and anatomical points, main and detail views P1 documents (annotations, views), P2 annotation rows and records
annotation ↔ observation relations (finding_zone_id, opens_view_id) P1, integrity rules (R2 core, 21 codes)
vertebral configurations P1 vertebral_config, P2 region inferences
bilingual data P1, P3 slug over every name.fr / name.en
historical references curation/references.json (authored, never generated), P1
existing reports and bilans P5 (migration map, 612 leaves), the R2 rebase oracles, the R4 PDF fixtures, the alias mechanism
no sheet validated by a migration the seed inserts a draft (insert_sheet_draft, status draft); publication stays a platform-admin action

B.5 Slices after R5.1 (proposal for the checkpoint; nothing started)

Slice Deliverable Proof
R5.1 this freeze and study oracles committed, generator test green, checkpoint report
R5.1b F1 fix in json.rs; zod_oracle compares the recorded JS text tie unit tests; every R2/R3/R4 suite unchanged (no tie in the draft)
R5.2 legacy importer in Rust (crates/core/src/curation/legacy): SVG sources through the R3 reader, catalog, regions, ids, anomalies, stats; import-equine emits legacy-import/ P1 bytes of document.json, legacy-id-map.json, assets.json reproduced exactly; IMPORT-REPORT compared as parsed tables (P2 pre-Prettier digest) or byte-exact if the Prettier markdown effect is nil
R5.3 changeset schema, targets, ops, applier, migration map, stats, seed SQL P1 document.json, sheet-ref.json, report-migration-map.json, equine_draft.sql byte-exact under the decided fingerprint; P2 825 records equal; 612-leaf suite ported
R5.4 review overlays and app drawings on the R3 layer and compose; CHANGELOG and REVIEW-REPORT parsed-tree comparison against the xmldom files (declaration, prefixes, whitespace declared); markdown as parsed tables
R5.5 upload-sheet-assets binary (reqwest or ureq, service key from the environment), operator runbook local stack run, SUPABASE.md §3.1
R5.6 D-R5 switch: fingerprint convention, aliases, the one membership test in the gated readers (your go per file), seed regenerated once by the Rust tool, CHANGELOG old archives open with no message; a report on an older content revision still rebases; N-1 reader green
R5.7 consumer census, packages/anatomy and packages/domain deleted (packages/domain/tests/fixtures/report-v2.*.json moved under crates/core/tests/fixtures first, r4.census.test.ts xmldom assertion inverted), @xmldom/xmldom and prettier leave the anatomy lockfile, docs (CLAUDE.md map, WORKSPACE, packages/README, RUST.md §11), ledger of the deleted tests grep census, pnpm check:all, test counts reconciled

Order constraint: R5.6 needs the owner's D-R5 decision; R5.2 needs R5.1b.

B.6 Test ledger

crates/core/tests/curation/LEDGER.tsv at R5.7: one row per TS case of packages/anatomy/tests and packages/domain/tests at the freeze commit (file, title, disposition ported / replaced / kept / deleted, target). Counts at the freeze: anatomy 131 cases in 16 files (the R5.1 generator adds 6), domain 262 in 13 files.