test(anatomy): R5.1 freeze of the TypeScript pipeline (oracles r5: intermediates, ids, fingerprint matrix) and the D-R5 study scripts; RUST.md §11

This commit is contained in:
bchanot committed 2026-10-22 20:18:12 +02:00
1 parent 9c0117a692
commit 60cc0f5fa8
8 files changed
+33630 -1

No files matched your search

+203 -1
View File
@@ -2089,7 +2089,7 @@ R5 prerequisites (the pipeline port): `packages/anatomy` (curation pipeline,
importer, `render/`, the review drawings) and `packages/domain` are the last
TypeScript business packages; their oracles (the seed golden, the 612-leaf
semantics, the determinism and count suites, the R3 layer oracle) are in
place; `documentSha256` will change once (D-R5) and every older archive will
place; `documentSha256` will change once (D-R5 rev 2; revised by the R5.1 study, §11.4 to §11.6) and every older archive will
rebase on import; `@xmldom/xmldom`, `tsx` and `prettier` on generated files
leave with them.
@@ -2561,3 +2561,205 @@ branch, all in 0dc1ba2 (R4.8, authorised), none after. Merge: `gitflow
finish` on `feature/rust-r4-pdf-core` (no-ff into develop, branch
deleted local and remote), recorded in `.claude/memory/journal.md`.
Release 0.2.0, faunosteo.com and R5 untouched.
## 11. R5: the anatomical pipeline in Rust (R5.1 freeze and study, 2026-10-22)
Plan: `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md`. Branch
`feature/rust-r5-pipeline-core`, cut from develop 9c0117a (R4 merged). The
owner's brief of 2026-10-22 authorises R5.1 only: characterise and freeze
the TypeScript pipeline, study the fingerprint decision D-R5, stop before
any Rust pipeline code and before any change to `documentSha256`.
### 11.1 Inventory of the TypeScript pipeline
`.claude/audits/R5_PIPELINE_INVENTORY.md`. `packages/anatomy`: 43 source
files, 4,897 lines (legacy importer 1,842, curation 2,329, frozen renderer
553, CLI 119), 131 test cases in 16 files, runtime libraries Prettier
3.8.3, @xmldom/xmldom 0.9.12 and Zod 3.25.76 (the first two declared as
devDependencies). It emits 18 generated files: 7 JSON and 3 Markdown
through Prettier, 5 SVGs through xmldom's serialiser (the cleaned
`horse.svg` and 4 review drawings), 1 raw SQL template, 2 byte copies of
the legacy SVGs. The authored changeset holds 45 changes (14 A, 22 B, 9 C),
687 ops (587 set, 45 cite, 33 rename, 18 add, 4 remove, 0 move) and 48
references. `packages/domain` is imported by `packages/anatomy` only (5
runtime symbols, each with its R2 counterpart); the crate tests read
`packages/domain/tests/fixtures/report-v2.*.json`, and
`r4.census.test.ts` pins xmldom under `packages/anatomy`: both move at
the deletion slice. The upload tool (`upload-sheet-assets.ts`, 99 lines)
depends on the `assets.json` contract only: bucket
`anatomical-sheet-assets`, key `<sha256>.svg`, `upsert: false`, bytes
re-hashed against the manifest, RPC `confirm_sheet_asset_upload`.
JavaScript facts a Rust port must reproduce or declare: key insertion
order of the working document (`review_status` appended last, not the
schema order a typed struct emits), `JSON.stringify(v, null, 2)` then
Prettier (measured effect on today's data: short scalar arrays collapsed
when they fit in 80 columns, a trailing LF; Markdown: table padding only,
134, 863 and 52 lines in the three reports), xmldom's serialiser
(declaration kept, `/>` without space, attributes in document order,
namespace declared once), `deterministicUuid` (first 16 bytes of sha256 of
the name, version nibble 5, variant 10xx), `slugify` (NFD, combining marks
U+0300 to U+036F stripped, `toLowerCase`, `[^a-z0-9]+` → `-`, 64 chars;
never fires on today's data), UTF-16 `<` sorting, `Math.round` toward +∞
on −x.5, `Number('')` = 0, Unicode `\s`, `split(':', 2)`, fatal
`TextDecoder` for the SVGs. No clock, no randomness, no localeCompare.
### 11.2 Oracles frozen (`crates/core/tests/oracles/r5/`)
Generator `packages/anatomy/tests/r5.pipeline-oracles.test.ts`
(`MAKE_R5_ORACLES=1` writes; otherwise the pipeline must still match its
freeze; 6 cases, among them a two-run determinism check of every artifact
and the ledger check against the committed folders). Never regenerated
without the owner.
| Oracle | Content | Size |
|---|---|---|
| `pipeline.json` | the ImportResult without the document (sheetId, viewBoxes, stats, source digests, assets, 27 anomalies, issues, region inferences, zone stats, annotation rows), the DraftResult without the documents (46 change summaries, 825 records with before and after, stats before and after, issues, migration stats 792 = 571 + 180 + 7 + 34), the ledger of the 16 text artifacts (folder, file, bytes, sha256; the 2 legacy SVG copies are pinned by the golden test; for the 3 Markdown files the sha256 before Prettier and `prettier_changed: true`) | 1.3 MB |
| `ids.json` | `deterministicUuid` over the 3 pipeline names and 3 edges; `slugify` over 17 edge labels and every `name.fr` / `name.en` of both documents (801 rows) | 87 KB |
| `fingerprints.json` | per JSON artifact: today's file-bytes sha256, A (`JSON.stringify(v, null, 2)` with and without LF, compact), B (RFC 8785); for the two documents also the validated form (defaults filled) under A and B; the ES6 text of the 65 non-integer numbers; the two RFC 8785 vectors | 13 KB |
The committed generated folders stay the byte oracle (`seed.golden.test.ts`
pins them to a fresh run); the migration semantics stay pinned by
`report-migration-map.json`, the 612-leaf suite and the R2 oracle
`migration-map.draft.json`; the review drawings by their xmldom bytes and
the R3 layer and highlight oracles. `pnpm import:equine` was not run (it
rewrites the folders; the golden proves the same bytes).
### 11.3 Tests and results
`pnpm --filter @faunosteo/anatomy test:run`: 137 cases in 17 files (131 of
the frozen pipeline plus the 6 of the generator), green;
`@faunosteo/domain` 262, green; lint and typecheck of the anatomy
package green; `git status` shows no change under `supabase/seed`, the
goldens, the R2 to R4 oracles or the R4 digests.
### 11.4 Historical results: no gap, one defect found in the core
Every generated file reproduces byte for byte from a fresh run. One
fingerprint has ever been produced, `27c7383c…2660`, unchanged since
0a17e39 (2026-10-10), carried by 99 of the 216 committed PDFs, 8 JSON
fixtures and 6 Rust oracle files (`.claude/audits/R5_DOCUMENT_SHA256_CENSUS.md`).
F1, found by the study (`crates/core/src/json.rs`, `js_number`): the
shortest digits come from `format!("{:e}")`, which rounds an exact
decimal tie upward, where ECMAScript, Python and `zmij` (serde_json's
formatter, already locked) round to even. `840.1692504882812` (legacy
annotation `cx`, from Horse.svg, a float32-exact value) prints
`…813`. Measured: 1 of the 8 long decimals of the legacy SVGs, 7,401 of
100,000 float32-exact values in 0..2000, 769 of 100,000 random doubles;
`zmij` differs on none. Over the 134 JSON fixtures of the repository,
Node and Rust `JSON.stringify(v, null, 2)` differ on 2 files, both the
legacy baseline, both on that value. Invisible today: the curated draft
holds no tie value, and the R2 contract test `sheet_contract.rs` re-prints
its expected text through `to_js_pretty`, so it cannot see a
number-printing difference. Reached by the canvas layer (R3), the PDF
writer, the archive attachment and `report-compare`. Not fixed in R5.1:
it changes core bytes for tie values (D-R11), so the fix (digits from
`zmij`, the tie values as unit tests, `zod_oracle` compared against the
recorded JS text) waits for the owner's go and precedes any Rust importer.
The §10.10 sentence "`documentSha256` will change once (D-R5) and every
older archive will rebase on import" describes D-R5 rev 2; the owner's
Annex C amendment (BDR-045) supersedes it, and §11.5 studies both.
### 11.5 D-R5: the fingerprint, facts and comparison
Today: `documentSha256 = sha256(Prettier(JSON.stringify(document, null,
2)))`, the exact bytes of `draft/document.json` in its input form
(defaults omitted), copied into `sheet-ref.json` and the migration map,
compared and never recomputed by the rebase (fingerprint only), the
envelope invariant and the export (sheetId and fingerprint). The database
stores the document as `jsonb` and no fingerprint.
Matrix on the real data (`node scripts/r5-fingerprints.mjs`; cross-checked
by a throwaway Rust experiment on the core crate: `to_js_pretty`,
`to_js_compact` and a 40-line RFC 8785 canonicaliser over `js_number`,
RFC vectors passed, idempotent):
| Artifact | Today | A `to_js_pretty` + LF | B RFC 8785, input form | B, validated form |
|---|---|---|---|---|
| draft/document.json | 27c7383c…2660 | 90f668a3…aa1d | a56e4a24…88a1 | 12d78872…74e2 |
| legacy-import/document.json | 91e459f4…14c2 | 8c94a464…adc2 | d8b5f2cf…81eb | 1217637e…c024 |
| draft/report-migration-map.json | 4139c51d…fce9 | 4139c51d…fce9 | f082b66f…2ca6 | |
Rust reproduces every draft digest byte for byte; the legacy digests
differ through F1 only. The three columns of one row are three encodings
of one value (parse and deep-compare); a content change moves every
column. Keys: all ASCII in every artifact (UTF-16, code-point and byte
order coincide; the Rust canonicaliser sorts by `encode_utf16()` and
passes the RFC vector with U+0080, ö, €, 😀 and U+FB33). Numbers: 29 and
36 non-integer tokens, none negative, no exponent, no `-0`, none beyond
2^53, all already in ECMAScript `Number::toString` form (what JCS
mandates). Unicode: 788 non-ASCII strings, all NFC, no control, no escape,
no astral, no lone surrogate (I-JSON satisfied; a lone surrogate is
refused by the core, L1). Cost: A costs nothing (`to_js_pretty` exists);
B costs about 40 lines in `json.rs`, no crate (`serde_jcs` not needed;
`sha2` would become a normal dependency of the core or stay in the tool).
Determinism: B is a pure function of the parsed value and Node 24 and Rust
1.99 agree on every draft artifact; A depends on the writer's layout
rules. Database: a PostgreSQL 17 `jsonb` round trip
(`packages/supabase/scripts/r5-jsonb-fingerprint.mjs`, PGlite) keeps B for
both documents and breaks today's value and A (keys reordered, numeric
scale kept): B is the only candidate a service could recompute from
`anatomical_sheet_versions.content`.
| | A: bytes the Rust pipeline emits | B: RFC 8785 of the validated document | C: Rust reproduces Prettier's bytes |
|---|---|---|---|
| value at the switch | changes once | changes once | unchanged |
| alias mechanism needed to avoid a visible migration | yes | yes | no |
| survives a reformat, a writer change, a jsonb round trip | no | yes | no |
| code | 0 lines | ~40 Rust + 15 Node, no crate | a Prettier-json subset printer, bound to a formatter's layout |
| source | D-R5 rev 2 | Annex C, BDR-045 | rejected rev 1 (Node form) |
Recommendation to the owner: B over the validated form, the alias
mechanism of §11.6, the convention named in `sheet-ref.json`, F1 fixed
first. C is possible on this data (the only Prettier effect is the
short-array collapse) but keeps the identity chained to a formatter.
### 11.6 Compatibility mechanism for the old fingerprints (proposal)
The sheet side carries the convention and the aliases; the report
contract does not change. `sheet-ref.json`, the bundle payload `ref` and,
at Step 13, the stored sheet version carry `document_sha256` (the current
convention's value) plus a `fingerprint` block: `algorithm`
(`rfc8785-sha256`) and `aliases`, the fingerprints of the same content
under earlier conventions, computed once at the switch from the frozen
`fingerprints.json` (`27c7383c…2660` for the current draft). The rebase,
the envelope invariant and the export treat a fingerprint equal to the
current value or to an alias as the same document (one membership test in
`report/rebase.rs`, `archive/read.rs`, `session.rs`,
`pdf/write/input.rs`: gated files, the owner's go per file). A new report
is stamped with the current value; an archive stamped `27c7…` opens with
no message and no rebase; an archive on an older content revision rebases
as today; no archive, fixture or oracle is rewritten. Report v3 keeps
`sheet.documentSha256` (64 hex): the N-1 reader and every fixture stay
valid.
### 11.7 Residual risks of R5.1
F1 unfixed (declared, bounded: no tie value in the draft); the Rust
canonicaliser exists as an experiment only (not in the crate);
`validated_equals_input: false` in `fingerprints.json` records that the
input and validated forms differ by the filled defaults, so the decided
convention must name its form; `supabase/seed/**` is not in
`.prettierignore` (a root `pnpm format` would reformat Rust-written
files at the switch); the horse.svg asset bytes are xmldom's (re-cleaning
with the R3 reader would change the asset sha, id, document, SQL and the
Storage object); the slugify and region tables depend on ICU 78.3 (never
exercised by today's data).
### 11.8 Recommended slicing (nothing started)
R5.1b F1 fix (owner's go). R5.2 legacy importer (`crates/core`
curation/legacy, `import-equine` bin): document.json, legacy-id-map.json,
assets.json byte-exact, IMPORT-REPORT by table padding or parsed tables.
R5.3 changeset applier, migration map, stats, seed SQL: the draft JSON
files and the SQL under the decided fingerprint, the 825 records equal,
the 612-leaf suite ported. R5.4 review overlays and app drawings on the R3
layer and compose (parsed-tree comparison), CHANGELOG and REVIEW-REPORT.
R5.5 `upload-sheet-assets` bin, operator runbook. R5.6 the D-R5 switch
(convention, aliases, the membership test in the gated readers, seed
regenerated once by the Rust tool, `.prettierignore`, CHANGELOG). R5.7
consumer census, fixtures moved, `packages/anatomy` and `packages/domain`
deleted, xmldom and Prettier out of the anatomy lockfile, docs and ledger.
R5.6 needs the D-R5 decision; R5.2 needs R5.1b.