test(anatomy): R5.1 freeze of the TypeScript pipeline (oracles r5: intermediates, ids, fingerprint matrix) and the D-R5 study scripts; RUST.md §11
This commit is contained in:
1 parent
9c0117a692
commit
60cc0f5fa8
8 files changed
+33630
-1
No files matched your search
+203
-1
@@ -2089,7 +2089,7 @@ R5 prerequisites (the pipeline port): `packages/anatomy` (curation pipeline,
|
||||
importer, `render/`, the review drawings) and `packages/domain` are the last
|
||||
TypeScript business packages; their oracles (the seed golden, the 612-leaf
|
||||
semantics, the determinism and count suites, the R3 layer oracle) are in
|
||||
place; `documentSha256` will change once (D-R5) and every older archive will
|
||||
place; `documentSha256` will change once (D-R5 rev 2; revised by the R5.1 study, §11.4 to §11.6) and every older archive will
|
||||
rebase on import; `@xmldom/xmldom`, `tsx` and `prettier` on generated files
|
||||
leave with them.
|
||||
|
||||
@@ -2561,3 +2561,205 @@ branch, all in 0dc1ba2 (R4.8, authorised), none after. Merge: `gitflow
|
||||
finish` on `feature/rust-r4-pdf-core` (no-ff into develop, branch
|
||||
deleted local and remote), recorded in `.claude/memory/journal.md`.
|
||||
Release 0.2.0, faunosteo.com and R5 untouched.
|
||||
|
||||
## 11. R5: the anatomical pipeline in Rust (R5.1 freeze and study, 2026-10-22)
|
||||
|
||||
Plan: `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md`. Branch
|
||||
`feature/rust-r5-pipeline-core`, cut from develop 9c0117a (R4 merged). The
|
||||
owner's brief of 2026-10-22 authorises R5.1 only: characterise and freeze
|
||||
the TypeScript pipeline, study the fingerprint decision D-R5, stop before
|
||||
any Rust pipeline code and before any change to `documentSha256`.
|
||||
|
||||
### 11.1 Inventory of the TypeScript pipeline
|
||||
|
||||
`.claude/audits/R5_PIPELINE_INVENTORY.md`. `packages/anatomy`: 43 source
|
||||
files, 4,897 lines (legacy importer 1,842, curation 2,329, frozen renderer
|
||||
553, CLI 119), 131 test cases in 16 files, runtime libraries Prettier
|
||||
3.8.3, @xmldom/xmldom 0.9.12 and Zod 3.25.76 (the first two declared as
|
||||
devDependencies). It emits 18 generated files: 7 JSON and 3 Markdown
|
||||
through Prettier, 5 SVGs through xmldom's serialiser (the cleaned
|
||||
`horse.svg` and 4 review drawings), 1 raw SQL template, 2 byte copies of
|
||||
the legacy SVGs. The authored changeset holds 45 changes (14 A, 22 B, 9 C),
|
||||
687 ops (587 set, 45 cite, 33 rename, 18 add, 4 remove, 0 move) and 48
|
||||
references. `packages/domain` is imported by `packages/anatomy` only (5
|
||||
runtime symbols, each with its R2 counterpart); the crate tests read
|
||||
`packages/domain/tests/fixtures/report-v2.*.json`, and
|
||||
`r4.census.test.ts` pins xmldom under `packages/anatomy`: both move at
|
||||
the deletion slice. The upload tool (`upload-sheet-assets.ts`, 99 lines)
|
||||
depends on the `assets.json` contract only: bucket
|
||||
`anatomical-sheet-assets`, key `<sha256>.svg`, `upsert: false`, bytes
|
||||
re-hashed against the manifest, RPC `confirm_sheet_asset_upload`.
|
||||
|
||||
JavaScript facts a Rust port must reproduce or declare: key insertion
|
||||
order of the working document (`review_status` appended last, not the
|
||||
schema order a typed struct emits), `JSON.stringify(v, null, 2)` then
|
||||
Prettier (measured effect on today's data: short scalar arrays collapsed
|
||||
when they fit in 80 columns, a trailing LF; Markdown: table padding only,
|
||||
134, 863 and 52 lines in the three reports), xmldom's serialiser
|
||||
(declaration kept, `/>` without space, attributes in document order,
|
||||
namespace declared once), `deterministicUuid` (first 16 bytes of sha256 of
|
||||
the name, version nibble 5, variant 10xx), `slugify` (NFD, combining marks
|
||||
U+0300 to U+036F stripped, `toLowerCase`, `[^a-z0-9]+` → `-`, 64 chars;
|
||||
never fires on today's data), UTF-16 `<` sorting, `Math.round` toward +∞
|
||||
on −x.5, `Number('')` = 0, Unicode `\s`, `split(':', 2)`, fatal
|
||||
`TextDecoder` for the SVGs. No clock, no randomness, no localeCompare.
|
||||
|
||||
### 11.2 Oracles frozen (`crates/core/tests/oracles/r5/`)
|
||||
|
||||
Generator `packages/anatomy/tests/r5.pipeline-oracles.test.ts`
|
||||
(`MAKE_R5_ORACLES=1` writes; otherwise the pipeline must still match its
|
||||
freeze; 6 cases, among them a two-run determinism check of every artifact
|
||||
and the ledger check against the committed folders). Never regenerated
|
||||
without the owner.
|
||||
|
||||
| Oracle | Content | Size |
|
||||
|---|---|---|
|
||||
| `pipeline.json` | the ImportResult without the document (sheetId, viewBoxes, stats, source digests, assets, 27 anomalies, issues, region inferences, zone stats, annotation rows), the DraftResult without the documents (46 change summaries, 825 records with before and after, stats before and after, issues, migration stats 792 = 571 + 180 + 7 + 34), the ledger of the 16 text artifacts (folder, file, bytes, sha256; the 2 legacy SVG copies are pinned by the golden test; for the 3 Markdown files the sha256 before Prettier and `prettier_changed: true`) | 1.3 MB |
|
||||
| `ids.json` | `deterministicUuid` over the 3 pipeline names and 3 edges; `slugify` over 17 edge labels and every `name.fr` / `name.en` of both documents (801 rows) | 87 KB |
|
||||
| `fingerprints.json` | per JSON artifact: today's file-bytes sha256, A (`JSON.stringify(v, null, 2)` with and without LF, compact), B (RFC 8785); for the two documents also the validated form (defaults filled) under A and B; the ES6 text of the 65 non-integer numbers; the two RFC 8785 vectors | 13 KB |
|
||||
|
||||
The committed generated folders stay the byte oracle (`seed.golden.test.ts`
|
||||
pins them to a fresh run); the migration semantics stay pinned by
|
||||
`report-migration-map.json`, the 612-leaf suite and the R2 oracle
|
||||
`migration-map.draft.json`; the review drawings by their xmldom bytes and
|
||||
the R3 layer and highlight oracles. `pnpm import:equine` was not run (it
|
||||
rewrites the folders; the golden proves the same bytes).
|
||||
|
||||
### 11.3 Tests and results
|
||||
|
||||
`pnpm --filter @faunosteo/anatomy test:run`: 137 cases in 17 files (131 of
|
||||
the frozen pipeline plus the 6 of the generator), green;
|
||||
`@faunosteo/domain` 262, green; lint and typecheck of the anatomy
|
||||
package green; `git status` shows no change under `supabase/seed`, the
|
||||
goldens, the R2 to R4 oracles or the R4 digests.
|
||||
|
||||
### 11.4 Historical results: no gap, one defect found in the core
|
||||
|
||||
Every generated file reproduces byte for byte from a fresh run. One
|
||||
fingerprint has ever been produced, `27c7383c…2660`, unchanged since
|
||||
0a17e39 (2026-10-10), carried by 99 of the 216 committed PDFs, 8 JSON
|
||||
fixtures and 6 Rust oracle files (`.claude/audits/R5_DOCUMENT_SHA256_CENSUS.md`).
|
||||
|
||||
F1, found by the study (`crates/core/src/json.rs`, `js_number`): the
|
||||
shortest digits come from `format!("{:e}")`, which rounds an exact
|
||||
decimal tie upward, where ECMAScript, Python and `zmij` (serde_json's
|
||||
formatter, already locked) round to even. `840.1692504882812` (legacy
|
||||
annotation `cx`, from Horse.svg, a float32-exact value) prints
|
||||
`…813`. Measured: 1 of the 8 long decimals of the legacy SVGs, 7,401 of
|
||||
100,000 float32-exact values in 0..2000, 769 of 100,000 random doubles;
|
||||
`zmij` differs on none. Over the 134 JSON fixtures of the repository,
|
||||
Node and Rust `JSON.stringify(v, null, 2)` differ on 2 files, both the
|
||||
legacy baseline, both on that value. Invisible today: the curated draft
|
||||
holds no tie value, and the R2 contract test `sheet_contract.rs` re-prints
|
||||
its expected text through `to_js_pretty`, so it cannot see a
|
||||
number-printing difference. Reached by the canvas layer (R3), the PDF
|
||||
writer, the archive attachment and `report-compare`. Not fixed in R5.1:
|
||||
it changes core bytes for tie values (D-R11), so the fix (digits from
|
||||
`zmij`, the tie values as unit tests, `zod_oracle` compared against the
|
||||
recorded JS text) waits for the owner's go and precedes any Rust importer.
|
||||
|
||||
The §10.10 sentence "`documentSha256` will change once (D-R5) and every
|
||||
older archive will rebase on import" describes D-R5 rev 2; the owner's
|
||||
Annex C amendment (BDR-045) supersedes it, and §11.5 studies both.
|
||||
|
||||
### 11.5 D-R5: the fingerprint, facts and comparison
|
||||
|
||||
Today: `documentSha256 = sha256(Prettier(JSON.stringify(document, null,
|
||||
2)))`, the exact bytes of `draft/document.json` in its input form
|
||||
(defaults omitted), copied into `sheet-ref.json` and the migration map,
|
||||
compared and never recomputed by the rebase (fingerprint only), the
|
||||
envelope invariant and the export (sheetId and fingerprint). The database
|
||||
stores the document as `jsonb` and no fingerprint.
|
||||
|
||||
Matrix on the real data (`node scripts/r5-fingerprints.mjs`; cross-checked
|
||||
by a throwaway Rust experiment on the core crate: `to_js_pretty`,
|
||||
`to_js_compact` and a 40-line RFC 8785 canonicaliser over `js_number`,
|
||||
RFC vectors passed, idempotent):
|
||||
|
||||
| Artifact | Today | A `to_js_pretty` + LF | B RFC 8785, input form | B, validated form |
|
||||
|---|---|---|---|---|
|
||||
| draft/document.json | 27c7383c…2660 | 90f668a3…aa1d | a56e4a24…88a1 | 12d78872…74e2 |
|
||||
| legacy-import/document.json | 91e459f4…14c2 | 8c94a464…adc2 | d8b5f2cf…81eb | 1217637e…c024 |
|
||||
| draft/report-migration-map.json | 4139c51d…fce9 | 4139c51d…fce9 | f082b66f…2ca6 | |
|
||||
|
||||
Rust reproduces every draft digest byte for byte; the legacy digests
|
||||
differ through F1 only. The three columns of one row are three encodings
|
||||
of one value (parse and deep-compare); a content change moves every
|
||||
column. Keys: all ASCII in every artifact (UTF-16, code-point and byte
|
||||
order coincide; the Rust canonicaliser sorts by `encode_utf16()` and
|
||||
passes the RFC vector with U+0080, ö, €, 😀 and U+FB33). Numbers: 29 and
|
||||
36 non-integer tokens, none negative, no exponent, no `-0`, none beyond
|
||||
2^53, all already in ECMAScript `Number::toString` form (what JCS
|
||||
mandates). Unicode: 788 non-ASCII strings, all NFC, no control, no escape,
|
||||
no astral, no lone surrogate (I-JSON satisfied; a lone surrogate is
|
||||
refused by the core, L1). Cost: A costs nothing (`to_js_pretty` exists);
|
||||
B costs about 40 lines in `json.rs`, no crate (`serde_jcs` not needed;
|
||||
`sha2` would become a normal dependency of the core or stay in the tool).
|
||||
Determinism: B is a pure function of the parsed value and Node 24 and Rust
|
||||
1.99 agree on every draft artifact; A depends on the writer's layout
|
||||
rules. Database: a PostgreSQL 17 `jsonb` round trip
|
||||
(`packages/supabase/scripts/r5-jsonb-fingerprint.mjs`, PGlite) keeps B for
|
||||
both documents and breaks today's value and A (keys reordered, numeric
|
||||
scale kept): B is the only candidate a service could recompute from
|
||||
`anatomical_sheet_versions.content`.
|
||||
|
||||
| | A: bytes the Rust pipeline emits | B: RFC 8785 of the validated document | C: Rust reproduces Prettier's bytes |
|
||||
|---|---|---|---|
|
||||
| value at the switch | changes once | changes once | unchanged |
|
||||
| alias mechanism needed to avoid a visible migration | yes | yes | no |
|
||||
| survives a reformat, a writer change, a jsonb round trip | no | yes | no |
|
||||
| code | 0 lines | ~40 Rust + 15 Node, no crate | a Prettier-json subset printer, bound to a formatter's layout |
|
||||
| source | D-R5 rev 2 | Annex C, BDR-045 | rejected rev 1 (Node form) |
|
||||
|
||||
Recommendation to the owner: B over the validated form, the alias
|
||||
mechanism of §11.6, the convention named in `sheet-ref.json`, F1 fixed
|
||||
first. C is possible on this data (the only Prettier effect is the
|
||||
short-array collapse) but keeps the identity chained to a formatter.
|
||||
|
||||
### 11.6 Compatibility mechanism for the old fingerprints (proposal)
|
||||
|
||||
The sheet side carries the convention and the aliases; the report
|
||||
contract does not change. `sheet-ref.json`, the bundle payload `ref` and,
|
||||
at Step 13, the stored sheet version carry `document_sha256` (the current
|
||||
convention's value) plus a `fingerprint` block: `algorithm`
|
||||
(`rfc8785-sha256`) and `aliases`, the fingerprints of the same content
|
||||
under earlier conventions, computed once at the switch from the frozen
|
||||
`fingerprints.json` (`27c7383c…2660` for the current draft). The rebase,
|
||||
the envelope invariant and the export treat a fingerprint equal to the
|
||||
current value or to an alias as the same document (one membership test in
|
||||
`report/rebase.rs`, `archive/read.rs`, `session.rs`,
|
||||
`pdf/write/input.rs`: gated files, the owner's go per file). A new report
|
||||
is stamped with the current value; an archive stamped `27c7…` opens with
|
||||
no message and no rebase; an archive on an older content revision rebases
|
||||
as today; no archive, fixture or oracle is rewritten. Report v3 keeps
|
||||
`sheet.documentSha256` (64 hex): the N-1 reader and every fixture stay
|
||||
valid.
|
||||
|
||||
### 11.7 Residual risks of R5.1
|
||||
|
||||
F1 unfixed (declared, bounded: no tie value in the draft); the Rust
|
||||
canonicaliser exists as an experiment only (not in the crate);
|
||||
`validated_equals_input: false` in `fingerprints.json` records that the
|
||||
input and validated forms differ by the filled defaults, so the decided
|
||||
convention must name its form; `supabase/seed/**` is not in
|
||||
`.prettierignore` (a root `pnpm format` would reformat Rust-written
|
||||
files at the switch); the horse.svg asset bytes are xmldom's (re-cleaning
|
||||
with the R3 reader would change the asset sha, id, document, SQL and the
|
||||
Storage object); the slugify and region tables depend on ICU 78.3 (never
|
||||
exercised by today's data).
|
||||
|
||||
### 11.8 Recommended slicing (nothing started)
|
||||
|
||||
R5.1b F1 fix (owner's go). R5.2 legacy importer (`crates/core`
|
||||
curation/legacy, `import-equine` bin): document.json, legacy-id-map.json,
|
||||
assets.json byte-exact, IMPORT-REPORT by table padding or parsed tables.
|
||||
R5.3 changeset applier, migration map, stats, seed SQL: the draft JSON
|
||||
files and the SQL under the decided fingerprint, the 825 records equal,
|
||||
the 612-leaf suite ported. R5.4 review overlays and app drawings on the R3
|
||||
layer and compose (parsed-tree comparison), CHANGELOG and REVIEW-REPORT.
|
||||
R5.5 `upload-sheet-assets` bin, operator runbook. R5.6 the D-R5 switch
|
||||
(convention, aliases, the membership test in the gated readers, seed
|
||||
regenerated once by the Rust tool, `.prettierignore`, CHANGELOG). R5.7
|
||||
consumer census, fixtures moved, `packages/anatomy` and `packages/domain`
|
||||
deleted, xmldom and Prettier out of the anatomy lockfile, docs and ledger.
|
||||
R5.6 needs the D-R5 decision; R5.2 needs R5.1b.
|
||||
Reference in new issue
Block a user