chore(memory): R5 plan, pipeline inventory and documentSha256 census audits, TODO R5.1, journal, LRN-070, EVAL-026
This commit is contained in:
1 parent
60cc0f5fa8
commit
64d4595e73
7 files changed
+669
-1
No files matched your search
@@ -0,0 +1,204 @@
|
||||
# R5 census: sheet fingerprint `documentSha256`
|
||||
|
||||
Date 2026-10-22, branch feature/rust-r5-pipeline-core (cut at 9c0117a). Read-only census by an analyzer agent, saved by the orchestrator. Facts only, no recommendation.
|
||||
Names covered: `documentSha256` (camelCase contracts), `document_sha256` (sheet-ref.json, bundle payload, Rust fields), `SheetRef` / `SheetSnapshotRef` / `BundledRef`, `sheet-ref.json`.
|
||||
|
||||
## Summary
|
||||
|
||||
1. There is exactly one producer: `sheetRef()` in packages/anatomy/src/curation/pipeline.ts:159-165. It computes sha256 over `formatJson(document)`, which is the Prettier 3.8.3 json printer run over `JSON.stringify(v,null,2)`.
|
||||
2. Nothing in Rust, SQL, apps/app, packages/supabase or scripts computes a document fingerprint. In crates/core, `sha2` is a dev-dependency only.
|
||||
3. SQL has no fingerprint column, CHECK, trigger or RPC. Measured on PGlite (PG 17.5), `jsonb::text` of the draft hashes to eaf891f4…, not 27c7383c…. jsonb reorders keys (length, then bytes) and keeps numeric scale (`1.50`, `1.0`).
|
||||
4. One real value exists: 27c7383c445c73dc2001ebf4794a3ea63d416910bbbaa0d7848dd720de7c2660. It equals sha256(draft/document.json, 302,794 B), has been unchanged since 0a17e39 (2026-10-10), and belongs to sheetId 503dc301-… (uuid derived from `faunosteo:sheet:equine:v1`) with versionId null.
|
||||
5. That value sits in sheet-ref.json, in report-migration-map.json, in 8 app JSON fixtures, in 6 Rust oracle files, and inside the attachments of 99 of the 216 committed PDFs.
|
||||
6. The rebase decides on the fingerprint alone (rebase.rs:40). The envelope invariant (read.rs:116-119) and the export (input.rs:206-207) compare sheetId and fingerprint. versionId is never compared anywhere.
|
||||
7. Mismatch on import or draft restore: the report is rebased. `report.sheet` takes the loaded ref, unknown zones and findings get `unresolved: true`, labels are filled only where absent, nothing is deleted. Messages: `home.import.rebased`, `editor.draft_rebased`.
|
||||
8. Same fingerprint with a different sheetId: no rebase, so the report keeps the foreign sheetId and a later export is refused with `sheet_mismatch`, shown as the generic `editor.export_failed`.
|
||||
9. Both v2 imports and new reports get their stamp from the copies the pipeline writes (map.sheet, sheet-ref.json). No runtime check verifies that those copies agree or match document.json.
|
||||
10. `BundledRef.document_sha256` is loaded without any format check (loaded.rs:29-36). Report and snapshot refs are checked against `^[0-9a-f]{64}$`.
|
||||
11. The hashed bytes depend on Prettier's layout. The file differs from `JSON.stringify(v,null,2)+LF` from line 17 on, because Prettier collapses short arrays. It also differs from the parsed form: Zod or serde fill defaults, 173,136 vs 245,636 UTF-16 units compact.
|
||||
12. Draft data: 208 numbers (29 non-integer, none negative, no exponent, no `-0`, all already in ECMAScript `Number#toString` form). No escape of any kind, no astral character, no empty container. Every key is ASCII, and key order is schema order, not sorted.
|
||||
13. The docs disagree with each other. BDR-045 / Annex C says "existing fingerprints untouched". RUST.md:2092 and TODO:307 still say "will change once".
|
||||
14. Doc errors: RUST_ARCHITECTURE_REVIEW.md:145 says "lean document" (the fingerprint covers the full document) and cites the deleted `packages/pdf export.ts:88`. STEP6:89 says "every run changes" the fingerprint, but the pipeline is deterministic and the golden test pins it.
|
||||
15. Open: which form gets canonicalised (input file or parsed), what "compatibility" with 27c7… means concretely, who computes the fingerprint for database-served sheets (Step 13), and whether versionId or sheetId should take part in the rebase decision.
|
||||
|
||||
## 1. Producers (who computes a fingerprint)
|
||||
|
||||
| Layer | Site | What it does |
|
||||
|---|---|---|
|
||||
| TS pipeline | packages/anatomy/src/curation/pipeline.ts:158-165 `sheetRef()` | `documentSha256: sha256Hex(documentJson)`, with `sheetId: draft.sheetId` and `versionId: null` |
|
||||
| | pipeline.ts:185-186 | `documentJson = await formatJson(draft.document)`. The same text is written as `document.json` (:203) and embedded verbatim in the seed SQL (:216-221, seed.ts:21-25 and :73-80 dollar-quoted `::jsonb`) |
|
||||
| | pipeline.ts:205-211 | writes `sheet-ref.json` `{sheet_id, version_id, document_sha256, schema_version, species_id}` |
|
||||
| | pipeline.ts:187-194 → curation/migration-map.ts:24 | the same `ref` object becomes `report-migration-map.json` `.sheet` |
|
||||
| | packages/anatomy/src/legacy/ids.ts:15-17 | `sha256Hex = createHash('sha256').update(data).digest('hex')` (node:crypto, UTF-8 of the string) |
|
||||
| | packages/anatomy/src/format.ts:9-26 | `formatJson(v) = prettier.format(JSON.stringify(v,null,2), {parser:'json', printWidth:80, …})`. Prettier `^3.4.2` (packages/anatomy/package.json:31), locked 3.8.3 (pnpm-lock.yaml:1894) |
|
||||
| TS legacy importer | legacy/artifacts.ts:17 | writes legacy-import/document.json through `formatJson`. It produces no fingerprint and no sheet-ref |
|
||||
| TS app | none | apps/app/vite/sheet-bundle.ts:44-75 `JSON.parse`s document.json and re-serialises the payload (`JSON.stringify(payload)`, :99). The browser never holds the hashed bytes. provider.ts:46-57 copies `ref` |
|
||||
| TS packages | none | packages/domain, packages/supabase and packages/core-wasm/src have no document hash |
|
||||
| Rust | none | crates/core: `sha2` is a `[dev-dependencies]` entry (Cargo.toml:41-44, oracle digests). crates/worker: sha2 hashes WebP output only (job.rs:8, :210, :264). crates/wasm: nothing |
|
||||
| SQL | none | no column, CHECK, trigger or RPC hashes `content`. sha256 columns exist only for assets: `anatomical_sheet_assets.sha256` (20261008110000_anatomical_sheets.sql:154-164) and `public.assets.sha256` (20261013100000_files_foundation.sql:74-75) |
|
||||
| Scripts | none | scripts/browser/lib/core-module.mjs:45 and scripts/r4-j2-kit.mjs:51 pass `sheet-ref.json` through. wasm-stamp.mjs and r4-afm-data.mjs hash other inputs |
|
||||
|
||||
On the SQL side, `anatomical_sheet_versions.content` is `jsonb` (anatomical_sheets.sql:105-108, 2 MiB cap). Its schema version is pinned by CHECK (`document_schema_version = 2`, 20261010100000_sheet_document_schema_v2.sql:11-12). Measured on PGlite 0.3.16 (PostgreSQL 17.5, in-memory), one run:
|
||||
|
||||
```
|
||||
jsonb_text_sha: eaf891f46d3149ecc8734c51ccca57a5459c9c1e5d81776445b25bba351d5d54 (file: 27c7383c…)
|
||||
jsonb_text_bytes: 184105 stable re-cast: true
|
||||
head: {"name": {"en": "Horse", "fr": "Cheval"}, "views": [{"id": "main", "kind": "main", ...
|
||||
numeric samples [1.50, 1e2, -0, 743.0451049804688, 1.0] -> [1.50, 100, 0, 743.0451049804688, 1.0]
|
||||
key order {"fr","en","schema_version","id"} -> {"en","fr","id","schema_version"} duplicate key -> last wins
|
||||
```
|
||||
|
||||
jsonb::text orders keys by length then bytes and uses `", "` / `": "` separators. It keeps numeric scale (`1.50`, `1.0`), unlike JS and JCS. `1e2` becomes `100` and `-0` becomes `0`.
|
||||
|
||||
## 2. Consumers (read / compare)
|
||||
|
||||
| Site | file:line | Behaviour |
|
||||
|---|---|---|
|
||||
| Contract `SheetRef` | crates/core/src/domain/report/v3.rs:51-71 | `{sheetId, versionId\|null, documentSha256}`. `deserialize_with = "sha256"` → `js::is_sha256_hex` (js.rs:186-188, `^[0-9a-f]{64}$`). No `deny_unknown_fields` |
|
||||
| Contract `SheetSnapshotRef` | archive/snapshot.rs:39-55 | adds schemaVersion and speciesId, `deny_unknown_fields`, same sha256 check |
|
||||
| Bundle load | domain/sheet/loaded.rs:29-36, :104-111 | `BundledRef.document_sha256: String` copied into `LoadedSheet.reference` without format check or recomputation |
|
||||
| Index to TS | sheet/index_data.rs:25, :112 | exposes `ref` (`SheetIndexData.ref`) → apps/app/src/services/sheetIndex.ts:28, :63-66 |
|
||||
| New report | ops.rs:53-60, :117-125 → editor/report_ops.rs:56 | stamps the `SheetRef` it receives (validated). App: NewReportRoute.tsx:77 `sheet.sheet.ref` (from provider.ts `availabilityOf`, raw bundle) → reportStore.ts:354-366 → reportOps.ts:21-29 `report.new` |
|
||||
| v2 migration | domain/report/read.rs:91 | `sheet: map.sheet.clone()`: a migrated v2 report takes the migration map's ref |
|
||||
| Rebase | domain/report/rebase.rs:40-47 | only `report.sheet.document_sha256 == sheet.reference.document_sha256`: equal → report returned untouched, `rebased:false` |
|
||||
| | rebase.rs:48-62 | otherwise `rebased.sheet = sheet.reference.clone()` (sheetId, versionId and sha all replaced), observations rebased, `reparse` through the v3 schema |
|
||||
| Envelope invariant | domain/archive/read.rs:94-119 | `snapshot.ref.sheet_id == report.sheet.sheet_id && snapshot.ref.document_sha256 == report.sheet.document_sha256`. If false: `SnapshotRejection::SheetMismatch`, envelope dropped, report kept (:86-89) |
|
||||
| Session entries | domain/session.rs:76-101 (draft), :111-135 (import) | read, then rebase with the archived snapshot. A rebase failure becomes `RestoreFailure::Rebase` → `rebase_failed` (ops.rs:327) or `ImportFailure::Invalid` → `invalid` (ops.rs:351-357) |
|
||||
| Snapshot build | archive/snapshot_rules.rs:54-66 | copies the loaded ref into `sheetSnapshot.ref`. The snapshot `document` is the lean document, so the ref names the full document, not the archived one |
|
||||
| Export check | pdf/write/input.rs:204-210 | `sheet_id !=` or `document_sha256 !=` → `ExportErrorKind::SheetMismatch` (`sheet_mismatch`, :50, :66) |
|
||||
| Export envelope | pdf/write/export.rs:26-37 | writes `report` (with its `sheet`) and `sheetSnapshot` (with `ref`) as given, `JSON.stringify(…,null,2)` form. `export_from_envelope` (:116-137) re-exports from the envelope's own report and snapshot. The PDF layout draws no ref |
|
||||
| PDF import | pdf/import.rs:22-30, facade.rs:66-77 | byte scan → `import_archive` (same rebase) |
|
||||
| Rescue copy | facade.rs:80-100 | `restore_draft` semantics with no archive |
|
||||
| `sheet_ref` op | facade.rs:43 | echo of the contract (tests) |
|
||||
| report-compare | bin/report_compare.rs:32-66 | loads `sheet-ref.json` as-is. compare/mod.rs:133 reads the archive with no rebase. compare/lines.rs:347-367: sheetId and sha equal → `sheet … (draft, matches the bundled sheet)`, else `sheet … !! written on another sheet document (sha <12 hex>…)`. Exit code unaffected (exit 2 only on unreadable entries, run.rs:252) |
|
||||
| App, import messages | apps/app/src/app/routes/HomeRoute.tsx:41-100, :205-229 | see §4 |
|
||||
| App, draft restore | state/reportStore.ts:262-293; EditorRoute.tsx:58-67, :104-139 | see §4 |
|
||||
| App, error mapping | core/worker/outcome.ts:65-67 | `invalid` (rebase refused) → `ImportError('generic')`. EditorRoute.tsx:262: every export failure, `sheet_mismatch` included → `editor.export_failed` |
|
||||
| packages/domain (frozen; only packages/anatomy imports it, type-only for SheetRef) | report/v3.ts:31-38; rebase.ts:114-135; archive/envelope.ts:96-101; archive/snapshot.ts:24, :60-68, :262-276; report/migration-map.ts:17; report/migrate.ts:121; anatomical-sheet/sheet-index.ts:23 | same semantics as the Rust (rebase on sha only, invariant on sheetId+sha) |
|
||||
| N-1 reader | packages/core-wasm/tests/n-1/domain/** | byte-identical to packages/domain/src (`diff -r` empty). Used by pdf-export.test.ts:16, :192 and measure.test.ts:14 |
|
||||
| packages/supabase | none | publish gate anatomical-sheets.ts:74-98 checks shape and integrity only. scripts/upload-sheet-assets.ts:58-59 hashes asset bytes |
|
||||
| SQL RPCs | anatomical_sheets.sql:735-762 `create_sheet_draft` copies `content` verbatim from a non-draft version (same document, new `gen_random_uuid` id). :764-805 `publish_sheet_version` runs `sheet_document_issues` and `unknown_assets`, no hash. No reports table exists |
|
||||
|
||||
## 3. Stored values
|
||||
|
||||
Only one non-synthetic fingerprint has ever been committed. `git log` on sheet-ref.json shows one commit, 0a17e39 (2026-10-10), and document.json has not changed since that commit.
|
||||
|
||||
| Value | Where (file:line) | Identifies |
|
||||
|---|---|---|
|
||||
| `27c7383c445c73dc2001ebf4794a3ea63d416910bbbaa0d7848dd720de7c2660` | supabase/seed/anatomical-sheets/equine/draft/sheet-ref.json:4; draft/report-migration-map.json:6 | sha256 of the committed draft/document.json bytes (302,794 B, verified with `sha256sum`). sheetId `503dc301-a8b1-50d4-9e44-0919b2fc2916` = `deterministicUuid('faunosteo:sheet:equine:v1')` (verified; importer.ts:329), versionId null, curated draft schema v2 |
|
||||
| same | apps/app/tests/fixtures/report-v3.demo.json:24; r4/reports/L.json:24; r4/reports/CAP.json:24; pdf/r3/store-path.report.json:22 | Report v3 fixtures stamped on the bundled draft |
|
||||
| same | apps/app/tests/fixtures/goldens/r2/envelope.demo.fr.nologos.json:29,279; envelope.v2full…:29,160; envelope.v2unresolved…:24,763 | report.sheet + sheetSnapshot.ref of frozen envelopes |
|
||||
| same | crates/core/tests/oracles/editor.json (18 lines, 1669…2475); migration-map.draft.json:1; report-v3.demo.json:1 | R2 oracles frozen from the TS |
|
||||
| same | oracles/r4/compare/compare.json:9; oracles/r4/tolerance.json:6 | sheet pins, asserted equal to the loaded ref (pdf_compare.rs:266-270) |
|
||||
| same | oracles/r4/text.json:17807-17808 | an AFM width sample string, not a fingerprint use |
|
||||
| same | attachments of 99 committed PDFs (pdf/r3 13, r3/real 1, r4/read 21, r4/tolerance 50, step6 3, fixtures root 1, compare scenarios 10). Visible uncompressed in r4/read/scan.{crlf-unspaced-type-params,indirect-length,wrong-direct-length}.pdf | exported archives of the draft |
|
||||
| `fff…f` (64) | 1 PDF in r4/read; archive_rules.rs:777; domain archive.envelope.test.ts:189; r2-characterisation.test.ts:676; anatomy curation.migration-map.test.ts:15 | synthetic: snapshot of another sheet |
|
||||
| `bbb…b` | 1 PDF in compare scenarios; report_rules.rs:397; reportStore.test.ts:201; HomeRoute.archive.test.tsx:69; domain tests | synthetic "other sheet" |
|
||||
| `aaa…a`, `ccc…c`, `eee…e`, `000…0` | oracles/characterisation.json, oracles/verdicts.json (≈600 `a`), Rust tests (`"0".repeat(64)` in sheet_index.rs:120, editor_rules.rs:43, sheet_contract.rs:35, anatomy_canvas_op.rs:21, pdf_layout.rs:295), app tests (`'c'.repeat(64)` in heldSession/quarantine/rescueCopy, `'0'` in EditorRoute.test.tsx:223) | synthetic. Part of the `c` values are asset hashes |
|
||||
| `91e459f4c0ac8043831242c9fccde01f75f1023464db909f1c29571aa8a414c2` | not stored: sha256 of legacy-import/document.json, computed for this census | legacy baseline has no sheet-ref |
|
||||
| `eaf891f4…5d54` | not stored: sha256 of the draft's `jsonb::text` (PGlite) | §1 |
|
||||
|
||||
The other 64-hex values in the repo (89 files) are unrelated to the fingerprint: asset hashes (draft horse.svg `0ea5202f…` in draft/assets.json and equine_draft.sql:18, two legacy-import asset hashes), PDF and PNG digests in oracles, R4 golden digests, wasm and build digests under scripts/browser/results, and AFM data headers.
|
||||
|
||||
The PDF scan was a regex over raw bytes plus zlib inflation of every `stream`. 116 PDFs carry no fingerprint: r4/read hostile and v2 cases (110), 3 compare scenarios, 2 legacy v2 PDFs and 1 tolerance case.
|
||||
|
||||
## 4. Semantics on mismatch
|
||||
|
||||
Import (PDF or attachment JSON), session.rs:111-135, in this order.
|
||||
|
||||
1. The report is read first. A v3 report is kept as written. A v2 report is migrated and stamped with `map.sheet` (read.rs:91), which is 27c7…/503dc301 today.
|
||||
2. Envelope v1: the snapshot passes the strict gate, then the invariant. The snapshot ref and the report ref of the same file must agree on sheetId and fingerprint (read.rs:116-119). If they don't, the envelope is dropped (`snapshotRejected: 'sheet_mismatch'`) and the report is kept. Message: `home.import.snapshot_rejected` (fr.ts:46-47, en.ts:44-45) with `home.import.snapshot_reason.sheet_mismatch` ("fiche différente de celle du bilan", fr.ts:53-54; "sheet differs from the report's", en.ts:51-52). Headline `home.import.degraded`.
|
||||
3. Rebase onto the loaded sheet (rebase.rs). If the fingerprint is equal, nothing changes. If it differs, `report.sheet` becomes the loaded ref.
|
||||
- A zone becomes `unresolved: true` when its `annotationId` or `viewId` is unknown (:85-101). Its label snapshots are filled from the archived snapshot only when absent (`??=`, :103-112).
|
||||
- A finding becomes `unresolved: true` when `(findingZoneId, path)` is unknown (:137-153).
|
||||
- Already-flagged items are left alone and not re-counted. Ids, paths, severity, comments and labels are all kept. "unresolved" means "the loaded sheet cannot place this item", not "deleted".
|
||||
- The app shows `home.import.rebased` with the count: "Bilan ouvert sur la fiche actuelle, différente de celle de l'archive. Éléments non retrouvés : {count} (conservés avec leurs libellés)." (fr.ts:55-56, en.ts:53-54).
|
||||
- The headline is `home.import.complete` (fr.ts:39, en.ts:37) when 0 items are lost, else `home.import.degraded` (fr.ts:40-41, en.ts:38-39) (HomeRoute.tsx:70-84). An accepted envelope adds `home.import.archive_ok` (fr.ts:44-45).
|
||||
- The imported envelope is kept unchanged (`osteo:archive`) with its original refs. Only the working report is rebased.
|
||||
4. A sheetId that differs while the fingerprint is equal triggers no rebase, because rebase.rs:40 ignores sheetId. The foreign sheetId stays, and a later export is refused (`sheet_mismatch`, input.rs:206) as the generic `editor.export_failed` (EditorRoute.tsx:262). A different sheetId with a different fingerprint is rebased and the sheetId replaced.
|
||||
|
||||
Draft restore (reportStore.ts:262-293 → core `draft.restore`) uses the same rebase. When the draft was rebased, items were flagged and the practitioner has not confirmed replacement, the raw draft goes to `osteo:quarantine:draft-pre-rebase`, or is held if that slot cannot be written. The editor then shows `editor.draft_rebased` (fr.ts:89-90; en.ts:83-84). A rescue copy (HomeRoute.tsx:222-229) gets the same message. If the rebased report fails the schema (a defect of the loaded ref, such as a malformed bundle sha), the result is `rebase_failed`: the draft stays where it is and the app answers `core_unavailable` (reportStore.ts:266-271).
|
||||
|
||||
Export: the snapshot ref and the report ref must agree on sheetId and fingerprint. After any rebase both equal the loaded ref. The envelope carries the fingerprint twice (`report.sheet` and `sheetSnapshot.ref`, docs/PDF-FORMAT.md:60-70, :121-122), and the visible PDF text carries none. report-compare prints a header line only (§2) and never rebases.
|
||||
|
||||
## 5. Document data shape (script output)
|
||||
|
||||
```
|
||||
== draft/document.json bytes 302794 sha256 27c7383c…2660 BOM False CR False tabs False trailing LF True lines 8284
|
||||
top-level keys: schema_version species_id name description scope_disclaimer attribution vertebral_config references categories views annotations finding_zones
|
||||
numbers: {'int': 179, 'float': 29} (no negative, no exponent, no -0) float paths: geometry.points 28, geometry.cx 1
|
||||
float examples 743.0451049804688 1418.5 556.2 … max 16 significant digits; ending .0: 0; int range 2..1780
|
||||
raw escapes in file: \u 0 \n 0 \" 0 \\ 0 \/ 0 \t 0
|
||||
non-ASCII code points 1831, distinct 18: é 1354, è 139, à 106, ï 58, ° 45, ê 37, ç 22, â 19, « 15, » 15, – U+2013 6, ô 5 …; astral 0
|
||||
strings 3724, non-ASCII 788, control chars 0; max string length 521 code points ($.scope_disclaimer.fr)
|
||||
non-ASCII keys 0; duplicate-key objects 0; objects 1951, arrays 370
|
||||
objects with >=2 keys: sorted (UTF-16) 560, not sorted 1390 (e.g. $ top level; every {fr, en}); single-key 1
|
||||
empty objects 0; empty arrays 0; nulls 66 (doi 27, vertebral_region_code 22, url 13, license, view_box, svg_asset_id, name); booleans true 55, false 0
|
||||
max depth 10 at $.finding_zones[0].categories[0].children[0].children[0].name.fr
|
||||
== legacy-import/document.json bytes 224109 sha256 91e459f4…14c2 same top-level keys, LF, no BOM
|
||||
numbers int 159, float 36 (max 17 significant digits, 1424.2662353515625); no negative/exponent/-0; escapes 0; astral 0
|
||||
non-ASCII 1164 code points, 11 distinct (adds — U+2014 ×4, É ×2); strings 2889 (650 non-ASCII); max length 253
|
||||
objects 1808 (586 sorted, 1158 not, 64 single-key); empty arrays 1 ($.references); nulls 30; max depth 10
|
||||
```
|
||||
|
||||
Node checks:
|
||||
|
||||
```
|
||||
draft: file === JSON.stringify(v,null,2)+LF: false; first diff line 17 '"authors": ["Faunosteo"],' (Prettier collapses short arrays, e.g. "[1655, 545]"); 8284 vs 8695 lines
|
||||
draft: number tokens 208, not in ECMAScript Number#toString form 0, unsafe integers 0 (legacy: 195 / 0 / 0)
|
||||
draft: compact UTF-8 bytes 174,974; JSON.stringify(raw) 173,136 UTF-16 units vs JSON.stringify(Zod.parse(raw)) 245,636 (first diff: count_range:null default added)
|
||||
legacy: same pattern, 110,854 vs 190,448
|
||||
```
|
||||
|
||||
What these checks show:
|
||||
|
||||
- The hashed bytes are the input form: defaults are omitted (no empty arrays, no `count_range`). The parsed form adds those defaults, so its bytes differ.
|
||||
- Key order is construction or schema order, not sorted.
|
||||
- Numbers already print as ECMAScript does, and no string needs escaping.
|
||||
- The layout comes from Prettier's line fitting (printWidth 80), which is not `JSON.stringify` layout.
|
||||
|
||||
## 6. Documentation statements
|
||||
|
||||
| file:line | Statement (quoted or condensed) |
|
||||
|---|---|
|
||||
| docs/development/RUST_ARCHITECTURE_REVIEW.md:55-60 | "`documentSha256` is today the sha256 of the Prettier-formatted document text. It is computed once by the pipeline and only compared at runtime; a different hash sends an old archive through the rebase path … The study therefore does not chain the pipeline to Prettier (D-R5)" |
|
||||
| …REVIEW.md:145 | "sha256 of the Prettier-formatted lean document text … (`rebase.ts:119`, `envelope.ts:100`, pdf `export.ts:88`)". "lean" is inaccurate (the full curated document is hashed), and `packages/pdf` was deleted at R4.8 |
|
||||
| …REVIEW.md:584 | `serde_jcs` listed under "Rejected" (Part V) |
|
||||
| …REVIEW.md:636, :655 | R5 row: "seed golden regenerated once (decided)". Compatibility: "regenerated seed (new `documentSha256`) … rebase tests both ways on the two seeds" |
|
||||
| …REVIEW.md:837 (Part X), :873 (Annex A) | D-R5 rev 2: "`documentSha256` = sha256 of the bytes the Rust pipeline emits; generated dirs leave Prettier's scope; the one-time hash change makes older archives … rebase … 0 unresolved items". Alternatives named: keep Prettier, or canonical JSON (RFC 8785, `serde_jcs`) |
|
||||
| …REVIEW.md:974 (Annex C, owner amendment) | "Preference for a deterministic fingerprint over a canonical JSON representation independent of the formatter … study RFC 8785 (JCS) or a strictly defined alternative … Explicit compatibility with the fingerprints and archives already produced. Existing fingerprints are not changed now: the decision is prepared for R5" |
|
||||
| docs/development/RUST.md:2092-2094 | "`documentSha256` will change once (D-R5) and every older archive will rebase on import" |
|
||||
| RUST.md:1902 | report-compare loads `sheet-ref.json` |
|
||||
| docs/PDF-FORMAT.md:66, :106 | snapshot `ref.documentSha256`; "sha256 of the full catalog document (provenance pointer)" |
|
||||
| PDF-FORMAT.md:121-122 | "Invariant: `report.sheet.documentSha256 === sheetSnapshot.ref.documentSha256` (and the sheet ids match). The export refuses to write a file that breaks it." |
|
||||
| PDF-FORMAT.md:175, :195-200 | closed reason `sheet_mismatch`; "When the archive's sheet differs from the loaded one, the report is rebased …" |
|
||||
| docs/development/ANATOMICAL_SHEET_MODEL.md:317 | `sheet-ref.json` "(sheet id, null version id, sha256 of the document)" |
|
||||
| docs/development/STEP5_MIGRATION_ANALYSIS.md:74, :94-95, :137 | "`documentSha256` the hash of `draft/document.json`" |
|
||||
| docs/development/STEP6_PDF_ARCHIVE_ANALYSIS.md:88-89 | pre-Step-6 state: "Nothing compares `report.sheet` with the loaded sheet … Every `pnpm import:equine` run changes the draft's `documentSha256`" (the second sentence conflicts with the deterministic golden) |
|
||||
| STEP6:124, :160-161, :290-293, :775-776 | ref = "full catalog document's hash, kept as a pointer, not as a hash of the snapshot"; invariant; rebase trigger; export `sheet_mismatch` |
|
||||
| .claude/memory/decisions.md:327 (BDR-038) | Report v3 `sheet {sheetId, versionId\|null, documentSha256}` |
|
||||
| decisions.md:336 (BDR-039) | the snapshot hash check was rejected as "self-referential; sha copied … as provenance pointer" (about assets) |
|
||||
| decisions.md:361 (BDR-043) | alternative rejected: "Prettier-chained sheet identity (hash only compared at runtime → one-time rebase accepted)" |
|
||||
| decisions.md:371 (BDR-045) | "D-R5 fingerprint → canonical JSON (RFC 8785 JCS or strictly defined alternative) independent of formatter, explicit compat with old fingerprints/archives, existing fingerprints untouched, decision prepared for R5" |
|
||||
| .claude/memory/learnings.md:363 (LRN-048) | "`documentSha256` = sha256 of Prettier-formatted text (only compared at runtime → mismatch = rebase, not loss)" |
|
||||
| .claude/tasks/TODO.md:307 | "seed golden regenerated once (new `documentSha256`, D-R5)" |
|
||||
| CLAUDE.md:277 | `draft/` holds `sheet-ref` |
|
||||
|
||||
## 7. Risks and open questions (analyzer)
|
||||
|
||||
Risks:
|
||||
|
||||
- Any change to the emitted bytes with identical content (Rust emitter, Prettier bump, printWidth) changes the value. Every archive and draft carrying 27c7… then rebases with a visible message, normally with 0 unresolved items. That covers 99 committed PDFs, 8 JSON fixtures and 6 Rust oracle files, which would need decisions.
|
||||
- map.sheet and sheet-ref.json are two copies with no runtime equality check. If they diverge, every v2 import is stamped with one and rebased onto the other (low: one pipeline value plus the golden).
|
||||
- The bundle fingerprint is not format-checked at load. A malformed value surfaces later as `rebase_failed`, generic `invalid` or a refused `report.new` (low).
|
||||
- versionId is never compared, and create_sheet_draft copies content verbatim. Once sheets come from the database (Step 13), versions with identical content would share a fingerprint and skip the rebase (medium).
|
||||
- If a fingerprint has to be computed from database `content`, the jsonb text differs from the file (key order, numeric scale). The current value cannot be reproduced from jsonb (medium at Step 13).
|
||||
- Stale or contradictory docs (REVIEW:145 "lean" and deleted citation, RUST.md:2092 and TODO:307 vs BDR-045, STEP6:89) could steer the R5 plan.
|
||||
|
||||
Open questions:
|
||||
|
||||
- Which statement is in force for R5? BDR-045 / Annex C ("existing fingerprints untouched, explicit compatibility") or RUST.md:2092 / TODO:307 ("will change once")?
|
||||
- What does "compatibility with fingerprints already produced" mean concretely: keep 27c7… for the current draft, an old→new equivalence, or a one-time rebase with a message?
|
||||
- Which form gets canonicalised: the input file form (defaults omitted) or the parsed form (defaults filled)? They differ by about 72k UTF-16 units.
|
||||
- For database-served sheets, who computes and stores the fingerprint (pipeline or operator, Rust service, SQL)? No column exists.
|
||||
- Should sheetId or versionId join the rebase decision (today it uses the fingerprint only), given that the export and the envelope check sheetId too?
|
||||
@@ -0,0 +1,297 @@
|
||||
# R5 inventory: the TypeScript anatomy pipeline (`packages/anatomy`)
|
||||
|
||||
Root: `/home/bchanot/Documents/faunosteo` (paths repo-relative). Branch `feature/rust-r5-pipeline-core` cut at 9c0117a. Read-only analysis by an analyzer agent (one in-memory probe of the pipeline functions for §4.3), saved by the orchestrator. Runtime: Node 24.21.0, ICU 78.3, Unicode 17.0.
|
||||
|
||||
## Summary
|
||||
|
||||
1. `packages/anatomy` has 43 source files and 4,897 lines: legacy 1,842, curation 2,329, render 553, cli 119, root 54. Tests are 2,553 lines; the upload script is 99.
|
||||
2. The runtime libraries are Prettier 3.8.3, @xmldom/xmldom 0.9.12 and Zod 3.25.76, plus node:crypto, node:fs and node:util. Prettier and xmldom are declared as devDependencies but used at runtime.
|
||||
3. There are 18 generated files. Ten go through Prettier: 7 JSON and 3 Markdown. Five go through the xmldom XMLSerializer (horse.svg and 4 review SVGs). One is a raw SQL template, and two are byte copies of the legacy SVGs.
|
||||
4. `documentSha256` is the sha256 of the UTF-8 bytes of the Prettier-formatted `draft/document.json`, trailing newline included (`pipeline.ts:159-165,185`, `ids.ts:15-17`). Checked equal to the sha256 of the committed file. `equine_draft.sql` does not carry it; it embeds the document text instead.
|
||||
5. That hash is pinned in 18 tracked files (seed, crate oracles, app goldens, R4 PDFs). Any byte change in the draft document cascades to all of them.
|
||||
6. Measured on today's data, Prettier changes only Markdown table padding (134, 863 and 52 lines in the three reports). For JSON it changes layout only: short arrays collapsed onto one line and a trailing newline. The data is identical after re-parsing.
|
||||
7. The JSON key order is JS insertion order of the working document, which is never re-parsed by Zod. For example, `review_status` is appended last on annotations by the B/C review rule.
|
||||
8. UUIDs are built from the first 16 bytes of sha256(name), with the version nibble set to 5 and variant bits 10. Names are `faunosteo:asset:<sha>` and `faunosteo:sheet:equine:v1` (gives 503dc301…). Both checked.
|
||||
9. JS-specific hazards: number printing, `Math.round`, `Number('') = 0`; Unicode `\s`, `split(':', 2)`, NFD plus `toLowerCase`; UTF-16 sorting and slicing; xmldom's `/>`, escaping and namespace rules; Prettier's printNumber and its string-width function. None uses dates or randomness, and nothing calls localeCompare.
|
||||
10. Tests: 111 declarations, 131 cases once `it.each` rows are expanded. The golden test compares the exact text of every generated file; no comparison works on parsed trees.
|
||||
11. Five tests depend on the TS `render/` code. Three use it directly (render.layer, render.compose, r3.layer-oracles). Two use it through `renderDraftArtifacts` → `app-review` → `composeViewSvg` (seed.golden, curation.pipeline).
|
||||
12. 22 source files import `@faunosteo/domain`. Only 5 runtime symbols are used: the document schema, `validateSheetDocument`, the schema version, the stable-id pattern and `ReferenceSchema`. All already have Rust counterparts from R2.
|
||||
13. `@faunosteo/anatomy` has no code consumer outside itself, only the root `import:equine` script. `@faunosteo/domain` is imported only by `packages/anatomy`.
|
||||
14. Deleting the packages breaks things beyond imports: Rust tests read `packages/domain/tests/fixtures/report-v2.{full,minimal}.json`, and `apps/app/tests/unit/r4.census.test.ts:80-90` asserts xmldom sits under `packages/anatomy`.
|
||||
15. Open decision: the D-R5 hash rule as written (hash the bytes Rust emits) versus the owner's amendment in BDR-045 (a canonical-JSON fingerprint, decision "prepared for R5").
|
||||
|
||||
## 1. Package and tooling
|
||||
|
||||
- `packages/anatomy/package.json`: exports `.` → `src/index.ts` and `./render` → `src/render/index.ts` (:10-13); script `import:equine` = `tsx src/cli/import-equine.ts` (:20); dependencies `@faunosteo/domain` workspace and `zod` ^3.24.1 (:22-25); `@xmldom/xmldom` ^0.9.8 and `prettier` ^3.4.2 are devDependencies (:28, :31), yet `src/format.ts` and `src/legacy/svg.ts` import them at runtime, as do `curation/{assets,overlay,app-review}.ts`.
|
||||
- Lockfile (importer `packages/anatomy`): zod 3.25.76, @xmldom/xmldom 0.9.12, prettier 3.8.3, tsx 4.23.15, typescript 5.9.3, vitest 3.2.7, @types/node 20.19.41. Each of prettier and xmldom has a single version in the whole lockfile.
|
||||
- `vitest.config.ts`: environment node, globals, include `tests/**/*.test.ts`. `tsconfig.json`: ES2022, bundler resolution, strict, noEmit, covers src and tests. `eslint.config.js`: typescript-eslint recommended.
|
||||
- Root wiring: `package.json:28` `import:equine` = `pnpm --filter @faunosteo/anatomy import:equine`; `package.json:36` `assets:upload` → `packages/supabase/package.json:21` `node scripts/upload-sheet-assets.ts` (Node's native type stripping, not tsx).
|
||||
- Prettier options (`src/format.ts:9-19`) are identical to the root `.prettierrc.json`. Root `format` = `prettier --write .` (`package.json:24`). `.prettierignore` does not exclude `supabase/seed/**`, so a root format run also rewrites the generated `.json` and `.md` files.
|
||||
|
||||
## 2. Lines of code (wc -l)
|
||||
|
||||
| Area | Files | Lines |
|
||||
|---|---|---|
|
||||
| src/legacy | 15 | 1,842 |
|
||||
| src/curation | 19 | 2,329 |
|
||||
| src/render | 5 | 553 |
|
||||
| src/{index,format,markdown}.ts | 3 | 54 |
|
||||
| src/cli/import-equine.ts | 1 | 119 |
|
||||
| src total | 43 | 4,897 |
|
||||
| tests (16 suites + helpers.ts + fixtures/mini-sheet.ts) | 18 | 2,553 |
|
||||
| packages/supabase/scripts/upload-sheet-assets.ts | 1 | 99 |
|
||||
|
||||
## 3. Modules (path (lines): exports with line, inputs → outputs, external deps)
|
||||
|
||||
### 3.1 Shared and entry
|
||||
|
||||
| Module | Exports | In → Out | Deps |
|
||||
|---|---|---|---|
|
||||
| index.ts (3) | re-exports legacy, curation, render | | |
|
||||
| format.ts (30) | formatJson:21, formatMarkdown:28 | value → Prettier(`JSON.stringify(v,null,2)`, parser json); text → Prettier markdown | prettier |
|
||||
| markdown.ts (21) | cell:6, table:11, code:19 | null → `-`; escapes `\|` and LF only; tables unpadded (Prettier pads them) | |
|
||||
| cli/import-equine.ts (119) | none (main) | repo → writes legacy-import/ and draft/; prints 2 summary lines | node:fs/promises, node:path, node:url |
|
||||
|
||||
CLI behaviour: REPO_ROOT is 4 levels up (:32). Strings are written with `writeFile(…,'utf8')`, otherwise `copyFile` (:37-44). `removeStale` deletes every file in the two folders that the run did not produce (:47-55). The legacy folder gets `renderArtifacts` output plus the two SVG copies (:104-107). The draft folder gets `renderDraftArtifacts` output plus `assets/<file>` = cleanedSvg (:108-111).
|
||||
|
||||
### 3.2 legacy/ (importer, Step 4b)
|
||||
|
||||
| Module | Exports | In → Out | Deps |
|
||||
|---|---|---|---|
|
||||
| sources.ts (54) | EQUINE_SOURCE_PATHS:10, EQUINE_{LEGACY_IMPORT,CURATION,DRAFT}_DIR:20,23,26, readEquineSources:32 | root → SVG bytes, catalog text, vertebrae text | node:fs, node:path |
|
||||
| types.ts (115) | Anomaly, AssetRecord, LegacyIdMap, ImportResult, … | types | domain (types) |
|
||||
| ids.ts (54) | sha256Hex:15, deterministicUuid:19, assetIdFromSha256:33, sheetIdFor:37, slugify:42, isStableId:52 | | node:crypto, domain STABLE_ID_PATTERN |
|
||||
| svg.ts (116) | parseAnnotatedSvg:94 | SVG text → viewBox, one record per data-anno-id (first element wins, elementCount), anomalies | xmldom DOMParser |
|
||||
| vertebrae.ts (51) | parseVertebralFormula:16, vertebralRegions:45 | vertebrae.ts text, parsed with regexes (not imported) → counts, "varies A–B" | |
|
||||
| regions.ts (46) | inferRegionCode:36 | FR then EN label → code/root; root table :8-14 | |
|
||||
| categories.ts (98) | normaliseCategoryId:37, mapCategories:82 | annotations → categories in first-seen order; 2 slug fixes :14-17; authored labels :20-29 | |
|
||||
| catalog.ts (249) | parseLegacyCatalog:73, categoryKind:81, countTree:169, assertNothingDropped:206, mapCatalog:233 | pathologies.json → lean finding zones (optional keys only when present); throws if output count ≠ input | |
|
||||
| catalog-checks.ts (102) | catalogAnomalies:96 | foreign prefix, anamnesis variants, cross-zone duplicates | |
|
||||
| annotations.ts (207) | zoneNameKey:37, mapAnnotation:184 | annotation + map `fr\nen` → zone → input annotation, row, region, anomalies | |
|
||||
| authored-labels.ts (41) | authoredLabelAnomalies:30 | document → label.authored anomalies | |
|
||||
| importer.ts (343) | importEquineLegacy:313, MAIN/HEAD_VIEW_ID:50-51 | EquineLegacyInput → ImportResult | domain schema and validate |
|
||||
| report.ts (325) | renderImportReport:310 | ImportResult → French markdown, 9 sections, trailing `\n` | |
|
||||
| artifacts.ts (22) | renderArtifacts:13 | ImportResult → 4 formatted texts | format.ts |
|
||||
|
||||
Facts about the importer: the document written is the unparsed input form (`importer.ts:322`); the Zod parse (:317) only feeds `validateSheetDocument` (:341). Two guards throw: annotations read ≠ written (:223-228), and inline sources present (:234-239). The trigger id is `${opensDetail}-trigger` (`annotations.ts:117`); an observable annotation's id is the zone id, or `slugify(name.en)` as a fallback (:171). The slugify fallbacks never fire on today's data: `legacy-id-map.json` has `finding_nodes: {}`, and all 26 observables resolve to a zone.
|
||||
|
||||
### 3.3 curation/ (Steps 4c and 5)
|
||||
|
||||
| Module | Exports | In → Out | Deps |
|
||||
|---|---|---|---|
|
||||
| schema.ts (172) | OpSchema:101, ChangeSchema:110, ChangesetSchema:132, ReferencesFileSchema:141, needsConfirmation:164, changeSummary:168, ASSET_CHANGE_ID:29 | strict Zod; priority default `medium` (:119); level C needs a question (:123-126); duplicate ids refused (:128-148) | zod, domain ReferenceSchema |
|
||||
| sources.ts (38) | readEquineCuration:26 | curation/{references,changes}.json → parsed | node:fs |
|
||||
| targets.ts (195) | parseTarget:58, formatTarget:73, locate:131, zoneFindings:151, allFindings:158, findingScope:167, childList:185, childPrefix:192 | target grammar (:2-5) over plain JSON | |
|
||||
| fields.ts (80) | SETTABLE table (:9-43), splitField:45, readField:62, writeField:72 | dot paths; missing intermediate objects created | |
|
||||
| ops.ts (225) | applyOp:209, assertDeclared:184 | set, rename, add, remove, move, cite → in-place edits plus records | node:util isDeepStrictEqual |
|
||||
| rename.ts (57) | propagateRename:54 | link table (:14-23) and citations | |
|
||||
| types.ts (49) | ChangeRecord, record:35 | `structuredClone` of before/after; undefined → null | |
|
||||
| apply.ts (134) | applyChangeset:122, curatedIssues:103 | baseline + references + changeset → document + records; B/C changes → `to_confirm` (:46-73) | domain schema and validate |
|
||||
| assets.ts (59) | stripLegacyAnnotations:24, cleanedAsset:38, parseViewBox:49 | removes every `[data-anno-id]` element, and the root `style` when it equals the legacy value exactly (:17, :31-33) | xmldom parse and serialise |
|
||||
| migration-map.ts (178) | buildReportMigrationMap:127 | baseline, final, legacyIdMap, records, sheet ref → map + stats | |
|
||||
| stats.ts (133) | documentStats:77, statRows:113 | document → counts and French rows | |
|
||||
| changelog.ts (132) | renderChangelog:119, compactValue:29, recordSubject:38 | records + stats → markdown | |
|
||||
| review-report.ts (206) | renderReviewReport:201 | 10 fixed sections | |
|
||||
| labels.ts (29) | KIND/LEVEL/PRIORITY_LABELS, LEVELS | French labels | |
|
||||
| overlay.ts (214) | overlayMarkers:49, renderReviewOverlay:185 | cleaned SVG + before/after documents + view → review SVG | xmldom, node:util |
|
||||
| app-review.ts (26) | renderAppReviewDrawing:20 | cleaned SVG + document + view → `composeViewSvg` → text | xmldom, render/compose |
|
||||
| seed.ts (125) | renderDraftSeedSql:114, DOCUMENT_DOLLAR_TAG:13, DRAFT_CHANGE_NOTE:19 | sheetId, assets, documentJson → SQL | |
|
||||
| pipeline.ts (224) | buildEquineDraft:112, renderDraftArtifacts:181, file-name constants (:41, :155-156) | see §4 | |
|
||||
| index.ts (53) | re-exports | | |
|
||||
|
||||
Facts about the curation pipeline: CHG-000 (the asset step: 3 `set` ops, `pipeline.ts:68-104`) is prepended, and an authored CHG-000 is refused (:106-110). The references file replaces the document registry before any change (`apply.ts:128`). `curatedIssues` runs twice (`apply.ts:132`, `pipeline.ts:135`). The real changeset has 45 changes, CHG-001 to CHG-045: 14 A, 22 B, 9 C; 11 carry a question; 34 set an explicit priority. It holds 687 ops: 587 set, 45 cite, 33 rename, 18 add, 4 remove, 0 move. It cites 48 references. `move` is exercised only by tests.
|
||||
|
||||
### 3.4 render/ (frozen TS renderer; R3 counterpart in `crates/core/src/anatomy/{layer,compose}.rs`)
|
||||
|
||||
| Module | Exports | Role |
|
||||
|---|---|---|
|
||||
| layer.ts (261) | renderAnnotationLayer:232, styleForView:75, viewBoxOf:258, MAIN/DETAIL_VIEW_STYLE:49,53, FALLBACK_COLOR:59 | document + view (+ states) → LayerElement tree |
|
||||
| dom.ts (38) | createLayerElement:24, SVG_NS:9 | tree → DOM via createElementNS/setAttribute |
|
||||
| serialize.ts (25) | serializeLayer:13 | escapes `& < > "` only; called only by tests |
|
||||
| compose.ts (196) | composeViewSvg:54, highlightLayer:82, HALO_FILTER_ID:20 | sets viewBox, removes width/height, appends the layer last; the halo `<defs>` is created as the first child |
|
||||
| index.ts (33) | re-exports | |
|
||||
|
||||
The pipeline's only consumer of `render/` is `curation/app-review.ts:3`.
|
||||
|
||||
## 4. Generated artifacts
|
||||
|
||||
### 4.1 Producers
|
||||
|
||||
Text is written by `import-equine.ts:41` (utf8).
|
||||
|
||||
| File | Bytes | Producer | Inputs | Formatting |
|
||||
|---|---|---|---|---|
|
||||
| legacy-import/document.json | 224,109 | artifacts.ts:17 | result.document | JSON.stringify(v,null,2) → Prettier json |
|
||||
| legacy-import/legacy-id-map.json | 1,323 | artifacts.ts:18 | result.legacyIdMap | same |
|
||||
| legacy-import/assets.json | 408 | artifacts.ts:19 | result.assets | same |
|
||||
| legacy-import/IMPORT-REPORT.md | 22,402 | artifacts.ts:20 → report.ts:310 | ImportResult | Prettier markdown |
|
||||
| legacy-import/assets/Horse.svg, Horse_head.svg | 145,770 / 139,079 | import-equine.ts:63-74 | legacy-sources | copyFile (byte copy) |
|
||||
| draft/document.json | 302,794 | pipeline.ts:185, :203 | draft.document | formatJson |
|
||||
| draft/assets.json | 203 | pipeline.ts:204 | draft.assets | formatJson |
|
||||
| draft/sheet-ref.json | 217 | pipeline.ts:205-211 | ref, schema_version, species_id | formatJson |
|
||||
| draft/report-migration-map.json | 85,627 | pipeline.ts:187-194, :212 → migration-map.ts:127 | baseline, final, legacyIdMap, records, ref, `main` | formatJson |
|
||||
| draft/CHANGELOG.md | 1,114,533 | pipeline.ts:195-200, :215 → changelog.ts:119 | records, statsBefore/After, migration stats | Prettier markdown |
|
||||
| draft/REVIEW-REPORT.md | 108,015 | pipeline.ts:201, :216 → review-report.ts:201 | changes, records, document, statsBefore, legacy anomalies | Prettier markdown |
|
||||
| draft/review/annotations-{main,head}.svg | 141,278 / 137,275 | pipeline.ts:141-153 → overlay.ts:185 | cleanedSvg, baseline, draft, view | xmldom XMLSerializer, raw |
|
||||
| draft/review/app-{main,head}.svg | 146,232 / 139,462 | pipeline.ts:168-178 → app-review.ts:20 → compose.ts:54 | cleanedSvg, draft, view | xmldom XMLSerializer, raw |
|
||||
| draft/equine_draft.sql | 607,446 | pipeline.ts:217-222 → seed.ts:114 | sheetId, assets, documentJson, change note | string template, raw, ends `\n` |
|
||||
| draft/assets/horse.svg | 132,997 | import-equine.ts:110 ← assets.ts:24 | Horse.svg, decoded as fatal UTF-8 | xmldom XMLSerializer, raw |
|
||||
|
||||
Authored inputs (never generated): `legacy-sources/`: Horse.svg 145,770, Horse_head.svg 139,079, pathologies.json 174,156, vertebrae.ts 303; `curation/`: changes.json 226,769, references.json 17,766.
|
||||
|
||||
### 4.2 JSON shapes
|
||||
|
||||
- pathologies.json: `{schema_version:"2.0", scope_disclaimer, species{3}, zones{26}}`.
|
||||
- changes.json: `{changes[45]}`. Each change has id, level, kind, title, rationale, reference_ids, [question], [priority], ops.
|
||||
- references.json: `{references[48]}`; keys id, type, citation, url, doi.
|
||||
- document.json: `{schema_version, species_id, name, description, scope_disclaimer, attribution, vertebral_config, references, categories[5], views[2], annotations, finding_zones}`; legacy-import: 27 annotations, 26 finding zones; draft: 28 annotations, 27 finding zones.
|
||||
- legacy-id-map.json: `{views{1}, annotations{27}, categories{5}, finding_nodes{0}, assets{2}}`.
|
||||
- assets.json: `[{file, sha256, byte_size, svg_asset_id}]` (2 legacy, 1 draft).
|
||||
- sheet-ref.json: `{sheet_id, version_id:null, document_sha256, schema_version:2, species_id}`, snake_case.
|
||||
- report-migration-map.json: `{schema_version:1, sheet{sheetId, versionId, documentSha256}, main_view_id, views{head:"head"}, annotations{27×{id, name{fr,en}}}, finding_paths{792, 34 null}}`; `sheet` is camelCase.
|
||||
|
||||
### 4.3 What Prettier changes (measured in memory by the probe)
|
||||
|
||||
Markdown: only table cell padding and separator widths change: IMPORT-REPORT 134 lines, CHANGELOG 863, REVIEW-REPORT 52. No other line changes, and the line counts are identical. Column widths come from Prettier's string-width function (prettier 3.8.3 `plugins/markdown.js`): an all-ASCII string counts its length; otherwise, per code point, C0/C1 controls, U+0300–036F and U+FE00–FE0F count 0, East Asian Wide/Fullwidth characters and emoji count 2, everything else counts 1.
|
||||
|
||||
JSON: the data is identical after re-parsing; only layout changes: arrays of primitives collapse onto one line when they fit in 80 columns (`"authors": ["Faunosteo"]`, `[1655, 545]`); objects stay expanded, because JSON.stringify put a newline after `{`; a trailing `\n` is added.
|
||||
|
||||
Numbers: Prettier prints JSON numbers through printNumber (lowercase, `e+21` becomes `e21`, leading zeros of the exponent and trailing zeros stripped). There are 0 exponent-form numbers in the JSON today, 29 fractional values in draft/document.json.
|
||||
|
||||
### 4.4 xmldom 0.9.12 serialisation facts
|
||||
|
||||
- The XML declaration and whitespace text nodes round-trip: horse.svg keeps 703 lines and starts with `<?xml version="1.0" encoding="UTF-8"?>`.
|
||||
- Attributes keep document order. Values are escaped on `<>&"\t\n\r` (`< > & " 	 `). Text nodes are escaped on `<&>`.
|
||||
- A childless element is self-closed as `/>` with no space. Observed: `<g id="osteo-overlays">` became `<g id="osteo-overlays"/>` once its children were stripped.
|
||||
- A namespace declaration is written only when not already in scope, so layer children carry no xmlns.
|
||||
- Nothing follows the root: the files end with `</svg>`. Nodes added for the layer and overlay are appended with no whitespace between them.
|
||||
- The inputs contain no NBSP, tab, CR or BOM, and no comment in horse.svg.
|
||||
|
||||
## 5. Identities and hashes
|
||||
|
||||
- documentSha256: `documentJson = await formatJson(draft.document)` (`pipeline.ts:185`), then `sha256Hex(documentJson)` (`pipeline.ts:159-165`); `sha256Hex` hashes the string as UTF-8 (`ids.ts:15-17`); the same string is written as draft/document.json (`pipeline.ts:203`). Verified: 27c7383c…2660 matches both the committed file and the probe's recomputation.
|
||||
- Carried in sheet-ref.json `document_sha256` (`pipeline.ts:208`) and report-migration-map.json `sheet.documentSha256` (`pipeline.ts:192` → `migration-map.ts:162`).
|
||||
- equine_draft.sql does not carry it. It embeds `$equine_doc$` + `documentJson.trimEnd()` + `$equine_doc$::jsonb` (`seed.ts:79`) twice, in the insert (:82-95) and the converge statement (:101-112). It also embeds the cleaned asset sha256 (`seed.ts:61`).
|
||||
- The hash is pinned in 18 tracked files: draft/{sheet-ref,report-migration-map}.json; crates/core/tests/oracles/{editor,migration-map.draft,report-v3.demo}.json; crates/core/tests/oracles/r4/{compare/compare,text,tolerance}.json; apps/app/tests/fixtures/report-v3.demo.json; apps/app/tests/fixtures/goldens/r2/envelope.{demo,v2full,v2unresolved}.fr.nologos.json; apps/app/tests/fixtures/pdf/r3/store-path.report.json; apps/app/tests/fixtures/pdf/r4/read/scan.{crlf-unspaced-type-params,indirect-length,wrong-direct-length}.pdf; apps/app/tests/fixtures/r4/reports/{CAP,L}.json. (The fingerprint census counts 99 PDFs once the compressed attachments are inflated.)
|
||||
- UUIDs (`ids.ts:19-31`): first 16 bytes of sha256(UTF-8 name); byte 6 = (b & 0x0f) | 0x50; byte 8 = (b & 0x3f) | 0x80; lowercase 8-4-4-4-12. Names `faunosteo:asset:<sha256 hex>` (:33-35) and `faunosteo:sheet:<species>:v<n>` (:37-39). Verified: sheet 503dc301-a8b1-50d4-9e44-0919b2fc2916, horse.svg 7f767f74-3b36-51a7-8465-52dd7e2ead4a.
|
||||
- Other hashes: legacy asset sha = raw file bytes (`importer.ts:67-75`); cleaned asset sha = `TextEncoder().encode(cleanedSvg)` (`pipeline.ts:118`); source digests (pathologies.json, vertebrae.ts) = sha256 of the decoded then re-encoded text (`importer.ts:77-80`).
|
||||
|
||||
## 6. JavaScript-specific behaviours (a Rust port could reproduce these differently)
|
||||
|
||||
Object key insertion order drives every JSON artifact. The draft document is the working form and is never re-parsed. An edited key stays in place and a new key is appended (`fields.ts:76-79`). `review_status` is appended last by the B/C rule (`apply.ts:57`): the draft's annotation keys end `…, vertebral_region_code, review_status`. Added items keep the key order authored in changes.json (`ops.ts:150`). References follow the Zod schema order, with url/doi defaulting to null (`packages/domain/src/anatomical-sheet/schema.ts:200-207`). ChangeSummary follows Zod output order: `priority` sits at its schema position, and `question` is omitted when absent.
|
||||
|
||||
JSON.parse rules: integer-like keys iterate first (`catalog.ts:240`; `catalog-checks.ts:14-16,42,72`; `migration-map.ts:137-138,150`); today there are 0 integer-like zone keys and 0 integer-like legacy ids. A duplicate key keeps the last value.
|
||||
|
||||
Map/Set insertion order: `svg.ts:97` first element per id, in document order; `categories.ts:71-79` first seen, the name filled later without moving the entry; also `catalog-checks.ts:41,71`, `overlay.ts:54,72`, `migration-map.ts:152`.
|
||||
|
||||
Object identity: `apply.ts:36-43,53` key a Map/Set by item object; `ops.ts:38` (`!== self`) and `ops.ts:159,176` (`indexOf(item)`) compare identity.
|
||||
|
||||
Sorting: `catalog-checks.ts:78` count descending, then `<` on strings (UTF-16 code-unit order; the helper at :64-67 is misnamed byCodePoint); `review-report.ts:153-164` priority rank, then id compared with `<`. The sort is stable; there is no localeCompare in src.
|
||||
|
||||
Number to string (ECMAScript rules): `format.ts:22` and `changelog.ts:33` (JSON.stringify); `annotations.ts:61` and the report.ts tables; `layer.ts:99-107,171-175,260`; `overlay.ts:82-84,91,105,113,131,139-143,168,193`; `seed.ts:62`. Rust has `crates/core/src/json.rs` `js_number:18`, `to_js_pretty:171`, `to_js_compact:200`; serde_json is built with `preserve_order` and `float_roundtrip`. There is no equivalent of Prettier's layout.
|
||||
|
||||
Floating-point arithmetic: `layer.ts:171` max(2, r − 12); `overlay.ts:105` cy − r − 4; `:109-113` centroid = sum ÷ n; `:131` fontSize/8; `:168` y + (i+1)·fontSize·1.3 left to right; `:201` Math.round(width/90), where a −x.5 value rounds toward +∞, unlike Rust's f64::round.
|
||||
|
||||
Number() parsing: `svg.ts:54-56` `Number(null)` and `Number('')` give 0, whitespace trimmed, hex and exponents accepted; `svg.ts:37-47` and `assets.ts:50-53` `.trim().split(/[\s,]+/).map(Number)` (a leading comma yields a 0 entry, an odd point count drops the last number); `overlay.ts:197-199` splits without trim.
|
||||
|
||||
Regex classes: `\s` is Unicode whitespace, NBSP and U+FEFF included (`svg.ts:40`, `assets.ts:52`, `overlay.ts:198`, `vertebrae.ts:17,20,27`); `\w` and `\d` are ASCII-only, no `u` flag (`vertebrae.ts:20`; `schema.ts:112` `^CHG-\d{3,}$`); `STABLE_ID_PATTERN` is `/^[a-z0-9][a-z0-9_-]{0,63}$/`; the combining range is written as literal characters U+0300–U+036F (`ids.ts:45`, `regions.ts:23`).
|
||||
|
||||
Normalisation and case: `normalize('NFD')`, strip U+0300–036F, `toLowerCase()` (`ids.ts:42-50`, `regions.ts:22-24`), ICU 78.3 tables; slugify ends with `.slice(0,64)`; no toUpperCase.
|
||||
|
||||
UTF-16 lengths: `compactValue` caps text at 120 units and cuts with `slice(0,119) + '…'` (`changelog.ts:34-35`), which can split a surrogate pair. Zod `.min(1)` also counts UTF-16 units.
|
||||
|
||||
Splitting and trimming: `trim` and `trimEnd` use the JS whitespace set (`svg.ts:39`, `assets.ts:51`, `seed.ts:79`); `split(':', 2)` discards the remainder (`migration-map.ts:82`), whereas Rust's splitn keeps it.
|
||||
|
||||
Decoding: TextDecoder fatal mode for the SVGs (`importer.ts:83`, `pipeline.ts:114`), which strips a leading BOM; `Buffer.toString('utf8')` is non-fatal (U+FFFD, BOM kept) for pathologies.json and vertebrae.ts (`sources.ts:51-52`); `readFile(…,'utf8')` reads the curation files (`curation/sources.ts:23`).
|
||||
|
||||
Deep equality and copies: `isDeepStrictEqual` ignores key order, NaN equals NaN, +0 differs from −0 (`ops.ts:91`, `overlay.ts:63`); `structuredClone` deep copies (`types.ts:46-47`, `fields.ts:79`, `ops.ts:109,150`, `apply.ts:127-128`).
|
||||
|
||||
Zod semantics: curation schemas are strict objects, discriminated by `op`; `.int()` accepts 2.0; defaults apply. Reference url = `z.string().url()` + `/^https?:\/\//i` (domain `schema.ts:189-192`); Rust has `domain/sheet/schema.rs:100-107` with `js::is_url`.
|
||||
|
||||
Markdown escaping: `cell()` escapes `|` and LF only, not CR (`markdown.ts:6-9`). No clock, no randomness: no Date, no Math.random, no randomUUID in src.
|
||||
|
||||
## 7. Dependency on packages/domain
|
||||
|
||||
22 source files import it, 17 of them type-only.
|
||||
|
||||
| Runtime symbol | Used in | Domain source |
|
||||
|---|---|---|
|
||||
| AnatomicalSheetDocumentSchema | legacy/importer.ts:2, curation/apply.ts:2 | anatomical-sheet/schema.ts:295 |
|
||||
| validateSheetDocument | importer.ts:4, apply.ts:3 | anatomical-sheet/validate.ts:465 |
|
||||
| SHEET_DOCUMENT_SCHEMA_VERSION (=2) | importer.ts:3, legacy/report.ts:1 | schema.ts:18 |
|
||||
| STABLE_ID_PATTERN | legacy/ids.ts:2 | schema.ts:21 |
|
||||
| ReferenceSchema | curation/schema.ts:1 | schema.ts:200 |
|
||||
|
||||
Type-only symbols (schema.ts unless noted): AnatomicalSheetDocumentInput (:315) in 13 files; Geometry (:322); SheetView (:321); SheetIssue (validate.ts:20); Reference (:210); ViewBox (:320); Laterality (:323); FindingCategoryKind (:325), FindingNodeInput (:227), LocalizedOptionalText (:213); AnnotationCategory (:319); ReportMigrationMap (report/migration-map.ts:33) and SheetRef (report/v3.ts:38).
|
||||
|
||||
Test runtime symbols: AnatomicalSheetDocumentSchema, validateSheetDocument, ReportMigrationMapSchema (report/migration-map.ts:15); curation.migration-leaves only: ReportV2Schema (report/v2.ts:59), findingPathKey (report/migration-map.ts:35), migrateReportV2ToV3 (report/migrate.ts:106), readReportDocument (report/migrate.ts:156).
|
||||
|
||||
Rust counterparts from R2: `crates/core/src/domain/sheet/{schema,validate}.rs` and `domain/report/{migration_map,v2,v3,read}.rs`.
|
||||
|
||||
## 8. Tests
|
||||
|
||||
Declarations counted by grep; `it.each` rows expanded by hand. (The orchestrator ran the suites: anatomy 131/131 in 16 files, domain 262/262.)
|
||||
|
||||
| Suite (lines) | Decl | Cases | What it asserts | Files read | Render dep |
|
||||
|---|---|---|---|---|---|
|
||||
| curation.apply (529) | 29 | 41 | each op and its records; error messages carry `CHG-001 op 0`; determinism (:425); purity (:429); B/C rule | fixtures/mini-sheet.ts | no |
|
||||
| curation.assets (79) | 5 | 5 | 30 annotated elements + titles removed, 270 paths kept, declaration kept, style removed, deterministic + XML re-parse, sha, parseViewBox | legacy-sources/Horse.svg | no |
|
||||
| curation.migration-leaves (271) | 8 | 8 | 612 leaves: 442 identical, 141 moved (134 + 7), 29 removed; 26 observable annotations; spine regions | committed legacy-import/document.json, draft/document.json, draft/report-migration-map.json + domain migrate | no |
|
||||
| curation.migration-map (183) | 5 | 5 | schema-valid map, fates, every target exists; synthetic rename/move/remove/re-home | runDraft, mini-sheet | no |
|
||||
| curation.overlay (83) | 4 | 4 | moved/removed markers, legend, detail framing | mini-sheet | no |
|
||||
| curation.pipeline (110) | 6 | 6 | 0 integrity error, 1 asset, head view_box 60 100 500 380, Cd 15–21, carpus, 811 nodes, CHANGELOG rows, 10 headings, SQL never publishes, CHG-000 refused | runDraft + renderDraftArtifacts | indirect |
|
||||
| curation.schema (85) | 5 | 9 | committed curation files parse; rejections; duplicates | curation/*.json | no |
|
||||
| document (23) | 2 | 2 | 0 issue; determinism | real sources | no |
|
||||
| importer.counts (103) | 8 | 8 | 27/26, 2 views 1 trigger, 5 spine codes, 9 bilateral, 792/612/41 on both sides | sources + regex over the SVGs | no |
|
||||
| importer.guards (47) | 3 | 3 | guard messages; dollar-quote tag | inline SVG, real import | no |
|
||||
| importer.special-cases (128) | 10 | 10 | region, laterality, trigger, ssb, anamnesis, lean form, authored labels | real import, pathologies.json | no |
|
||||
| legacy.regions (43) | 4 | 4 | inferRegionCode | none | no |
|
||||
| r3.layer-oracles (162) | 1 | 1 | 8 layer trees equal the frozen JSON | crates/core/tests/oracles/sheet.draft.json and r3/layer.json | direct |
|
||||
| render.compose (230) | 10 | 11 | highlight, halo `<defs>`, framing; app-{main,head}.svg string-equal | runDraft, draft/assets/horse.svg, draft/review/app-*.svg | direct + xmldom |
|
||||
| render.layer (158) | 5 | 5 | pair/zone/trigger, styles, states, DOM + serializeLayer | runDraft, mini-sheet | direct |
|
||||
| seed.golden (87) | 6 | 9 | see below | both generated folders | indirect |
|
||||
| Total | 111 | 131 | | | |
|
||||
|
||||
Golden test (`seed.golden.test.ts`): `beforeAll` re-runs the import and the draft in memory (:29-34); legacy-import: 4 files compared as `readFile(…,'utf8')` string === fresh render (:37-44); no SQL file in legacy-import (:46-50); the SVG copies compared with Buffer.equals against the sources, plus sha and size (:52-62); draft: one `it` loops over every file `renderDraftArtifacts` returns, comparing strings (:66-70); the draft folder must hold exactly the produced files plus `assets/<file>` (:72-78); horse.svg text === cleanedSvg, plus sha and size (:80-86). No comparison is done on parsed data.
|
||||
|
||||
Determinism tests: `document.test.ts:16-22` (two imports, toEqual, plus `renderArtifacts`), `curation.apply.test.ts:425`, `curation.assets.test.ts:46`. No test rendered the draft folder twice before R5.1 (the golden pins it against the committed bytes); `r5.pipeline-oracles.test.ts` now does.
|
||||
|
||||
## 9. Consumers
|
||||
|
||||
`@faunosteo/anatomy` outside the package: `package.json:28` (script filter) and `packages/README.md:12` only. No import in apps/, packages/*, scripts/, crates/, docker/, the Dockerfile, the vite configs or the eslint configs. Rust cites it in comments only (`crates/core/src/anatomy/{layer,compose}.rs`, `tests/anatomy_layer.rs`).
|
||||
|
||||
`@faunosteo/domain` outside its package: imported by `packages/anatomy` only (package.json, 22 source files, 8 test files). `apps/app/eslint.config.js:68-72` bans it in the app. `apps/app/tests/unit/r4.census.test.ts` :41-57 and :59-78 assert that apps/app and packages/core-wasm neither import nor depend on it; :80-90 asserts the lockfile lists `@xmldom/xmldom` under exactly `['packages/anatomy']`. `packages/core-wasm/tests/n-1` is a verbatim copy, not an import.
|
||||
|
||||
Path-level dependents: `crates/core/tests/report_rules.rs:141-143` reads report-v2.full.json (5 times) and report-v2.minimal.json (4 times); `archive_rules.rs:204,603,801,820,911` reads `packages/domain/tests/fixtures`; `archive_support/mod.rs:81` documents horse-slice.ts as its source.
|
||||
|
||||
Consumers of the generated artifacts: the app (`apps/app/vite.config.ts:13` + `vite/sheet-bundle.ts:46-63` bundle document, sheet-ref, migration map, assets.json and the assets; `apps/app/tests/helpers/legacyLeaves.ts:17` reads legacy-import/document.json); the core-wasm test suites (runtime, anatomy, measure, serialisation, pdf-import, pdf-import.budget, pdf-export, pdf-export.session); `scripts/browser/lib/core-module.mjs:25`; the crate tests (anatomy_canvas_op, anatomy_compose, sheet_index, sheet_rules, sheet_contract, editor_rules, fixtures_round_trip, the pdf_* suites, archive_support); the database (`supabase/config.toml:44` lists equine_draft.sql in sql_paths; `supabase/tests/007_equine_seed.sql` plans 19 pgTAP checks; the PGlite harness `packages/supabase/tests/db/harness.ts:21`, `seeds.test.ts`); the crate oracles `sheet.{draft,legacy-import}.json` and `migration-map.draft.json` carry the seed path as `source`.
|
||||
|
||||
## 10. Upload tool
|
||||
|
||||
`packages/supabase/scripts/upload-sheet-assets.ts` (99 lines): reads `<dir>/assets.json` (:48-53); validates each entry (plain file name, sha `/^[0-9a-f]{64}$/`, id `/^[0-9a-f-]{36}$/`, :28-40); re-hashes each file and compares sha and size (:56-63); uploads `<sha256>.svg` to bucket `anatomical-sheet-assets` as image/svg+xml with upsert false, an "already exists" error tolerated (:77-84); calls RPC `confirm_sheet_asset_upload(asset_id)` (:86-89). Needs `SUPABASE_URL` and `SUPABASE_SERVICE_ROLE_KEY` (:42-46, :71-75), resolves the directory from INIT_CWD (:67-70), imports @supabase/supabase-js plus node:crypto, node:fs and node:path. It imports nothing from packages/anatomy; it depends only on the assets.json contract.
|
||||
|
||||
## 11. ANATOMICAL_SHEET_MODEL.md (docs/development)
|
||||
|
||||
§6 starts at :304; the folder table at :313-317; changeset fields, ops and target grammar :319-331; asset cleaning and CHG-000 :333-337; review drawings :339-344; Step 4c content (:346) says "44 changes + CHG-000, 48 references" while the file now holds 45 authored changes (CHG-045, colours, documented at :401-404); head view :368-375; status semantics :377-384; migration map (:393-399): 792 = 571 identical + 180 renamed or moved + 7 re-homed + 34 removed, matching CHANGELOG.md:35-43 and the committed map; seed SQL :406-410; import rules :412-429; validation tiers §2.1 at :108.
|
||||
|
||||
## 12. Risks (analyzer)
|
||||
|
||||
- Any byte change in draft/document.json changes documentSha256: formatter, key order, number text or the trailing newline. That rewrites 18 pinned files and makes every older archive rebase once (certain for a Rust emitter under D-R5 as written).
|
||||
- Typed serde round-trips emit schema key order, not the working form's insertion order (e.g. `review_status` last), which produces a byte diff (high).
|
||||
- Prettier's JSON layout and its Markdown padding (string-width function) are not reproduced by serde_json or a plain padder (high if bytes must be preserved, none if the files are regenerated by decision).
|
||||
- Re-cleaning Horse.svg with the Rust reader (roxmltree plus its own serialiser, BDR-053) instead of xmldom changes the horse.svg bytes. That cascades into a new asset sha and svg_asset_id, a document change, new SQL, a new Storage object to upload, and changes to the anatomy_compose and pdf oracles that read horse.svg (medium to high).
|
||||
- Deleting packages/domain breaks the crate tests that read its fixtures (certain unless the fixtures move).
|
||||
- `r4.census.test.ts:80-90` fails once xmldom leaves the lockfile (certain).
|
||||
- Edge-input differences (Unicode `\s`, `Number('')`, `split` limit, `Math.round` sign, UTF-16 slice) are not triggered by today's inputs (no NBSP, tab, CR, BOM, astral character or exponent found) (low).
|
||||
- NFD and lowercase table differences between ICU 78.3 and a Rust crate: slugify never fires on today's data (low).
|
||||
- Prettier Markdown escaping could alter more than padding on future content (today: padding only) (low to medium).
|
||||
|
||||
## 13. Open questions (analyzer)
|
||||
|
||||
- Which fingerprint rule binds R5: D-R5 as written, or the BDR-045 amendment?
|
||||
- Byte-preserving port or one-time regeneration? Either the 18 files are reproduced byte for byte (Prettier JSON layout, Markdown padding, xmldom serialiser, insertion order), or they are regenerated once and checked by parsed-tree comparison.
|
||||
- horse.svg: keep the xmldom-cleaned bytes (asset 7f767f74…) as a frozen input, or re-derive them through the Rust reader (new sha, operator upload)?
|
||||
- Where do `packages/domain/tests/fixtures/report-v2.*.json` live after `packages/domain` is deleted?
|
||||
- Upload tool HTTP crate: the review (:300) names reqwest, the worker uses ureq. Which one?
|
||||
- Prettier on the seed: should `supabase/seed/**` join `.prettierignore` (today a root `pnpm format` rewrites the generated JSON and Markdown)?
|
||||
- Stale comment: `vertebrae.ts:3` points to `apps/app/src/domain/species/equine/vertebrae.ts`; the source is `legacy-sources/vertebrae.ts`.
|
||||
@@ -192,3 +192,9 @@ Method: memory bench 3 engines × demo/L × 6 exports + Chromium L 16 + raster c
|
||||
Result: verifier pass 1 ECARTS(1): core with no feature did not compile (`restore_copy` used gated `pdf::read` const; workspace feature unification via crates/wasm hid it in rust-check/check:all) → gated + no-feature clippy in rust-check; pass 2 ECARTS(1): §6 lacked J2 merge-gate row → added. Security pass 1 BLOCK(1): `.gitleaks.toml` allowlist (R4.1, owner-approved in journal) lacks owner `[gated]` marker → owner action; MEDIUM serve.mjs `--host ""` bound every interface.
|
||||
Anomalies: my first host fix = exact-string denylist → bypassed by `0`, `0::`, `::ffff:0.0.0.0`, scoped `::` (re-gate 1); rebuilt resolve-then-BlockList; re-gate 2 found post-bind check dead for IPv6 (`family` 'IPv6' vs 6) and my fstat-first CLI read opened FIFOs before the type check (regression). Each fix of a gate finding introduced the next finding: 3 re-gates for 2 small fixes. Contract CHECKs used root vitest path (absent): oracle defect.
|
||||
Action: owner checkpoint; floor 21 DELETED_TEST (R4.8 deletions, approved in brief §1) need owner `[gated]` clarification line to clear formally.
|
||||
|
||||
## EVAL-026 | 2026-10-22 | R5.1 freeze + D-R5 study — analyzers ×2, oracle generator, cross-stack digests
|
||||
|
||||
Method: 2 analyzer agents (pipeline inventory 43 files, fingerprint census: producers, consumers, 99 PDFs, docs), both refused to write files (role rule) → orchestrator saved reports; generator test (`MAKE_R5_ORACLES=1`) + freeze re-run green 6/6, anatomy 137, domain 262; Node scripts (matrix, PGlite jsonb) + scratchpad Rust binary on core crate: digests compared value by value; tie census 200k values.
|
||||
Anomalies: R2 oracle test self-referential on number printing (LRN-070) → F1 undetected since R2; docs contradict (BDR-045 vs RUST.md §10.10 / TODO "will change once") → cross-reference added; analyzer inventory counted "10 anomalies" wrong in orchestrator draft (actual 27, records 825 not 690, 16 ledger artifacts) → corrected from oracle counts before commit.
|
||||
Action: F1 fix proposed, owner go; D-R5 recommendation B + aliases; checkpoint delivered, STOP.
|
||||
@@ -318,3 +318,7 @@ Owner verdict: J1.7 PASS w/ reserve, M3 complementary, J2.3 option 1 final, logo
|
||||
## 2026-10-21 (R4 closed and merged)
|
||||
|
||||
Owner: J1 + J2 PASS on computer, Android, real iPhone Safari iOS (declared as a whole; model/versions/timings/memory not given → marked missing, not invented); R4.12 validated; merge authorized → BDR-067. Consolidation 3a7937e: R4_J1/J2_RESULTS + MINI_RECETTE, RUST.md §10.15 (risk table, deadlines, final checks), PDF_IMPORT_HARDENING §7 (items 3/19 unblocked, 5/6/7/9/20 open), CHANGELOG headers, TODO R4.13. Final checks on 22bbfb4 (code identical through 3a7937e): check:all green 22:15:50 (app 434, domain 262, core-wasm 58, anatomy 131, supabase 40, worker TLS + debian13 digest), audit --prod clean, gitleaks 111 commits clean (tree: ignored `.env.worker` only), no local path/personal id, merge-tree clean, goldens 8b86b92 / oracles 4439fd2 / digests d492bf7 untouched, 51 deletions all 0dc1ba2. `gitflow finish feature rust-r4-pdf-core` → merge 96ae75e (parents 66909b6 + 3a7937e, 113 commits), develop pushed = origin, branch deleted local + origin, main 605ee7c untouched, release/0.2.0 untouched. Not done: release 0.2.0, deploy, DNS, R5, D2/D4/D5. Next: owner signal for R5 (entry: plan D-R5, `packages/anatomy` pipeline → `crates/core`, seed golden regenerated once).
|
||||
|
||||
## 2026-10-22 (R5.1 freeze + D-R5 study)
|
||||
|
||||
Owner: R4 bilan validated, R5.1 authorised (freeze TS pipeline, study D-R5, stop before Rust pipeline + fingerprint change). Branch `feature/rust-r5-pipeline-core` from 9c0117a. Two analyzers (inventory, documentSha256 census) → `.claude/audits/R5_*.md`. Oracles `oracles/r5/{pipeline,ids,fingerprints}.json` via `r5.pipeline-oracles.test.ts` (MAKE_R5_ORACLES=1; 6 cases, determinism two runs, ledger vs committed). Study scripts `scripts/r5-fingerprints.mjs`, `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`; throwaway Rust experiment (scratchpad) reproduced every draft digest byte-for-byte, JCS 40 lines no crate, RFC vectors pass; jsonb round trip keeps B only. Found F1: `js_number` tie rounding (Grisu up vs ES6 even; `840.1692504882812`, 7.4 % of float32 values; `zmij` matches) hidden by self-referential R2 oracle test → LRN-070, EVAL-026, not fixed (owner go). Plan `2026-10-22-rust-r5-pipeline-core.md`, RUST.md §11, TODO R5.1 ticked. Recommendation: B on validated form + alias mechanism (old `27c7…` stays valid, no visible migration). STOP at checkpoint. No Rust pipeline, no fingerprint change, no merge.
|
||||
@@ -519,3 +519,8 @@ Apply: when an implementation is swapped under a test that spied the old one, re
|
||||
|
||||
- Context7 said nginx `listen ... http2` deprecated → `http2 on;`, but live VPS sends `Server: nginx/1.24.0` and 1.24 rejects `http2 on` (`unknown directive`). Apply: before changing a runbook directive, read the target's version (Server header, `nginx -v`) and `nginx -t` both forms in throwaway containers of each version; document both when the fleet straddles.
|
||||
- WHATWG URL normalisation can create the escape it was meant to catch (`/..//host` → pathname `//host`, same origin for the probe). Apply: sanitiser re-validates the value it returns (not only the input), plus an idempotence test; fuzz with dot segments, `%2e`, tabs inside `..`.
|
||||
|
||||
## LRN-070 | 2026-10-22 | Rust `{:e}` shortest digits ≠ ECMAScript on exact decimal ties; oracle that re-prints expected through the port cannot see it
|
||||
|
||||
Found by R5.1 study. `js_number` (core json.rs) takes shortest digits from `format!("{:e}")` (Grisu): exact tie between two shortest candidates → Grisu picks upper, ECMAScript/Python/`zmij` pick even. `840.1692504882812` (= 840.16925048828125, float32-exact, Horse.svg legacy cx) → Rust `…813`. Rate: 7,401 / 100,000 float32-exact values in 0..2000, 769 / 100,000 random doubles, `zmij` (serde_json 1.0.154 formatter, locked) 0. Draft seed has no tie → every R2/R3/R4 suite green, but `sheet_contract.rs` `zod_oracle` computes expected = `to_js_pretty(oracle.output)` → same function both sides → blind. 134 JSON fixtures Node vs Rust pretty: 2 differ (legacy baseline, that value).
|
||||
Apply: shortest-float port → take digits from `zmij`/ryu-class crate, never `{:e}`; tie vectors (float32-exact values) in unit tests; oracle expected must be the recorded JS text, never re-printed by the port under test. Any R5 importer emitting legacy coordinates needs the fix first.
|
||||
+10
-1
@@ -304,7 +304,16 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo
|
||||
- [x] owner actions: J1 + J2 on the OD-6 devices (done 2026-10-21, iPhone included); still owner-side: delete the UFW 4173 rule after the recette sessions (`sudo ufw status numbered`, `sudo ufw delete <n>`)
|
||||
- [ ] owner action (non-blocking): `packages/pdf/node_modules` leftover (13 pnpm symlinks, 6 bin shims, 1 vitest cache file, 0 tracked or useful files) → `rm -rf packages/pdf/node_modules && rmdir packages/pdf` (the recursive delete was refused to Claude by the permission layer)
|
||||
- [ ] outside R4 (owner 2026-10-17, non-blocking): worker container ignores SIGTERM (Docker kill after 10 s; chore, e.g. `init: true`); README test counter stale (367 vs 1,015)
|
||||
- [ ] R5 curation pipeline + importer + upload tool: seed golden regenerated once (new `documentSha256`, D-R5), `packages/anatomy` + `packages/domain` deleted, docs map; all before Step 13
|
||||
- [ ] R5 curation pipeline + importer + upload tool (plan `.claude/tasks/plans/2026-10-22-rust-r5-pipeline-core.md`, branch `feature/rust-r5-pipeline-core` from 9c0117a; owner 2026-10-22: R5.1 authorised, stop before any Rust pipeline code and before any `documentSha256` change; no merge without validation, no deletion of packages/anatomy or domain, no golden or historical change, no 0.2.0, no deploy, no SaaS feature): `packages/anatomy` + `packages/domain` deleted at the end, docs map; all before Step 13
|
||||
- [x] R5.1 freeze + D-R5 study (2026-10-22): inventory `.claude/audits/R5_PIPELINE_INVENTORY.md`, census `.claude/audits/R5_DOCUMENT_SHA256_CENSUS.md`, oracles `crates/core/tests/oracles/r5/{pipeline,ids,fingerprints}.json` (generator `packages/anatomy/tests/r5.pipeline-oracles.test.ts`, `MAKE_R5_ORACLES=1`), study scripts `scripts/r5-fingerprints.mjs` + `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`, RUST.md §11, anatomy 137/137, domain 262; F1 found: core `js_number` rounds exact ties upward (JS: to even; `zmij` matches), 1 legacy value, invisible on the draft, R2 contract test self-referential → checkpoint, STOP
|
||||
- [ ] owner decisions at the checkpoint: D-R5 (A / B validated form / C; alias mechanism; convention name), F1 fix go, byte-preserving vs one-time regeneration per artifact, horse.svg frozen bytes vs re-clean, `.prettierignore` on `supabase/seed/**`, upload tool HTTP crate
|
||||
- [ ] R5.1b F1 fix (`zmij` digits, tie unit tests, `zod_oracle` on the recorded JS text) after the owner's go
|
||||
- [ ] R5.2 legacy importer in Rust + `import-equine` bin (legacy-import/ byte-exact, IMPORT-REPORT by table padding or parsed tables)
|
||||
- [ ] R5.3 changeset applier, migration map, stats, seed SQL (draft JSON + SQL under the decided fingerprint, 825 records, 612 leaves)
|
||||
- [ ] R5.4 review overlays + app drawings on the R3 layer/compose (parsed trees), CHANGELOG + REVIEW-REPORT
|
||||
- [ ] R5.5 `upload-sheet-assets` bin + runbook (local stack run)
|
||||
- [ ] R5.6 D-R5 switch (convention, aliases, membership test in gated readers with per-file go, seed regenerated once by the Rust tool, `.prettierignore`, CHANGELOG)
|
||||
- [ ] R5.7 consumer census, domain fixtures moved, packages deleted, xmldom + prettier out of the lockfile, docs, ledger; `pnpm check:all`; merge only on the owner's signal
|
||||
- [ ] separate chore (D-R7): form constraints module + PGlite `pg_constraint`/config.toml parity test
|
||||
- [ ] Release 0.2.0 (branch release/0.2.0 = Steps 3–6, version.txt + package versions 0.2.0, CHANGELOG [0.2.0] 2026-10-11): deploy-trigger check done 2026-10-12 (CI = branches main/develop only, no tag trigger, no deploy job, docker job builds `faunosteo:ci` locally; hooks push only; DEPLOY.md deploy = manual on the VPS). Default 0.2.0 build carries NO sheet (SHEET_BUNDLE=none) → the live local tool must not be redeployed from main without `--build-arg SHEET_BUNDLE=draft`. Finish via /release-candidate (finish + tag span, tag push on explicit go); CHANGELOG back-merge conflict expected (Unreleased Step 7 above [0.2.0]). Awaiting the user's go.
|
||||
---
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# PLAN: Rust R5, the anatomical pipeline (legacy importer, curation, migration map, seed, upload tool), `packages/anatomy` and `packages/domain` deleted
|
||||
|
||||
- date: 2026-10-22 | rev 1 (R5.1 only; the later slices are a proposal for the R5.1 checkpoint) | branch: `feature/rust-r5-pipeline-core` (cut from develop 9c0117a, R4 merged)
|
||||
- spec: owner brief 2026-10-22 « Clôture R4 validée et autorisation R5.1 » (§A.0); `docs/development/RUST_ARCHITECTURE_REVIEW.md` II.2 rows "anatomy curation pipeline" and "upload-sheet-assets" (lines 299, 300), VI.1 row R5 (636), VI.3 (655), VI.4 (662 to 668), D-R5 (837, 873), Annex C D-R5 amendment (974); BDR-043, BDR-045; CLAUDE.md "Anatomical sheet model", "Schemas", archive-reader gate
|
||||
- facts: `.claude/audits/R5_PIPELINE_INVENTORY.md` (the TypeScript pipeline), `.claude/audits/R5_DOCUMENT_SHA256_CENSUS.md` (the fingerprint), the oracles under `crates/core/tests/oracles/r5/`, the study scripts `scripts/r5-fingerprints.mjs` and `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`
|
||||
- status: R5.1 authorised (freeze and study, no Rust pipeline, no fingerprint change). Checkpoint report due at the end of R5.1; D-R5 decided by the owner on the evidence.
|
||||
- not in scope of R5.1: any Rust implementation of the pipeline, any change to `documentSha256` or to the readers, the deletion of `packages/anatomy` or `packages/domain`, any golden or historical reference regeneration, release 0.2.0, deployment, SaaS features.
|
||||
|
||||
# Part A. For the owner
|
||||
|
||||
## A.0 Your brief (2026-10-22), applied as written
|
||||
|
||||
Branch from develop by gitflow (done). Existing plan and decisions read, no new architecture study. R5.1 = characterise the TypeScript pipeline and freeze its results (legacy import, draft, changesets, migration maps, CHANGELOG and REVIEW-REPORT, review drawings and overlays, SVG assets and upload, seed SQL, current fingerprints, determinism, compatibility with older versions and archives); reuse the existing oracles, fixtures and goldens where they suffice; keep the N-1 readers; never bend a historical reference to a new implementation. D-R5: a targeted, reproducible study of option A (sha256 of the bytes the Rust pipeline emits) against option B (sha256 of RFC 8785 canonical JSON) on the project's real data, a matrix of old and new fingerprints, an encoding change told apart from a content change, a versioned convention and a compatibility mechanism for the old fingerprints, no silent loss, no visible migration just to hide an algorithm change when a transparent and safe compatibility exists. Business invariants of §4 preserved by every later migration; no sheet declared validated by a technical migration. Stop before any Rust pipeline code and before any fingerprint change.
|
||||
|
||||
## A.1 What R5 delivers (the whole phase, for orientation)
|
||||
|
||||
The curation pipeline (legacy import, changeset application, migration map, CHANGELOG, REVIEW-REPORT, review drawings, cleaned asset, seed SQL) and the asset upload tool run in Rust (`crates/core` curation modules behind a feature, `cli` binaries `import-equine` and `upload-sheet-assets`), with the same inputs (legacy sources, `curation/changes.json`, `curation/references.json`) and the same semantics (ids, counts, 612-leaf migration, integrity rules). `packages/anatomy` and `packages/domain` leave the repository once nothing imports them; the N-1 Zod reader under `packages/core-wasm/tests/n-1` stays. The sheet fingerprint follows the D-R5 decision.
|
||||
|
||||
## A.2 R5.1 deliverables (this slice)
|
||||
|
||||
1. Inventory of the TypeScript pipeline: `.claude/audits/R5_PIPELINE_INVENTORY.md`.
|
||||
2. Oracles frozen with the TypeScript code (§B.2), generated by `MAKE_R5_ORACLES=1` from `packages/anatomy/tests/r5.pipeline-oracles.test.ts`, which otherwise checks the pipeline still matches its freeze.
|
||||
3. The D-R5 study (§B.3): facts, the fingerprint matrix, the compatibility mechanism, the versioned convention, costs, platform determinism, I-JSON and Unicode checks, the incidence on the archives already exported.
|
||||
4. The checkpoint report (the owner's §5 list), written in RUST.md §11 and delivered in the conversation.
|
||||
|
||||
## A.3 Findings that need your decision at the checkpoint
|
||||
|
||||
- D-R5 itself (§B.3.6): A, B or C (reproduce Prettier's bytes), the canonical form (input or validated), the compatibility mechanism, the convention's name.
|
||||
- F1, `js_number` tie rounding (§B.3.4): the core prints an exact decimal tie upward (`840.1692504882813`) where JavaScript, Python and `zmij` print to even (`…812`); 1 value of the legacy sources, 7.4 % of float32-exact coordinates, 0.8 % of random doubles. Invisible today (the curated draft holds no tie; the R2 contract test re-prints its expected text through the same function). The fix is small (digits from `zmij`, already locked through serde_json) and changes core bytes only for tie values; it precedes any Rust importer. Your go is needed because it touches a core serialiser (D-R11).
|
||||
- The documentation contradiction: BDR-045 and Annex C say "existing fingerprints untouched", RUST.md §10 and the TODO still say "will change once". The plan follows BDR-045 (the later amendment) until you say otherwise.
|
||||
- Byte-preserving port or one-time regeneration of the generated files (§B.5): the Rust pipeline can reproduce the JSON bytes only by emitting insertion order and Prettier's short-array rule, and the Markdown bytes by padding tables (the only Prettier effect measured); the review SVGs cannot match xmldom's serialiser and are compared as parsed trees. The recommendation is: JSON files byte-exact where cheap (`to_js_pretty` plus the one array rule) or regenerated once with a JCS-equality proof, Markdown byte-exact by padding, SVGs regenerated once with a tree proof, and `supabase/seed/**` added to `.prettierignore` at the switch (today a root `pnpm format` would reformat Rust-written files).
|
||||
- `draft/assets/horse.svg`: keep the xmldom-cleaned bytes as the frozen asset (sha `0ea5202f…`, id `7f767f74…`, already the Storage object) and have the Rust tool verify rather than regenerate it, or re-clean it with the R3 reader and accept a new asset sha, id, document, SQL and Storage upload. Recommendation: keep the frozen bytes (the cleaning was a one-time legacy transformation; the R3 reader already proves the file).
|
||||
|
||||
## A.4 Stop rules of R5.1
|
||||
|
||||
Stop and report when: a generated file does not reproduce from a fresh run (the golden test fails); an oracle generator would have to alter a committed generated file or golden; a study step needs a change to a gated reader; the branch would need a develop merge.
|
||||
|
||||
# Part B. The plan
|
||||
|
||||
## B.1 Inventory
|
||||
|
||||
See `.claude/audits/R5_PIPELINE_INVENTORY.md`: 43 source files, 4,897 lines (legacy 1,842, curation 2,329, render 553, cli 119), 2,553 lines of tests (131 cases, 16 files), runtime libraries Prettier 3.8.3, @xmldom/xmldom 0.9.12, Zod 3.25.76 (the first two declared as devDependencies). 18 generated files: 7 JSON and 3 Markdown through Prettier, 5 SVGs through xmldom's XMLSerializer, 1 raw SQL template, 2 byte copies. The real changeset: 45 changes (14 A, 22 B, 9 C), 687 ops (587 set, 45 cite, 33 rename, 18 add, 4 remove, 0 move), 48 references. Measured Prettier effect on today's data: Markdown table padding only (134, 863 and 52 lines), JSON short scalar arrays collapsed plus a trailing LF; the data is identical after re-parsing. The JSON key order is the working form's insertion order (`review_status` appended last), not the schema order a typed serde struct would emit. `@faunosteo/anatomy` has no consumer outside itself; `@faunosteo/domain` is imported by `packages/anatomy` only (5 runtime symbols, all with R2 counterparts); the crate tests read `packages/domain/tests/fixtures/report-v2.*.json` and `r4.census.test.ts` pins xmldom under `packages/anatomy`. Entry points: `importEquineLegacy` (legacy sources → ImportResult), `buildEquineDraft` (ImportResult + curation → DraftResult), `renderArtifacts` (4 legacy files), `renderDraftArtifacts` (document, assets, sheet-ref, migration map, 4 review SVGs, CHANGELOG, REVIEW-REPORT, seed SQL), `cli/import-equine.ts` (writes both folders, removes stale files), `packages/supabase/scripts/upload-sheet-assets.ts` (bucket `anatomical-sheet-assets`, key `<sha256>.svg`, `upsert: false`, bytes verified against `assets.json`, RPC `confirm_sheet_asset_upload`). JavaScript semantics to pin: `deterministicUuid` (sha256 of the name, version nibble 5, variant 10xx), `slugify` (NFD, combining marks U+0300 to U+036F stripped, `toLowerCase`, `[^a-z0-9]+` → `-`, trimmed, 64 chars), `sha256Hex` over UTF-8, `JSON.stringify(v, null, 2)` then Prettier json (short scalar arrays collapsed when they fit in 80 columns, objects always expanded), Prettier markdown, xmldom `XMLSerializer` for the review SVGs, `TextDecoder('utf-8', { fatal: true })`.
|
||||
|
||||
## B.2 Oracles frozen at R5.1 (`crates/core/tests/oracles/r5/`, generator `MAKE_R5_ORACLES=1`, never regenerated without the owner)
|
||||
|
||||
| Id | Content | Location | Used by |
|
||||
|---|---|---|---|
|
||||
| P1 | the committed generated folders `legacy-import/` (5 files) and `draft/` (14 files): the byte oracle, already pinned to a fresh run by `seed.golden.test.ts` | `supabase/seed/anatomical-sheets/equine/{legacy-import,draft}` | every R5 slice |
|
||||
| P2 | `pipeline.json`: legacy ImportResult without the document (sheetId, viewBoxes, stats, source digests, assets, 27 anomalies, issues, region inferences, zone stats, annotation rows); DraftResult without the documents (46 changes, assets, stats before and after, issues, migration stats, 825 change records with before and after); the ledger of the 16 text artifacts (folder, file, bytes, sha256; for the 3 markdown files the sha256 before Prettier and whether Prettier changed the text) | `oracles/r5/pipeline.json` (1.3 MB) | R5.2 to R5.5 |
|
||||
| P3 | `ids.json`: `deterministicUuid` over the 3 names the pipeline derives ids from plus 3 edges; `slugify` over 17 edge labels and every `name.fr` / `name.en` of both documents | `oracles/r5/ids.json` | R5.2 |
|
||||
| P4 | `fingerprints.json`: for the 7 JSON artifacts the fingerprint under each rule (today's file bytes, A pretty with and without LF, A compact, B RFC 8785; for the two documents also the validated form under A and B); the ES6 text of the 65 non-integer numbers of the two documents; the two RFC 8785 vectors | `oracles/r5/fingerprints.json` | D-R5, R5.2 |
|
||||
| P5 | migration semantics: `draft/report-migration-map.json` (P1) and the 612-leaf suite `curation.migration-leaves.test.ts`, plus the R2 oracle `oracles/migration-map.draft.json` and the Rust `archive_support` walk | existing | R5.3 |
|
||||
| P6 | the review drawings `draft/review/{annotations,app}-{main,head}.svg` (P1, xmldom bytes) and the R3 layer and highlight oracles | existing | R5.4 (parsed-tree comparison) |
|
||||
| P7 | the upload tool's contract (B.1) and the operator check SUPABASE.md §3.1 | documented | R5.5 |
|
||||
| P8 | determinism: two in-process runs byte-equal (test in the generator file); `pnpm import:equine` not run (it rewrites the folders; the golden proves the same bytes) | test | every slice |
|
||||
|
||||
## B.3 D-R5 study
|
||||
|
||||
### B.3.1 Today's rule
|
||||
|
||||
`documentSha256 = sha256Hex(formatJson(document))`: the Prettier-formatted JSON text of the curated document in its input form (defaults omitted), the exact bytes of `draft/document.json`, embedded verbatim in `equine_draft.sql` and copied into `sheet-ref.json` and `report-migration-map.json`. One value has ever existed: `27c7383c…2660`, unchanged since 0a17e39 (2026-10-10), inside 99 committed PDFs, 8 JSON fixtures and 6 Rust oracle files. Compared, never recomputed, by the rebase (fingerprint only), the envelope invariant and the export (sheetId and fingerprint). The database stores the document as `jsonb` and no fingerprint. Full census: `R5_DOCUMENT_SHA256_CENSUS.md`.
|
||||
|
||||
### B.3.2 The matrix (real data, `node scripts/r5-fingerprints.mjs`, cross-checked in Rust)
|
||||
|
||||
| Artifact | Today (file bytes) | A: `to_js_pretty` + LF | A without LF | B: RFC 8785 | B validated form |
|
||||
|---|---|---|---|---|---|
|
||||
| draft/document.json | 27c7383c…2660 | 90f668a3…aa1d | 79f143c0…eb51 | a56e4a24…88a1 | 12d78872…74e2 |
|
||||
| legacy-import/document.json | 91e459f4…14c2 | 8c94a464…adc2 (Node) | 6fb31371…7f66 (Node) | d8b5f2cf…81eb (Node) | 1217637e…c024 (Node) |
|
||||
| draft/report-migration-map.json | 4139c51d…fce9 | 4139c51d…fce9 (same) | 01fac788…3471 | f082b66f…2ca6 | n/a |
|
||||
| draft/assets.json, sheet-ref.json, legacy-id-map.json, legacy assets.json | file = A with LF (no short array to collapse) | | | | |
|
||||
|
||||
Rust reproduction (throwaway experiment on the core crate, `to_js_pretty`, `to_js_compact`, a 40-line JCS on `js_number`): every draft digest identical to Node, byte for byte; the legacy-import digests differ through the single tie value (F1, §B.3.4); the RFC vectors pass; JCS idempotent. Node vs Rust `JSON.stringify(v, null, 2)` over the 134 JSON fixtures of the repository: 2 differ, both the legacy baseline, both on that value; `oracles/characterisation.json` is not parseable by serde (lone surrogates, R2 L1 rule).
|
||||
|
||||
Encoding change vs content change: the today / A / B columns of one row are three encodings of one value (parse → deep equal; `validated_equals_input` is false only because the validated form fills defaults, not because content differs). A change of the draft content changes every column.
|
||||
|
||||
### B.3.3 Facts the brief asked for
|
||||
|
||||
- UTF-16 key order: every key of every artifact is ASCII; UTF-16 order equals code-point and byte order on this data. The Rust canonicaliser sorts by `encode_utf16()` anyway (the RFC vector with U+0080, ö, €, 😀 and U+FB33 passes).
|
||||
- Numbers: 208 (draft) and 195 (legacy) tokens, 29 and 36 non-integer, none negative, no exponent, no `-0`, no integer beyond 2^53; all already in ECMAScript `Number::toString` form. JCS mandates exactly that form; the core's `js_number` implements it with one tie defect (F1).
|
||||
- Unicode: 788 non-ASCII strings (precomposed French accents, « », –, °, —), all NFC, no control character, no escape, no astral character, no lone surrogate. JCS does not normalise; a lone surrogate (I-JSON forbids it) is refused by the core (L1) and never produced by the pipeline.
|
||||
- I-JSON: satisfied by both documents (no duplicate key, numbers within f64, valid Unicode).
|
||||
- Cost and dependencies: option A costs nothing (the core already has `to_js_pretty`, D-R11); option B costs about 40 lines in `crates/core/src/json.rs` over `js_number` and serde_json's escaper, no new crate (`serde_jcs`, listed as rejected in the review, is not needed; `zmij` 1.0.23 is already locked through serde_json 1.0.154); a 15-line Node twin for tests and tooling. Hashing: `sha2` becomes a normal dependency of the core (today dev-only) or the hash stays outside the library (the tool computes it); the wasm bundle grows by the sha256 code only if the browser must verify fingerprints, which no current path does.
|
||||
- Platform determinism: B is a pure function of the parsed value (sort, ES6 numbers, fixed escapes); Node 24 and Rust 1.99 agree on every draft artifact. A depends on the writer's layout rules; today's value depends on Prettier 3.8.3's line fitting.
|
||||
- Database: a PostgreSQL 17 `jsonb` round trip (PGlite, `packages/supabase/scripts/r5-jsonb-fingerprint.mjs`) keeps B for both documents and breaks today's value and A (keys reordered, scale kept). B is the only candidate a service could recompute from `anatomical_sheet_versions.content` at Step 13.
|
||||
|
||||
### B.3.4 F1: `js_number` prints exact ties upward
|
||||
|
||||
`crates/core/src/json.rs` takes the shortest digits from `format!("{:e}")` (Grisu); on an exact decimal tie between two shortest candidates ECMAScript picks the even one, Grisu the upper one. Measured: `840.1692504882812` (legacy annotation `cx`, from Horse.svg) prints `…813`; 7,401 of 100,000 float32-exact values in 0..2000 and 769 of 100,000 random doubles differ; `zmij` (serde_json's formatter) differs on none. Reached by the canvas layer (R3), the PDF writer, the archive attachment, the R2 contract tests. Invisible today: the curated draft carries no tie value, and `sheet_contract.rs` re-prints its expected text through `to_js_pretty`, so the oracle cannot see it. Proposed fix (after your go, before R5.2): digits from `zmij` (or `serde_json::to_string(&f64)`), the two tie values and the 8 long decimals of the legacy SVG as unit tests, `zod_oracle` comparing against the JS text recorded in the oracle rather than a re-print.
|
||||
|
||||
### B.3.5 Compatibility mechanism (proposal, not implemented)
|
||||
|
||||
The sheet side carries the convention and the aliases; the report contract does not change.
|
||||
|
||||
- `sheet-ref.json` (and the bundle payload `ref`, and `anatomical_sheet_versions` when Step 13 stores a fingerprint): `document_sha256` holds the current convention's value; a new `fingerprint` block names the convention (`"algorithm": "rfc8785-sha256"` for B, `"js-pretty-sha256"` for A) and lists `aliases`: the fingerprints of the same content under earlier conventions, computed once at the switch from the frozen P4 oracle (`27c7383c…2660` for the current draft). The alias is bound to identical content by construction (the oracle records old and new over the same bytes), so a report stamped `27c7…` is the same sheet, not a rebase.
|
||||
- The rebase and the envelope invariant treat a fingerprint equal to the current value or to an alias as "same document". Gated readers: `report/rebase.rs`, `archive/read.rs`, `session.rs`, `pdf/write/input.rs` (your ask-before-touch rule; the change is one membership test).
|
||||
- A new report is stamped with the current value; an old archive opens with no message and no rebase; an archive on an older content revision rebases as today. No archive is rewritten; the exported envelope keeps what it holds.
|
||||
- Report v3 keeps `sheet.documentSha256` (64 hex) so the N-1 reader and every fixture stay valid; the convention's name lives with the sheet, which is the one place that knows which rule produced its value.
|
||||
- Fixtures: 99 PDFs, 8 JSON fixtures and 6 oracles keep `27c7…` untouched (they remain valid archives of the same content through the alias).
|
||||
|
||||
### B.3.6 Options compared
|
||||
|
||||
| | A: bytes the Rust pipeline emits (`to_js_pretty` + LF) | B: RFC 8785 of the validated document | C: Rust reproduces Prettier's bytes |
|
||||
|---|---|---|---|
|
||||
| Value at the switch | 90f668a3… (changes once) | 12d78872… (changes once) | 27c7383c… (unchanged) |
|
||||
| Needs the alias mechanism to avoid a visible migration | yes | yes | no |
|
||||
| Survives a reformat, a writer change, a jsonb round trip | no | yes | no |
|
||||
| Code | 0 lines (exists) | ~40 lines Rust + 15 Node, no crate | a Prettier-json subset printer (fill rule for scalar arrays, 80 columns), bound forever to a formatter's layout |
|
||||
| Risk | identity re-tied to a serialiser (the Prettier problem moved to Rust) | one more concept; the canonical form must be named (input or validated) | fragile: any Prettier rule the data exercises later changes the value |
|
||||
| Review's row | D-R5 rev 2 | Annex C preference, BDR-045 | rejected rev 1 (in Node form) |
|
||||
|
||||
Recommendation: B over the validated form (what every reader holds, recomputable from the database after validation), the alias mechanism of §B.3.5, the convention named in `sheet-ref.json`, F1 fixed first. The old value `27c7…` stays valid forever as an alias; no archive sees a message for the switch. If you prefer no new concept at all, C is possible on this data (the only Prettier effect measured is the collapse of short scalar arrays), but it keeps the fingerprint chained to a formatter's layout, which Annex C refused.
|
||||
|
||||
## B.4 Business invariants (brief §4) and where they are proven
|
||||
|
||||
| Invariant | Oracle or test |
|
||||
|---|---|
|
||||
| stable ids | P3 (uuid, slug), P1 documents, `legacy-id-map.json` |
|
||||
| zones and anatomical points, main and detail views | P1 documents (annotations, views), P2 annotation rows and records |
|
||||
| annotation ↔ observation relations (`finding_zone_id`, `opens_view_id`) | P1, integrity rules (R2 core, 21 codes) |
|
||||
| vertebral configurations | P1 `vertebral_config`, P2 region inferences |
|
||||
| bilingual data | P1, P3 slug over every `name.fr` / `name.en` |
|
||||
| historical references | `curation/references.json` (authored, never generated), P1 |
|
||||
| existing reports and bilans | P5 (migration map, 612 leaves), the R2 rebase oracles, the R4 PDF fixtures, the alias mechanism |
|
||||
| no sheet validated by a migration | the seed inserts a draft (`insert_sheet_draft`, status draft); publication stays a platform-admin action |
|
||||
|
||||
## B.5 Slices after R5.1 (proposal for the checkpoint; nothing started)
|
||||
|
||||
| Slice | Deliverable | Proof |
|
||||
|---|---|---|
|
||||
| R5.1 | this freeze and study | oracles committed, generator test green, checkpoint report |
|
||||
| R5.1b | F1 fix in `json.rs`; `zod_oracle` compares the recorded JS text | tie unit tests; every R2/R3/R4 suite unchanged (no tie in the draft) |
|
||||
| R5.2 | legacy importer in Rust (`crates/core/src/curation/legacy`): SVG sources through the R3 reader, catalog, regions, ids, anomalies, stats; `import-equine` emits `legacy-import/` | P1 bytes of document.json, legacy-id-map.json, assets.json reproduced exactly; IMPORT-REPORT compared as parsed tables (P2 pre-Prettier digest) or byte-exact if the Prettier markdown effect is nil |
|
||||
| R5.3 | changeset schema, targets, ops, applier, migration map, stats, seed SQL | P1 document.json, sheet-ref.json, report-migration-map.json, equine_draft.sql byte-exact under the decided fingerprint; P2 825 records equal; 612-leaf suite ported |
|
||||
| R5.4 | review overlays and app drawings on the R3 layer and compose; CHANGELOG and REVIEW-REPORT | parsed-tree comparison against the xmldom files (declaration, prefixes, whitespace declared); markdown as parsed tables |
|
||||
| R5.5 | `upload-sheet-assets` binary (reqwest or ureq, service key from the environment), operator runbook | local stack run, SUPABASE.md §3.1 |
|
||||
| R5.6 | D-R5 switch: fingerprint convention, aliases, the one membership test in the gated readers (your go per file), seed regenerated once by the Rust tool, CHANGELOG | old archives open with no message; a report on an older content revision still rebases; N-1 reader green |
|
||||
| R5.7 | consumer census, `packages/anatomy` and `packages/domain` deleted (`packages/domain/tests/fixtures/report-v2.*.json` moved under `crates/core/tests/fixtures` first, `r4.census.test.ts` xmldom assertion inverted), `@xmldom/xmldom` and `prettier` leave the anatomy lockfile, docs (CLAUDE.md map, WORKSPACE, packages/README, RUST.md §11), ledger of the deleted tests | grep census, `pnpm check:all`, test counts reconciled |
|
||||
|
||||
Order constraint: R5.6 needs the owner's D-R5 decision; R5.2 needs R5.1b.
|
||||
|
||||
## B.6 Test ledger
|
||||
|
||||
`crates/core/tests/curation/LEDGER.tsv` at R5.7: one row per TS case of `packages/anatomy/tests` and `packages/domain/tests` at the freeze commit (file, title, disposition ported / replaced / kept / deleted, target). Counts at the freeze: anatomy 131 cases in 16 files (the R5.1 generator adds 6), domain 262 in 13 files.
|
||||
Reference in new issue
Block a user