docs(docs): RUST.md §10.10 R4.8 record (deleted, replaced, kept with reasons, census, non-regression table, risks, R5 prerequisites); CLAUDE.md, README, WORKSPACE, packages/README, TEST_STRATEGY, PDF-FORMAT, DEPLOY rollback R4 note and module size, hardening §5 pdf-lib closed, fixtures README, CHANGELOG R4 entry; TODO R4.8 closed; memory BDR-064, journal
This commit is contained in:
1 parent
e88769a707
commit
f36dbdd4ae
13 files changed
+238
-62
No files matched your search
+27
-20
@@ -1,9 +1,11 @@
|
||||
# PDF format and report archive
|
||||
|
||||
A Faunosteo PDF is both a document for the reader and an archive of the
|
||||
bilan. The engine lives in `packages/pdf` (`@faunosteo/pdf`) and renders from
|
||||
explicit inputs only: no network, no browser DOM, no access to the current
|
||||
catalog. The attachment carries everything a future reader needs to
|
||||
bilan. The engine lives in the Rust core (`crates/core/src/pdf`, reached
|
||||
through the wasm module in the browser and natively by the tools; the
|
||||
TypeScript `packages/pdf` it reproduced byte for byte was deleted at R4.8,
|
||||
`docs/development/RUST.md` §10) and renders from explicit inputs only: no
|
||||
network, no browser DOM, no access to the current catalog. The attachment carries everything a future reader needs to
|
||||
re-render the bilan when the anatomical sheet, the catalog, the labels or the
|
||||
branding have changed since the export. Design record:
|
||||
`docs/development/STEP6_PDF_ARCHIVE_ANALYSIS.md`.
|
||||
@@ -19,8 +21,9 @@ branding have changed since the export. Design record:
|
||||
asset), then zones, findings grouped by finding zone, free findings,
|
||||
severity, comment.
|
||||
- Conclusion page: conclusion, recommendations, next appointment.
|
||||
- Fonts: pdf-lib StandardFonts Helvetica, Helvetica-Bold (body) and
|
||||
Times-Roman (headings), WinAnsi encoding. Every printed string goes through
|
||||
- Fonts: the standard Type1 fonts Helvetica, Helvetica-Bold (body) and
|
||||
Times-Roman (headings), WinAnsi encoding, measured with the AFM widths
|
||||
pdf-lib 1.17.1 used (`crates/core/src/pdf/write/afm_data.rs`). Every printed string goes through
|
||||
`safeText`: control characters are stripped and any glyph WinAnsi cannot
|
||||
encode becomes `?` (en and em dashes, accented letters, `•`, `·` are fine;
|
||||
`⚠` is not, hence the `(!)` vet-referral marker).
|
||||
@@ -44,7 +47,7 @@ branding have changed since the export. Design record:
|
||||
|
||||
One native embedded file (`EmbeddedFiles`, `AFRelationship /Data`) named
|
||||
`osteo-report.json`, MIME `application/json`, description
|
||||
`Faunosteo report archive v1`, Flate-compressed by pdf-lib. Pretty-printed
|
||||
`Faunosteo report archive v1`, Flate-compressed (zlib level 6). Pretty-printed
|
||||
JSON, camelCase keys (the sheet document inside keeps its own snake_case
|
||||
schema):
|
||||
|
||||
@@ -227,15 +230,16 @@ The same caps are checked at export, so an export can never write a file its
|
||||
own reader refuses.
|
||||
|
||||
The branding URL rule exists because a URL is written into every page's
|
||||
link action: pdf-lib does not escape a literal string, so the engine writes
|
||||
the URI as a hex string and the schema refuses the characters that could
|
||||
close a literal. The export re-validates the branding and the labels it
|
||||
link action: a literal string is not escaped (pdf-lib did not, the core
|
||||
does not either), so the engine writes the URI as a hex string and the
|
||||
schema refuses the characters that could close a literal. The export re-validates the branding and the labels it
|
||||
receives (`ArchiveExportError('branding_invalid')` before any page is
|
||||
drawn), even from an archive that passed the reader. The URI itself is the
|
||||
hex form of the URL's ASCII bytes (`<68747470…>`), not UTF-16 with a BOM,
|
||||
so readers that take `/URI` as raw bytes (poppler) follow it. A PNG logo is
|
||||
checked before pdf-lib's decoder (UPNG) sees it, because that decoder trusts
|
||||
its input: it sizes its bitmap from whatever IHDR claims, its `inflateRaw`
|
||||
checked before the decoder sees it (the core's decoder reproduces UPNG's
|
||||
acceptance, the decoder pdf-lib used, so that an archived logo decodes to
|
||||
the same planes); the gate exists because that decoder trusted its input: it sizes its bitmap from whatever IHDR claims, its `inflateRaw`
|
||||
never checks the input bound (a truncated or crafted IDAT stream makes it
|
||||
read past the buffer and never return), and it scans text chunks for a NUL
|
||||
byte that may be missing. The gate therefore validates, in order: the
|
||||
@@ -247,10 +251,11 @@ sizes are not computed); an expected raw scanline size,
|
||||
`(ceil(width × bits per pixel / 8) + 1) × height`, of at most 16 MiB, the
|
||||
attachment cap the native inflater enforces (4096² at 8-bit RGBA is 67 MB
|
||||
of scanlines and is refused; a 512 px wide logo is under 1 MB). That cap
|
||||
bounds what is inflated, not what pdf-lib's decoder then allocates: the
|
||||
decoder still expands any accepted image to a 32-bit RGBA buffer, so the
|
||||
4096² pixel cap is what bounds memory (about 430 MB peak measured with four
|
||||
4096² logos; real logos are 512 px wide). Then the chunk list, which must end on an empty `IEND`
|
||||
bounds what is inflated, not what the decoder then allocates: the decoder
|
||||
still expands any accepted image to a 32-bit RGBA buffer, so the 4096² pixel
|
||||
cap is what bounds memory (about 430 MB peak measured with four 4096² logos
|
||||
on the TypeScript engine; real logos are 512 px wide; the core frees the
|
||||
planes as soon as the image object is written). Then the chunk list, which must end on an empty `IEND`
|
||||
that closes the file and may hold only the chunk types the decoder reads by
|
||||
fixed offsets or by declared length (`PLTE`, `tRNS`, `IDAT`, `IEND`,
|
||||
`gAMA`, `cHRM`, `sRGB`, `sBIT`, `pHYs`, `bKGD`): a second IHDR, an APNG
|
||||
@@ -271,11 +276,13 @@ fallback text and the export reports `logo_failed`.
|
||||
reader in a worker since R4.3) and the `report:compare` tool (the same
|
||||
reader, natively, since R4.7) read the attachment by byte scan, so a
|
||||
crafted object stream is never inflated (tested with a 64 MiB bomb:
|
||||
milliseconds, no memory growth). pdf-lib stays on the export side, on trusted inputs, and
|
||||
in tests that inspect exported files. What remains before external users
|
||||
(`docs/development/PDF_IMPORT_HARDENING.md` §5): a Worker or server-side
|
||||
budget for any path that still parses PDF objects, the pdf-lib dependency
|
||||
decision, and read-side caps on report collections.
|
||||
milliseconds, no memory growth). No PDF library remains in the repository
|
||||
since R4.8: the core writes the files, the tests read them back with a
|
||||
reader of the writer's own format (`apps/app/tests/helpers/pdfFile.ts`).
|
||||
What remains before external users
|
||||
(`docs/development/PDF_IMPORT_HARDENING.md` §5): the server-side budget
|
||||
decision for any future path that parses PDF objects, and read-side caps
|
||||
on report collections.
|
||||
- The scanner reads what is in clear: an attachment whose stream dictionary
|
||||
is malformed beyond `/Length` and `endstream`, or an encrypted file, yields
|
||||
no candidate (`no_attachment`), as the previous reader did in practice.
|
||||
|
||||
@@ -110,9 +110,11 @@ hides its `/Keywords`, so a stripped attachment on such a file reads as
|
||||
brackets deeper than 64 levels (linear byte pre-scan, strings skipped).
|
||||
- Option C or a server-side import (Step 14) with a hard memory and time
|
||||
budget per file, for any path that still parses PDF objects.
|
||||
- Dependency decision on pdf-lib (unmaintained since 2021): keep it for
|
||||
generation only, or move to the maintained fork; recorded as a Step 14
|
||||
question.
|
||||
- Dependency decision on pdf-lib (unmaintained since 2021): closed at R4.8
|
||||
(2026-10-19): pdf-lib left the repository with the TypeScript engine; the
|
||||
core's writer (`RUST.md` §10.7, §10.8) generates the PDFs and its reader
|
||||
reads them; no PDF library remains in the application, its tests or the
|
||||
tools (`apps/app/tests/unit/r4.census.test.ts` fails on a return).
|
||||
- Report collection caps on read (observations, zones, findings per
|
||||
observation) so a hostile report cannot freeze the editor before the
|
||||
export's own caps refuse it.
|
||||
|
||||
@@ -1960,3 +1960,134 @@ Residual risks and open points (R4.7):
|
||||
- A PDF over 50 MiB is refused by the entry check (`file too large`) before
|
||||
the reader, as before; the reader's own cap would say `not a readable
|
||||
PDF`.
|
||||
|
||||
### 10.10 R4.8: the TypeScript PDF engine deleted (2026-10-19)
|
||||
|
||||
Audit before every deletion (owner's R4.8 brief §2): every importer of
|
||||
`@faunosteo/pdf`, `pdf-lib`, `@pdf-lib/*`, `@resvg/resvg-js`, `@xmldom/xmldom`
|
||||
and `@faunosteo/domain` was listed across the application, the glue package,
|
||||
the tests, the scripts, the pipeline, the workspace files, the Dockerfile,
|
||||
the CI files and the docs (`grep` census, now the `r4.census` suite). The
|
||||
application's last consumer of the engine (the TypeScript import path,
|
||||
tests only since R4.3) and the glue's (the session suite's compose check,
|
||||
the engine comparison) were moved onto the core or deleted first; the
|
||||
pipeline's consumers (`packages/anatomy/src/curation/app-review.ts` →
|
||||
`render/compose.ts`, 22 files → `@faunosteo/domain`) stay until R5.
|
||||
|
||||
Deleted (70 files, 12,010 lines of TypeScript):
|
||||
|
||||
- `packages/pdf/**` (3,122 lines of engine, 8,211 of tests): the export,
|
||||
the layered reader, the byte-scan attachment reader, the SVG sanitiser and
|
||||
the view composition, the page renderers, the branding gate, the labels,
|
||||
the 140 engine cases and the R3 and R4 oracle generators
|
||||
(`r3.{highlight,sanitiser}-oracles`, `r4.{dates,inflate,png,reader,text}-oracles`)
|
||||
with their helpers (`resvg`, the reader-input crafters).
|
||||
- `apps/app/scripts/compare-reports.ts` and `compare/{report,sheet}.ts`
|
||||
(589 lines, the TypeScript CLI replaced at R4.7), `tests/unit/scripts.compare.test.ts`.
|
||||
- `apps/app/src/pdf/import.ts` (the TypeScript scanner behind
|
||||
`importReportFromPdf`, tests only since R4.3) and `tests/unit/pdf.import.test.ts`.
|
||||
- `tests/unit/pdf.export.app.test.ts` (T2 and scenario 3 through the
|
||||
engine's `exportFromEnvelope`: the re-export from the envelope alone is
|
||||
cargo's `export_from_envelope`, whose signature takes no sheet and no
|
||||
branding, byte-equal to the R4 files on 13 runs).
|
||||
- The application's generators `r4.{compare,tolerance,writer}-oracles` and
|
||||
`tests/helpers/r4-draw.ts` (the pdf-lib draw recorder); the glue's
|
||||
`pdf-export.ts-engine.test.ts` (the Node timings against the engine,
|
||||
recorded in §10.7).
|
||||
- Dependencies: `@faunosteo/pdf` (app, core-wasm), `pdf-lib`, `tsx` and
|
||||
the app's `report:compare` script (app), `@faunosteo/domain` (app: its
|
||||
two incidental test uses moved to `@faunosteo/core-wasm/types` and the
|
||||
N-1 Zod copy), the Dockerfile's `COPY packages/pdf/package.json`.
|
||||
`pnpm-lock.yaml` holds no `pdf-lib`, `@pdf-lib/*` or `@resvg/resvg-js`
|
||||
any more; `@xmldom/xmldom` only under `packages/anatomy`.
|
||||
|
||||
Replaced in the tests (no case ignored, no golden regenerated):
|
||||
|
||||
| Before | After |
|
||||
|---|---|
|
||||
| pdf-lib parsed the app export's file (texts, images, page count, Info, Filespec) | `tests/helpers/pdfFile.ts`, a reader of the writer's own format (objects, dictionaries, Flate streams through `node:zlib`, pages in `/Kids` order, Info, annotations, XObjects); `pdfRead.ts` keeps its content-stream parser on it |
|
||||
| pdf-lib read the links of the file | the same reader's annotations (`goldens.ts` `readLinks`) |
|
||||
| pdf-lib built the foreign PDFs of the rejection cases | `buildPdfWithAttachment` (a minimal file with one embedded file in clear) and the frozen hostile fixtures |
|
||||
| pdf-lib's `embedPng` checked the golden rasters | the XObject sizes read from the app export's file |
|
||||
| the engine composed the views of the `views.*.json` goldens | the SVGs the core's session asks the rasteriser for, captured at the mocked rasteriser in request order |
|
||||
| the engine's scanner read the attachment of the envelope and N-1 suites | the frozen O11 scanner, now exported by `@faunosteo/core-wasm/n-1` (`readAttachmentBytes`, `readAttachedDocument`): the R3 build's reader, as a rollback would read |
|
||||
| `importReportFromPdf` (TypeScript scanner + core) | `tests/helpers/pdfImport.ts`: the module's `SheetHandle.import_pdf` on a handle of the bundled sheet, the envelope mapped by the worker's `pdfImportOf` (the same kinds HomeRoute shows); the main-thread session still never calls the reader (OD-1) |
|
||||
| the session suite compared each request SVG with the engine's `composeHighlightedViewSvg` | structural facts per request (the view's frame, the highlight, no editor attribute) and the real-raster manifest digests; the text itself is cargo's (`pdf_session` against `compose_highlight`, `anatomy_compose` against the frozen R3 trees) |
|
||||
|
||||
Still TypeScript, with the reason (owner's R4.8 brief §6.4):
|
||||
|
||||
- `packages/anatomy` (the curation pipeline, the legacy importer, the frozen
|
||||
renderer `render/` and `@xmldom/xmldom`): R5's scope, consumed by
|
||||
`pnpm import:equine` and the seed golden; no app and no PDF consumer.
|
||||
- `packages/domain`: the pipeline's schemas until R5; the application and the
|
||||
glue no longer depend on it.
|
||||
- `packages/core-wasm/tests/n-1/{domain,scanner}`: the frozen N-1 readers
|
||||
(the R2 Zod schemas, the R3 byte-scan reader), never edited, permanent.
|
||||
- `apps/app/src/pdf/{export,labels,branding,browserRasterizer}.ts`: the
|
||||
application's side of the export (input assembly, the session driver, the
|
||||
browser's rasteriser), UI glue by the review's boundary.
|
||||
- `apps/app/src/core/worker/**`: the import worker (protocol, timers,
|
||||
outcome mapping), browser glue.
|
||||
- The test readers above (`pdfFile.ts`, `pdfRead.ts`, `pdfImport.ts`):
|
||||
tooling on the writer's own files, never product logic.
|
||||
- `scripts/r4-afm-data.mjs`: the generator of `afm_data.rs` from the frozen
|
||||
O4/O5 oracles (no pdf-lib: reads `afm.json` and `text.json`), kept so the
|
||||
tables stay reproducible.
|
||||
|
||||
Census and gates: `apps/app/tests/unit/r4.census.test.ts` (8 checks) fails
|
||||
on a reintroduced import of the engine, pdf-lib, the TS domain or xmldom
|
||||
under `apps/app/src` or `packages/core-wasm/src`, on a dependency added back,
|
||||
on a lockfile holding pdf-lib, on an export that does not open the core's
|
||||
session, on an import that does not go through the worker, on a root
|
||||
`report:compare` that is not the binary, and on a missing N-1 reader or
|
||||
oracle folder. The ledger (`scripts/r4-ledger.mjs`) now reads its 301 frozen
|
||||
cases from the committed TSV (the deleted suites cannot run), re-derives the
|
||||
dispositions and checks every `cargo` target against `cargo test --list`
|
||||
and every `vitest (same case…)` target against the live report of the
|
||||
surviving suites: 150 ported, 29 replaced, 89 kept, 33 deleted, 0 pending.
|
||||
Test counts after R4.8: apps/app 57 files, 427 cases (460 before: 42 of the
|
||||
deleted suites, 1 roundtrip case added, 8 census); packages/core-wasm 58
|
||||
(60); packages/pdf 146 → 0; cargo unchanged (R4.7: the compare suite
|
||||
included). `pnpm typecheck`, `pnpm -r lint`, `pnpm check` and
|
||||
`pnpm rust:check` green; `pnpm check:all` (the app image, the worker test
|
||||
stage and image) recorded below.
|
||||
|
||||
Non-regression (owner's R4.8 brief §3), each with its proof:
|
||||
|
||||
| Guarantee | Proof |
|
||||
|---|---|
|
||||
| the application imports through the core's reader | `r4.census` (HomeRoute → `core/worker`, `handler.ts` → `import_pdf_bytes`, no scanner identifier under `src`); the route and worker suites (`routes.HomeRoute.{import,archive,rescueCopy}`, `core.importWorker`) |
|
||||
| the export runs on the core's session | `r4.census` (`export.ts` → `openExportSession`); `pdf.goldens`: app-path bytes = the R4 digests at the fixed time |
|
||||
| `report:compare` is the binary | `r4.census` (root script → `scripts/report-compare.sh` → `--features cli --bin report-compare`); cargo `pdf_compare` spawns it |
|
||||
| v2 and v3 archives stay compatible | `pdf.roundtrip` (the frozen v2 PDFs migrated, unresolved items flagged), `migration.leaves` (612 leaves exported and read back through the module, 29 unresolved kept), cargo `pdf_import` O7/O8 |
|
||||
| historical PDFs stay readable | cargo `pdf_import` and `pdf_serialize::restoration_round_trip_loses_nothing` on the Step 6 and R3 fixtures; `pdf.goldens` on `report-v3.bare.pdf`; the N-1 suites on the R4 files |
|
||||
| the validated renders stay conformant | `pdf.goldens` (Step 6 text, images, links read from the file; the R4 digests), cargo `pdf_layout` (O1), `pdf_serialize` (O2, poppler 0 pixel recorded at R4.5) |
|
||||
| rescue copies and data-loss protections | `routes.HomeRoute.rescueCopy` (13), `routes.coreStopped` (15), `state.coreFailures` (5), the quarantine suites, unchanged |
|
||||
| no TypeScript engine active in parallel | `r4.census`; the only TypeScript readers left are the frozen N-1 copies, read by tests only |
|
||||
|
||||
Residual risks and open points (R4.8):
|
||||
|
||||
- The test reader `pdfFile.ts` reads the writer's own format only (one
|
||||
dictionary per line, direct `/Length`, classic xref): a change of the
|
||||
writer's layout changes it too; it is tooling, pinned by the suites that
|
||||
use it.
|
||||
- `tests/helpers/pdfImport.ts` runs the module's reader on the test
|
||||
process's main instance (a throwaway handle, freed): the application never
|
||||
does (OD-1); the worker path keeps its own suites.
|
||||
- A stale `packages/pdf/node_modules` folder (git-ignored symlinks of the
|
||||
previous install) may linger on a development machine after the pull:
|
||||
`rm -rf packages/pdf` by hand; the census checks the package files, not
|
||||
the folder.
|
||||
- The open points of §10.8 stand: the Chromium per-export growth (BDR-063,
|
||||
an open reserve), J1 and J2 on real devices before any merge, the WebKit
|
||||
reserves BDR-058/059, the hostile-file protections (`PDF_IMPORT_HARDENING`
|
||||
§5 gate open), the module at 519,795 B gzip of a 600 KB budget. R4.8
|
||||
measured nothing new on memory or time and claims nothing.
|
||||
|
||||
R5 prerequisites (the pipeline port): `packages/anatomy` (curation pipeline,
|
||||
importer, `render/`, the review drawings) and `packages/domain` are the last
|
||||
TypeScript business packages; their oracles (the seed golden, the 612-leaf
|
||||
semantics, the determinism and count suites, the R3 layer oracle) are in
|
||||
place; `documentSha256` will change once (D-R5) and every older archive will
|
||||
rebase on import; `@xmldom/xmldom`, `tsx` and `prettier` on generated files
|
||||
leave with them.
|
||||
@@ -15,7 +15,7 @@ Status: Step 0 deliverable, 2026-10-08. Current state: 32 files under
|
||||
| RLS | pgTAP, per table, per role | `supabase/tests/rls` | CI, release gate |
|
||||
| End-to-end | Playwright against `pnpm preview` + local Supabase | `apps/app/e2e` | CI nightly + release gate |
|
||||
| Migration | `supabase db reset` from zero + seed + regression fixtures | CI | every PR touching `supabase/` |
|
||||
| PDF regression | vitest golden tests: text extraction of generated PDFs, attachment roundtrip, raster hash on fixed inputs | `packages/pdf/tests` | CI |
|
||||
| PDF regression | cargo: the R4 oracles (O1 draw logs, O2 PDFs, O4 to O10) and the R4 digests of the writer's own bytes; vitest: the Step 6 goldens read back from the app export's file (`tests/helpers/pdfFile.ts`, no pdf-lib), the attachment roundtrip through the module's reader, the N-1 readers | `crates/core/tests/pdf_*.rs`, `apps/app/tests/unit/pdf.*.test.ts`, `packages/core-wasm/tests/pdf-*.test.ts` | CI |
|
||||
| SQL/TS contract | generated types diff check | CI | every PR touching `supabase/` |
|
||||
| Rust core and facade | `cargo test` (unit + fixture tests reading the repository JSON in place), `clippy -D warnings`, `cargo deny`; ts-rs types drift check | `crates/*`, `pnpm rust:check` (native, `scripts/rust-check.sh`; same file in the builder image), `pnpm check:all` | before every Rust commit and every merge; CI job `rust` prepared, no runner yet |
|
||||
| Rust/TS contract | vitest on the built module: byte equality with `JSON.stringify(…, null, 2)` on the real documents, ts-rs types against the Zod output types, the runtime (lazy load, trap → poison → offline re-instantiation, handle rebuild, cap, retry), panic and stack traps on `pkg-proofs`, served size budget and measurements | `packages/core-wasm/tests` (refused, never skipped, when `pkg/` or `pkg-proofs/` is missing or stale) | `pnpm check:all` (after the native `wasm:build`); CI `check` prepared |
|
||||
@@ -78,8 +78,10 @@ Target after Step 1: roughly 230–250 cases, all current ones untouched.
|
||||
- `supabase/seed/test/` provides two practitioners, one cabinet, two clients,
|
||||
three animals, the official sheet, one personal sheet, reports in each
|
||||
status. RLS tests reuse it.
|
||||
- Golden PDFs are compared by extracted text and attachment JSON, not by
|
||||
bytes (pdf-lib dates and ids vary); raster hash only on fixed-size inputs.
|
||||
- Golden PDFs are compared by the text, images and links read back from the
|
||||
file and by the attachment JSON; the writer's own bytes are pinned by the
|
||||
R4 digests at a fixed export time (`goldens/r4/pdf.*.sha256`), rewritten
|
||||
only for a declared structural change.
|
||||
|
||||
## 5. Gates before a merge, and the CI target
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ Future units, created only by the step that needs them:
|
||||
|---|---|---|
|
||||
| `apps/public-web` | SSR public site (landing, practitioner profiles, search) | plan Step 22 |
|
||||
| `crates/worker` | Rust job runner (avatars at R1, then emails, expiry, calendar sync, imports) — created at R0, replaces the planned `apps/worker` | R0 (2026-10-12) |
|
||||
| `packages/*` | shared domain, anatomy, pdf, i18n, supabase, config, ui | see `packages/README.md` |
|
||||
| `packages/*` | shared domain, anatomy, core-wasm, supabase (i18n, config, ui planned; `pdf` deleted at R4.8) | see `packages/README.md` |
|
||||
|
||||
Rust workspace (plan R0, `docs/development/RUST.md`): `Cargo.toml` at the
|
||||
root with `crates/core` (library), `crates/wasm` (browser facade, built into
|
||||
|
||||
Reference in new issue
Block a user