docs(docs): RUST.md §10.8 R4.6 record (buffers, memory, parity, E3/E4 three engines, release series, W1, risks); ledger targets filled; compare-exports script; roundtrip raster-failure case; TODO R4.6 closed; memory BDR-062, LRN-063/064, journal

This commit is contained in:
bchanot committed 2026-10-19 02:49:42 +02:00
1 parent cdcdda8e1c
commit 264c5fa3fc
14 files changed
+5230 -38

No files matched your search

+269 -1
View File
@@ -905,7 +905,7 @@ imports it and restyles `data-state` in place as before.
needs it.
- WASM budget: 396 KB of 600 KB used; R4 adds the PDF reader and writer.
## 10. R4: the PDF core in Rust (in progress; R4.1 freeze done 2026-10-18)
## 10. R4: the PDF core in Rust (in progress; R4.1 freeze 2026-10-18, export switched 2026-10-19)
Plan and contract: `.claude/tasks/plans/2026-10-17-rust-r4-pdf-core.md`
(revision 2, validated by the owner 2026-10-18 with the conditions of its
@@ -1612,3 +1612,271 @@ Residual risks and open points (R4.5):
- `MAX_ATTACHMENT_BYTES` keeps the 16 MiB envelope cap; an `exportedAt` that
is not an ISO UTC timestamp is a new closed kind `invalid_input`, never
reachable from the application.
### 10.8 R4.6: the application export on the core's session (2026-10-19, export switched)
What runs where. The core (`crates/core/src/pdf/write/session.rs`, `views.rs`)
holds an `ExportSession`: `new(input)` runs the R4.4 checks, builds the
envelope under 16 MiB and opens the file (header, Info, the three fonts);
`next_step()` answers the rasters the next step needs (`RasterRequest { key,
svg, widthPx, heightPx }`: the SVG logos in slot order, then each observation's
views, main view first, composed by R3's `compose_highlight` at the size
`rasterSizeOf` gives, the O4 cases pinned) or `done`; `add_raster(key, png)`
and `raster_failed(key)` answer one key each; `finish()` lays out the
conclusion page and the footers, writes the pages and the attachment, runs
the self-check, and hands the file out in parts (`take_part()`). The refusals
are closed kinds (`unknown_key`, `duplicate_key`, `out_of_order`), never a
trap. The wasm facade exposes the same struct (`crates/wasm`: `ExportSession`,
outcome envelopes); `packages/core-wasm` wraps it (`CoreExportSession`,
`openExportSession`): every call goes through `CoreRuntime.invoke`, so a trap
poisons the instance as any core call, and a session of a dropped instance
answers `core_unavailable` without touching the dead handle. The application
(`apps/app/src/pdf/export.ts`) drives it: the snapshot from the sheet handle,
the bundled branding, the labels (`labels.ts`, moved verbatim from the engine,
same key order), the clock, then the loop (`browserRasterizer` under
`Promise.all` per batch, `rasterFailed` as each raster rejects, `addRaster` in
request order once the batch settled, as the engine embedded), `finish`, a
`Blob` of the parts, `free()` in `finally`; leaving the editor aborts the
export (the session freed, no file). `EditorRoute` and its alerts are
unchanged. `exportReportToPdf` keeps its signature but returns the `Blob`.
Buffers, ownership and lives (OD-2, the owner's R4.6 item 2):
| Buffer | Owner | Lives | Size |
|---|---|---|---|
| the `ExportInput` JSON | the page (a string), copied into linear memory by the glue for `new()` | the call | report + snapshot + branding + labels (demo 338 KB) |
| `CheckedInput` (report, snapshot with its asset texts, index, the sanitised asset trees, branding, labels) | the session | until `free()` | demo: 3.7 MiB of linear memory after `new()` |
| the envelope text | the session | until the self-check | the attachment's size (demo 55 KB) |
| the file being written | the session's `Objects`: chunks never reallocated, the target 1 MiB doubling to 8 MiB per chunk, an object larger than the target in a chunk of its own, a sealed chunk shrunk to its bytes | until `take_part()` moves each one out (copied into the page, freed in the module) | the file |
| a request's SVG text | the core builds it, the page receives it as JSON | the `next_step()` call (module side), the batch (page side) | 143 KB per view |
| a raster PNG | the page's `Uint8Array`, copied into linear memory by the glue for `add_raster` | the call: decoded (RGBA, raw scanlines, RGB and alpha planes), deflated, written, then freed before the call returns | 0.5 MB (PNG), about 10 MB of transients for a 1200 × 943 raster |
| a logo raster | the session, until its batch completes (the branding embeds in slot order, as the engine did) | the branding step | the four logos |
| the draw log | the session | until `finish()` | the text calls (KB) |
So the module holds, at its highest, the file so far plus one decode plus the
parsed input plus the open chunk's slack; nothing of the rasters once their
object is written, nothing of the file once the parts are out. Measured on a
fresh instance per run, the shipped module in Node, the growth of linear
memory from before `new()` to after `free()` (linear memory never shrinks:
the high-water mark), file in parts:
| Run | views | PDF bytes | parts | linear before | linear after | growth / PDF | ms |
|---|---|---|---|---|---|---|---|
| demo, 2 × 1 stub | 7 | 235,948 | 1 | 1.25 MiB | 5.9 MiB | 20.6 | 52 |
| L, stub | 120 | 386,414 | 1 | 1.25 MiB | 6.6 MiB | 14.6 | 179 |
| CAP, stub | 1,000 | 1,599,445 | 2 | 1.25 MiB | 13.7 MiB | 8.2 | 1,195 |
| demo, real 1200 × 943 | 7 | 3,282,540 | 3 | 1.25 MiB | 18.0 MiB | 5.4 | 499 |
| L, real | 120 | 46,464,887 | 8 | 1.25 MiB | 62.3 MiB | 1.38 | 7,062 |
L with real rasters: 62 MiB held for a 44 MiB file, where the 120 PNGs alone
weigh 64 MB and the engine held every decoded plane until `save()` (plan
§B.4: about 0.45 GB); the plan's estimate for the session was 53 MB (file +
9 MB). What the module holds beyond the file is bounded (the test asserts it
under 24 MiB): the parsed input and trees (3.7 MiB), one decode (about 10
MiB), the open chunk's slack (up to 8 MiB). Stage by stage on L real: 1.3 MiB
at start, 5.0 after `new()`, 15.5 after the second raster (the first decode),
then 0.34 MB per raster for 0.364 MB of file per raster, 62.3 before
`finish()`, 62.3 after, 62.3 after the parts are out and the session freed.
Two findings on the way, measured and kept:
- A single growing `Vec` for the file (amortised doubling) spiked to 125 MiB
on L real: the reallocation holds the old and the new buffer while it
copies, and the decode transients sit between the file and the top of the
heap, so dlmalloc could not grow it in place. Chunks never reallocated
remove the spike.
- Chunks of 1 MiB cost 0.12 MB of fragmentation per view (a long-lived
allocation interleaved with 10 MB of transients every other raster: 74 MiB
for the same file); 8 MiB chunks cost none (62.3 MiB). The target starts at
1 MiB and doubles per chunk up to 8 MiB, so a small file pays little slack
(demo: one chunk).
Declared differences, each one a B2 or B4 class of plan §B.2.3, every O2
comparison and the poppler render re-run unchanged:
- The catalog and the page tree (objects 1 and 2) are written last (the xref
maps them); R4.5 wrote them first. The R4 digests (`goldens/r4/pdf.*.sha256`,
the writer's own bytes, not a TypeScript oracle) were rewritten once for
it, with the O2 suite, qpdf, pdfdetach and poppler (0 pixel) green on the
new layout.
- UTF-16BE hex strings are printed in upper case, as pdf-lib printed them
(R4.5 printed lower case: B4, the same decoded value).
- pdf-writer stays unused (R4.5).
- A view raster the decoder refuses is that view's `view_raster_failed`
warning, as `embedPng` throwing was in the engine; R4.5's one-shot path
made it `self_check_failed`.
- The self-check scans the bytes from the attachment stream object to the end
of its chunk, behind the file header, with the R4 reader: the reader finds
the attachment by scanning its bytes, never by the xref, and the file is
never held whole in the module. The bytes scanned are the file's.
Parity and restoration (cargo `pdf_session`, `pdf_serialize`, `pdf_layout`;
Node `pdf-export.session.test.ts`; the app suites):
| Check | Result |
|---|---|
| the session driven key by key = `export_pdf` one-shot (bytes, envelope, warnings); the file grows as the rasters arrive | demo, equal |
| the 14 O2 runs through the shipped module and `CoreExportSession` | the native digests |
| every view request = the TypeScript `composeHighlightedViewSvg` text of that (observation, view); every (observation, view) the snapshot holds requested once | 14 runs, all views |
| the real-raster run: the two SVGs the R4.1 rasteriser drew (`manifest.json` `svgSha256`, 1200 × 263 and 1200 × 943) | equal, sizes equal |
| `rasterSizeOf`, the O4 cases (JS `Number()` on the viewBox parts, the clamps) | 57 cases equal |
| warnings: a raster that fails before any embed, then a PNG the decoder refuses, view order; a logo warning at its slot whatever the arrival | as the engine recorded them |
| refusals: before `next()`, a key not in the batch, answered twice, `next()` with keys unanswered, `finish()` early or twice, after `free()`; garbage, empty and truncated PNGs | closed kinds, no trap |
| a trap mid-session (proofs build) | the instance poisoned, the session answers `core_unavailable`, `free()` safe, a new session on the recovered instance writes the digest |
| export Rust → import Rust (the module's `import_pdf`, the bundled sheet) | 14 runs: `envelope_v1`, the report equal, not rebased |
| export Rust → the kept TypeScript reader (`importReportFromPdf`, the app suites) → the core | the roundtrip, leaves, N-1 and envelope suites (27 cases): equal reports, the 612 leaves (29 unresolved) kept |
| N-1 (O11 scanner and the Zod copy: the R3 build and 0.2.0) | every R4 PDF of the O2 runs read; a stripped one `attachment_stripped` (R4.5 suite, unchanged) |
| the app path (`exportReportToPdf`, stub rasters, the clock fixed at `FIXED_EXPORTED_AT`) | the 12 R4 digests (3 reports × 2 locales × 2 logo modes) |
| the Step 6 goldens (text with page, position, size and font; images; links) read back from the app path's file | 12 runs equal, in page order (the recorder listed every footer last; the file lists each page's footer after its content) |
| `pdf.roundtrip` (16), `pdf.goldens` (13), `pdf.export.app` (2), `routes.EditorRoute.export` (6, abort on unmount added), `routes.coreStopped` (15), `state.coreFailures` (5), `migration.leaves` (2), `n-1.reader` (6), `archive.envelopes` (3) | green on the Rust export; the suites read the file's content streams (`tests/helpers/pdfRead.ts`) where they spied pdf-lib |
| the oracle generator `r4.writer-oracles` (17) | still verifies the frozen O1 / O2 oracles against the TypeScript engine, called directly with the application's inputs |
The engine leaves the application's export path: `apps/app/src` imports
`@faunosteo/pdf` for the kept reader only (`pdf/import.ts`); pdf-lib is no
longer in the application bundle (index chunk 922 KB). `packages/pdf` stays
until R4.8 as the R3 reference of the oracle suites and the tests' reader.
E3 and E4 in the browsers (`scripts/browser/export-baseline.mjs`, 2026-10-19,
`results/2026-10-19-ts-export.json` on the build of the last TypeScript-export
commit, `results/2026-10-19-r4-export.json` on the R4.6 build; both under the
same harness, the same host, the same evening, demo and L three runs each, CAP
once; the production CSP; `compare-exports.mjs` prints the tables). Time is
the page clock from the click to the PDF blob, the median of the runs:
| Engine | Report | TS | Rust | Rust / TS | TS file | Rust file |
|---|---|---|---|---|---|---|
| Chromium 156 | demo | 1,372 ms | 916 ms | 0.67 | 2,852,004 | 2,823,275 |
| Chromium | L | 19,286 ms | 14,218 ms | 0.74 | 43,893,082 | 43,536,994 |
| Chromium | CAP | stopped by the 4 GiB ceiling after 818 PNGs | 117.6 s | | | 360,935,689 |
| Firefox 157 | demo | 1,559 ms | 1,035 ms | 0.66 | 2,872,189 | 2,843,130 |
| Firefox | L | 24,559 ms | 16,574 ms | 0.67 | 44,105,143 | 43,743,780 |
| Firefox | CAP | stopped after 565 PNGs | 138.0 s | | | 362,665,313 |
| WebKit 27.2 | demo | 2,626 ms | 2,386 ms | 0.91 | 2,493,691 | 2,473,027 |
| WebKit | L | 52,251 ms | 49,538 ms | 0.95 | 38,557,714 | 38,286,426 |
| WebKit | CAP | stopped after 962 PNGs | 412.2 s | | | 317,323,081 |
E3 (Rust ≤ 1.2 × the TypeScript on demo and L in Chromium): 0.67 and 0.74,
met; Firefox 0.66 and 0.67, WebKit 0.91 and 0.95. E4: L completes on the
three engines (it did with the engine); CAP completes on the three (1,001
rasters, a 361 MB file) where the engine was stopped by the harness ceiling
on every engine, as at R4.1. The phone part of E4 is the owner's J2.6.
Where the time goes (medians; head = click to the first raster, raster phase
= first `toBlob` start to last `toBlob` end, tail = last raster to the PDF
blob; the Rust driver's User Timing adds the sum of its core calls and of its
rasteriser awaits):
| Engine | Report | TS head / raster phase / tail | Rust head / raster phase / tail | Rust core calls | Rust raster awaits |
|---|---|---|---|---|---|
| Chromium | demo | 62 / 794 / 518 | 27 / 803 / 80 | 487 | 415 |
| Chromium | L | 66 / 11,880 / 7,381 | 27 / 14,014 / 178 | 7,153 | 7,066 |
| Chromium | CAP | | 57 / 117,093 / 495 | 58,970 | 58,521 |
| Firefox | demo | 62 / 841 / 650 | 20 / 942 / 74 | 455 | 573 |
| Firefox | L | 67 / 14,762 / 9,731 | 25 / 16,382 / 173 | 6,684 | 9,862 |
| Firefox | CAP | | 60 / 137,464 / 474 | 55,024 | 82,803 |
| WebKit | demo | 63 / 2,193 / 370 | 20 / 2,283 / 81 | 455 | 1,918 |
| WebKit | L | 70 / 47,227 / 4,957 | 25 / 49,260 / 251 | 6,733 | 42,693 |
| WebKit | CAP | | 54 / 411,048 / 1,091 | 55,372 | 356,032 |
The engine's tail (its `save()`: every plane deflated at the end) is gone:
the Rust tail is the conclusion page, the pages, the attachment and the
self-check (80 to 250 ms on demo and L). The core's own work (decode, deflate
at level 6, write) is 7 s on L and 59 s on CAP in Chromium, interleaved with
the browser's rasterisation, which is the other half (7 s on L in Chromium,
43 s in WebKit); the export is bound by those two, in sequence per batch.
Memory in the browsers (content process RSS, MiB; growth = settled after −
settled before, forced GC in Chromium, idle elsewhere):
| Engine | Report | TS peak | Rust peak | TS growth | Rust growth |
|---|---|---|---|---|---|
| Chromium | demo | 307 | 264 | 17 | 28 |
| Chromium | L | 805 | 470 | 43 | 92 |
| Chromium | CAP | 3,739 (stopped) | 1,787 | | 560 |
| Firefox | demo | 378 | 291 | 77 | 78 |
| Firefox | L | 1,397 | 819 | 183 | 264 |
| Firefox | CAP | 3,487 (stopped) | 1,914 | | 826 |
| WebKit | demo | 658 | 593 | 277 | 144 |
| WebKit | L | 1,191 | 848 | 210 | 218 |
| WebKit | CAP | 3,871 (stopped) | 2,633 | | 552 |
The peak falls everywhere (L: 805 → 470 MiB in Chromium, 1,397 → 819 in
Firefox, 1,191 → 848 in WebKit). The settled growth after the export is
higher on L and CAP: it is the module's linear memory, which never shrinks
(plan §B.4 "the same high-water, until reload"): the file was held once in
the module (44 MB on L, 361 MB on CAP) plus the transients, and that memory
stays reserved for the page's next export instead of being returned. The
engine's growth was JavaScript heap, collected. On CAP the page keeps about
560 to 830 MiB after the export; CAP is the cap, not a bilan, and the engine
never finished it.
Release across exports (`scripts/browser/r4-export-series.mjs`,
`results/2026-10-19-r4-export-series.json`, `…-ts-export-series-chromium.json`,
`…-r4-export-series-chromium-12.json`): one editor tab, the start level, a
first export (init) and six more, the content RSS read after each (forced GC
in Chromium; idle elsewhere, indicative), the import series' criterion of
10 MiB over the last five:
| Engine | Report | Init | Per export | Last 5 | Peak content |
|---|---|---|---|---|---|
| Chromium, Rust | demo | 28.0 MiB | 1.5 / 2.5 / 2.8 / 2.5 / 3.4 / 1.2 | 12.3 MiB | 287 MiB |
| Chromium, Rust | L | 92.2 | 25.5 / 21.1 / 20.1 / 13.3 / 23.9 / 22.4 | 100.8 | 643 |
| Chromium, TypeScript (control) | demo | 16.1 | 3.1 / 3.3 / 2.1 / 1.6 / 2.2 / 1.6 | 10.9 | 330 |
| Chromium, TypeScript (control) | L | 44.6 | 20.7 / 14.8 / 19.6 / 18.3 / 22.1 / 22.7 | 97.6 | 957 |
| Chromium, Rust, 12 exports | L | 95.2 | 19.7 / 27.3 / 11.1 / 29.5 / 14.0 / 28.4 / 16.7 / 22.6 / 4.4 / 1.5 / 3.4 / −4.3 | 27.6 (the last five) | 686 |
| Firefox, Rust | demo | 95.1 | 93.9 / −6.7 / 103.3 / −50.4 / 96.5 / −41.8 | 100.8 | 596 |
| Firefox, Rust | L | 263.8 | 156.8 / 0.7 / 86.9 / −196.3 / 58.7 / 74.1 | 24.0 | 1,024 |
| WebKit, Rust | demo | 139.8 | 91.5 / 225.9 / −216.8 / 108.3 / −90.4 / −0.3 | 26.9 | 946 |
| WebKit, Rust | L | 326.7 | 153.7 / 14.7 / 1.2 / 1.1 / −0.4 / 2.3 | 19.0 | 1,992 |
Reading: the per-export growth of about 20 MiB on L in Chromium is the
browser's, not the session's: the TypeScript build shows the same figures on
the same page (control), and the module's linear memory, read in Node over
seven L exports with real rasters on one instance, stays at 62.3 MiB from
the first export to the last (the session frees everything at `free()` and
the next export reuses it). Over twelve exports Chromium's growth stops
after the eighth (the last five: 27.6 MiB, then a plateau), which points at
a bounded cache (canvas backing stores or the blob URLs kept 30 s by
`downloadBlob`), not at a leak that grows with use. Firefox and WebKit
sawtooth without a forced GC, as in the import series; WebKit L settles
after the second export. The criterion as written (10 MiB over five) is not
met by either build on L in Chromium; the behaviour is the engine's, the
module flat, the plateau reached. Kept as an observation for the owner
(BDR-058's reserves extend to it), with the phone series J2.7 due.
Release on the other paths (cargo `pdf_session`, Node
`pdf-export.session.test.ts`, the route test): success (every part out,
`free()`), a refused input (no session, the runtime stays `ready`), a raster
that fails (its warning, the export continues), a trap mid-session (the
instance poisoned and recovered, the session answers `core_unavailable`,
`free()` a no-op on the dead handle), a cancelled export (the editor left
mid-way: the signal aborts the driver, `free()` in `finally`, no download,
no alert). Dropping the session, finished or not, frees everything it holds
(Rust ownership); nothing of the file survives `free()` in the module.
Module size (W1): the shipped module with the writer and the session linked,
1,535,289 B raw, 519,730 B gzip -9 (the proofs build 531.6 KB at R4.5 held the
trap and allocation proofs too). Under the 550 KB report line and the 600 KB
budget; from 426 KB at R4.3, +94 KB for the whole writer. `DEPLOY.md`'s
served-size note is updated at R4.8 with the final module.
Residual risks and open points (R4.6):
- The settled growth after a large export is the module's linear memory
(the file once, the transients, the chunk slack), kept until reload: 92
MiB after L and 560 MiB after CAP in Chromium. Within the plan (OD-2: "the
same high-water, until reload"); the disposable export worker (S2) stays
the fallback if a device fails L (J2.6).
- The export is bound by the browser's rasterisation and the core's decode
and deflate (level 6) in sequence per batch: L takes 14 s in Chromium and
50 s in WebKit (the engine: 19 and 52). A lower deflate level or
overlapping the next batch's rasterisation with the core's writing would
cut it, at the price of E2 or of a second copy in flight; not done.
- The per-export RSS growth in Chromium is the browser's and plateaus; the
10 MiB criterion is not met by either build on L. Owner decision.
- `view_raster_failed` now also covers a PNG the decoder refuses and a view
that cannot be composed; a raster that never settles still hangs the
export (the engine's residual, unchanged; the abort on leaving the editor
frees the session).
- The headless runs validate nothing on a device: J2 (`R4_J2.md`) on the
owner's computer and iPhone, with the J1 import recette still due.