From 77e11a53e42825bfb42ddabfdbf1bbd47c590851 Mon Sep 17 00:00:00 2001 From: bchanot Date: Mon, 19 Oct 2026 11:13:42 +0200 Subject: [PATCH] =?UTF-8?q?docs(docs):=20RUST.md=20=C2=A710.9=20R4.7=20rec?= =?UTF-8?q?ord=20(tool=20on=20the=20core,=20O10=20parity=20158/72,=20A.3?= =?UTF-8?q?=20lines=20as=20printed,=20timing),=20CLAUDE.md=20map=20and=20c?= =?UTF-8?q?ommand,=20PDF-FORMAT;=20TODO=20R4.7=20closed;=20memory=20BDR-06?= =?UTF-8?q?3,=20journal?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .claude/memory/decisions.md | 6 +++ .claude/memory/journal.md | 1 + .claude/tasks/TODO.md | 6 ++- CLAUDE.md | 8 ++-- docs/PDF-FORMAT.md | 14 ++++--- docs/development/RUST.md | 82 ++++++++++++++++++++++++++++++++++++- 6 files changed, 106 insertions(+), 11 deletions(-) diff --git a/.claude/memory/decisions.md b/.claude/memory/decisions.md index 01210557..b97d1f00 100644 --- a/.claude/memory/decisions.md +++ b/.claude/memory/decisions.md @@ -475,3 +475,9 @@ Status: active. Context: owner decision 2026-10-19 on the R4.5 report (RUST.md §10.7). R4.6 brief: ExportSession + app driver, rasters as bytes one by one, OD-2 buffers documented, `view_raster_failed` on raster failure, parity + restoration (Rust → Rust, Rust → TS reader, N-1), E3/E4 three engines same data, shipped W1 measured, TS engine kept until R4.8, no merge, stop at R4.6. Decision (design, measured): session in main instance per OD-2. File written in chunks never reallocated (target 1 MiB doubling to 8 MiB), image object written at `add_raster` (PNG + planes freed in the call), logos kept until branding batch completes (slot order as engine), catalog + page tree objects written last, file handed to JS in parts → `new Blob(parts)`, self-check scans attachment chunk behind header. R4 digests (own bytes, not TS oracle) re-frozen once for it; O2 suite + qpdf + poppler re-run green. Hex strings upper-case as pdf-lib (B4). Alternatives measured and dropped: single `Vec` doubling (125 MiB for 44 MiB file, realloc spike), two buffers (2 × file at finish), 1 MiB fixed chunks (0.12 MB fragmentation per view). Status: active; owner checkpoint R4.6 pending. + +## BDR-063 | 2026-10-19 | R4.6 accepted provisionally; R4.7 report-compare authorised; Chromium 10 MiB series = open reserve + +Context: owner decision 2026-10-19 on the R4.6 report (RUST.md §10.8). Chromium per-export RSS growth on L (≈ 20 MiB, same in TS control, plateau after 8) accepted provisionally: not a leak validation, kept as open reserve; reports must split wasm linear memory / browser global / canvas-blob-URL retention / transient vs persistent; thresholds never changed, no FAIL turned PASS. J1 + J2 on real devices mandatory before any merge; NON TESTÉ kept. R4.7 per plan, no scope widening; S2 export worker not engaged without proof + new decision. +Decision (R4.7 design): `pdf::compare` pure module behind feature `cli` (= pdf-read + pdf-write, never in wasm), `Files` trait for fs; binary `src/bin/report_compare.rs` only std::fs file; seed sheet read at run time from `CARGO_MANIFEST_DIR/../../supabase/seed/.../draft` (no drawing); root `report:compare` → `scripts/report-compare.sh` (rust-env + cargo run --release --features cli). A.3 closed lines chosen: `report document invalid: ` (ZodError dumps), `attachment refused: lone UTF-16 surrogate escape at byte N` (surrogates kept/overwritten), `report version null cannot be read` (overflow → null, incl. envelopeVersion overflow where TS printed a rejected-envelope header). Alternatives dropped: `include_str!` of the seed (bakes content, rebuild on seed change); spawning the binary per O10 row in tests (230 spawns, slow): in-process `run` + 8 spawns instead. +Status: active; owner checkpoint R4.7 pending. diff --git a/.claude/memory/journal.md b/.claude/memory/journal.md index 7ed7ca7f..57853ad8 100644 --- a/.claude/memory/journal.md +++ b/.claude/memory/journal.md @@ -293,3 +293,4 @@ Owner validated R4.0 plan with conditions (BDR-057, plan §A.6, LRN-059/060). R3 - 2026-10-18 owner validated R4.4; R4.5 go (BDR-061). Subtasks R4.5a-f in TODO. Tooling present: qpdf 12.3.2, poppler 26.01 (pdftoppm, pdfdetach), crates.io reachable. - 2026-10-18 R4.5 done (d420f67..b44dd80): content emitter (60 streams byte-equal, numberToString 703 + V8 pow10 quirk), PNG gate + UPNG decoder (O6 108/108), own object writer (pdf-writer dropped: numberToString everywhere, no dep), serialiser, export + self-check, from_envelope; O2 14 runs equal (streams after B3, planes, links, Info, attachment), qpdf clean, poppler 0 px, O11 + N-1 + TS reader read R4 PDFs, wasm = native digests; Node TS→Rust demo 198→36 ms, L 772→57, CAP 5.7 s→0.22 s; writer linked 531.6 KB gzip (proofs build), shipped 426 KB. Gate PASS 4 LOW → hardenings. Open: one-shot path holds PNGs twice (R4.6 session must stream), L real rasters 5 s native (deflate 6), E3 judged at R4.6. STOP before R4.6. - 2026-10-19 owner validated R4.5; R4.6 go (BDR-062). R4.6 done (9e688b6..): ExportSession (decode on arrival, chunked never-realloc file 1→8 MiB, catalog last, parts → Blob, self-check on attachment chunk), views.rs (compose_highlight requests, raster_size_of O4 57/57), wasm ExportSession + CoreExportSession, app export.ts on session (abort on unmount, User Timing), labels.ts; app suites read content streams (pdfRead.ts), app-path bytes = R4 digests (re-frozen: object order B2, hex upper-case B4), Step 6 goldens via app path, 14 O2 runs through module = digests, request SVG = TS compose text, export Rust → import Rust / TS reader / N-1 ok. E3 Chromium demo 0.67, L 0.74 (Firefox 0.66/0.67, WebKit 0.91/0.95); E4 CAP completes 3 engines (TS stopped by ceiling); peaks lower (L Chromium 805→470 MiB). Linear memory L real 62 MiB flat over 7 exports; Chromium per-export RSS growth ≈ 20 MiB on L same as TS control, plateau after 8. Shipped 519.7 KB gzip (W1). rust-check 0, pnpm check 1,099 tests, ledger 11 pending (R4.7). J2 kit + procedure. STOP before R4.7. +- 2026-10-19 owner accepted R4.6 provisionally (Chromium series = open reserve); R4.7 go (BDR-063). R4.7 done: `pdf::compare` (feature `cli`), `report-compare` bin, `scripts/report-compare.sh` behind root `report:compare`, cargo `pdf_compare` 15 tests: O10 230 rows (158 exact, 72 declared A.3 → 62 `report document invalid`, 7 surrogate refusals, 3 `version null`), 11 scenario cases = ledger targets (301 rows, 0 pending), binary spawned 8×. Release binary 7 ms/PDF, 6 MiB RSS; module unchanged 1,535,289 B / 519,795 gzip. RUST.md §10.9, CLAUDE.md, PDF-FORMAT. STOP before R4.8. diff --git a/.claude/tasks/TODO.md b/.claude/tasks/TODO.md index aa286f52..c2a411bb 100644 --- a/.claude/tasks/TODO.md +++ b/.claude/tasks/TODO.md @@ -250,7 +250,11 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo - [x] R4.6f browsers (scripts/browser): Rust vs TS composed-view rasters pixel-identical (Chromium, Firefox, WebKit); E3/E4 harness demo/L/CAP, same data, TS vs Rust: total, raster, serialise, linear memory, JS heap, RSS; release proof on success / raster error / cancel (unmount) / recovery (trap mid-session); W1 shipped module with the writer linked (gzip -9; report > 550 KB) - [x] R4.6g docs: RUST.md §10.8 (buffers ownership and lifetimes, allocations, release proofs, E3/E4 tables, W1, deviations: catalog at the end, decode at arrival, logo batch order), DEPLOY.md if the module size line moves, fixtures README, ledger rows (pdf.goldens, pdf.roundtrip targets filled), memory (BDR, LRN, journal) - [x] R4.6h (done 2026-10-19; STOP before R4.7) owner recette procedure (visual + functional on the owner's devices: FR/EN, logos, annotations, real rasters, links, L on the phone), report, STOP before R4.7 - - [ ] R4.7 report-compare in Rust + - [ ] R4.7 report-compare in Rust (owner go 2026-10-19, R4.6 accepted provisionally; stop before R4.8) + - [x] R4.7a core `pdf/compare/{mod,lines,run}.rs` behind feature `cli` (= pdf-read + pdf-write): `esc` on the V8 `[\p{Cc}\p{Cf}\u2028\u2029]` table (`SAFE_TEXT_DROPPED`), `Comparison` (header, v3 and migrated observation lines, footer, normalized, counts), `render_comparison` / `render_summary` / `diff_lines`, `parse_args`, `run(sheet, argv, &dyn Files)` → stdout, stderr, exit code; closed failure wording (A.3): `report document invalid: ` for the Zod dumps, `attachment refused: lone UTF-16 surrogate escape at byte N` for the kept surrogates, `report version null cannot be read` for an overflow read as null; `as_str` on Severity, SeverityV2, LateralSide, SnapshotRejection + - [x] R4.7b `src/bin/report_compare.rs` (feature `cli`, the only std::fs file): seed sheet from `supabase/seed/anatomical-sheets/equine/draft` (ref, document, map; no asset), arguments relative to `INIT_CWD`, folders expanded, size pre-check, exit code; `scripts/report-compare.sh` (rust-env + cargo run --release --features cli) behind the root `report:compare`; rust-check feature lists widened to `cli` + - [x] R4.7c cargo `tests/pdf_compare.rs`: the 230 O10 invocations in-process (158 undeclared rows exact: stdout, stderr, exit; 72 declared rows against the Rust wording per A.3 class), scenario PDF digests = `inputs`, the 11 scenario cases named for the ledger, one spawn of the real binary (usage, INIT_CWD, output = in-process); ledger rows 135–145 targets filled, `scripts/r4-ledger.mjs` regenerated + - [x] R4.7d (done 2026-10-19; STOP before R4.8) docs: RUST.md §10.9 (binary, parity 158/72, A.3 lines as printed, timing), CLAUDE.md (command, map), fixtures README if touched, memory journal; rust-check + pnpm check green; report, STOP before R4.8 - [ ] R4.8 deletions + docs + ledger census - [ ] R4.9 verifier + security + owner final acceptance → merge on owner signal - [ ] outside R4 (owner 2026-10-17, non-blocking): worker container ignores SIGTERM (Docker kill after 10 s; chore, e.g. `init: true`); README test counter stale (367 vs 1,015) diff --git a/CLAUDE.md b/CLAUDE.md index 71e8cf3a..cb6bd93b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -118,7 +118,7 @@ pnpm db:types:check # CI diff pnpm build:preview # SHEET_BUNDLE=draft build (manual validation, preview deployments) pnpm preview:draft # same, then serves it pnpm import:equine # regenerate legacy-import/ and draft/ (document, migration map, review drawings) -pnpm report:compare ... # archive check: attachment vs its v3 reading, unresolved items flagged (docs/PDF-FORMAT.md) +pnpm report:compare ... # archive check: attachment vs its v3 reading, unresolved items flagged (docs/PDF-FORMAT.md); the core's `report-compare` binary since R4.7 (scripts/report-compare.sh, first run compiles) # Rust (docs/development/RUST.md) — native toolchain (user-level rustup, channel of rust-toolchain.toml) first pnpm rust:check # scripts/rust-check.sh: fmt --check, clippy -D warnings, cargo test (core, wasm, worker unit), ts-rs export, cargo deny @@ -191,7 +191,7 @@ faunosteo/ ├─ apps/app/ ← @faunosteo/app (everything below until packages/) ├─ public/ ← favicons, env.js, site header logo only (PDF logos live in src/assets/brand; no anatomy asset) ├─ vite/sheet-bundle.ts ← Vite plugin: `virtual:faunosteo-sheet` (SHEET_BUNDLE=draft|none) -├─ scripts/compare-reports.ts ← `pnpm report:compare`: archive PDF vs v3 reading (pure part in scripts/compare/report.ts) +├─ scripts/compare-reports.ts ← the frozen TS archive check CLI (pure part in scripts/compare/report.ts): O10 reference until R4.8, no longer behind `pnpm report:compare` ├─ src/ │ ├─ assets/brand/ ← the four PDF branding logos, bundled by Vite (?inline / ?raw / ?url), never fetched │ ├─ app/ @@ -280,10 +280,10 @@ faunosteo/ ├─ packages/README.md ← shared packages index ├─ packages/core-wasm/ ← @faunosteo/core-wasm: CoreRuntime (lazy, poison, recovery) + CoreExportSession (R4.6) + Node source, pkg/ shipped and pkg-proofs/ (built, git-ignored, BUILD_STAMP), ts-rs types (committed), frozen N-1 reader (tests/n-1), tests ├─ crates/ ← Rust workspace (Cargo.toml, rust-toolchain.toml, deny.toml, clippy.toml, .cargo/config.toml at the root) -│ ├─ core/ ← faunosteo-core: domain/{js,sheet,report,archive,editor,session} (contracts and rules), anatomy/{svg/{prescan,tree,sanitize,refs,number,serialize},layer,compose} (R3: the one SVG reader, the layer, canvas + highlight composition), ops.rs + facade.rs (`call`, `call_with_sheet`), json (JS number printing); tests/{oracles,conformance,fixtures} frozen from the TypeScript (oracles/r3, fixtures/svg/hostile); no I/O, no clock (lint-enforced) +│ ├─ core/ ← faunosteo-core: domain/{js,sheet,report,archive,editor,session} (contracts and rules), anatomy/{svg/{prescan,tree,sanitize,refs,number,serialize},layer,compose} (R3: the one SVG reader, the layer, canvas + highlight composition), pdf/{read,json_js,import,write,compare} (R4: reader, writer and session, the archive check tool behind feature `cli`), ops.rs + facade.rs (`call`, `call_with_sheet`), json (JS number printing), bin/report_compare.rs (`pnpm report:compare`, the one file allowed std::fs); tests/{oracles,conformance,fixtures,pdf/LEDGER.tsv} frozen from the TypeScript (oracles/r3, oracles/r4, fixtures/svg/hostile); no I/O, no clock (lint-enforced) │ ├─ wasm/ ← faunosteo-wasm: wasm-bindgen facade (core_version, call, SheetHandle; trap exports in the proofs build only) │ └─ worker/ ← faunosteo-worker: plain main; `decode` sandboxed child (rlimits, seccomp allowlist, image → libwebp), parent Spawner + RIFF check; R1 `run` loop: config, log (JSON lines, allowlist), keys, backend trait + http (ureq, PostgREST rpc + Storage objects), job, sweep, watchdog; Dockerfile, scripts/derive-seccomp.sh -├─ scripts/{rust-env,rust-check,wasm-build,check-all,avatar-e2e}.sh, wasm-stamp.mjs ← native Rust gate, wasm build (pkg + pkg-proofs + stamp), the pre-merge gate (`pnpm check:all`), the real-stack avatar e2e +├─ scripts/{rust-env,rust-check,wasm-build,check-all,avatar-e2e,report-compare}.sh, wasm-stamp.mjs ← native Rust gate, wasm build (pkg + pkg-proofs + stamp), the pre-merge gate (`pnpm check:all`), the real-stack avatar e2e, the archive check tool (`pnpm report:compare`) ├─ scripts/{dev-env,supabase,dev}.sh ← the DEV origin (LAN address) in one place; CLI wrapper; `pnpm dev` on that origin ├─ docker-compose.yml ← `app` service; `worker` service under the `worker` profile (read-only, cap_drop ALL, external Supabase network, `.env.worker`) ├─ docker/rust-builder.Dockerfile, rust-run.sh, csp-proof/ ← reproducible Rust builder (alternative) + the CSP proofs (R0 module, R2 app image: app-image.sh) diff --git a/docs/PDF-FORMAT.md b/docs/PDF-FORMAT.md index c8f353ea..dda34d5d 100644 --- a/docs/PDF-FORMAT.md +++ b/docs/PDF-FORMAT.md @@ -267,10 +267,11 @@ fallback text and the export reports `logo_failed`. ### Known limits -- pdf-lib is out of the import path (BLK-009): the browser and the - `report:compare` CLI read the attachment by byte scan, so a crafted object - stream is never inflated (tested with a 64 MiB bomb: milliseconds, no - memory growth). pdf-lib stays on the export side, on trusted inputs, and +- pdf-lib is out of the import path (BLK-009): the browser (the core's + reader in a worker since R4.3) and the `report:compare` tool (the same + reader, natively, since R4.7) read the attachment by byte scan, so a + crafted object stream is never inflated (tested with a 64 MiB bomb: + milliseconds, no memory growth). pdf-lib stays on the export side, on trusted inputs, and in tests that inspect exported files. What remains before external users (`docs/development/PDF_IMPORT_HARDENING.md` §5): a Worker or server-side budget for any path that still parses PDF objects, the pdf-lib dependency @@ -290,7 +291,10 @@ fallback text and the export reports `logo_failed`. `pnpm report:compare ...` reads the attachment of one or more PDFs and prints it next to its v3 reading on the bundled reference -sheet. It writes no output file: +sheet (the core's `report-compare` binary since R4.7, `RUST.md` §10.9: the +R4 reader and the R2 layered reader, the same lines as the TypeScript tool +it replaces, three failure lines reworded as declared there). It writes no +output file: - an envelope prints a header line `archive v1 (snapshot kept)` or `archive v1 (snapshot rejected: )`; diff --git a/docs/development/RUST.md b/docs/development/RUST.md index 7abc0caa..875badbc 100644 --- a/docs/development/RUST.md +++ b/docs/development/RUST.md @@ -905,7 +905,7 @@ imports it and restyles `data-state` in place as before. needs it. - WASM budget: 396 KB of 600 KB used; R4 adds the PDF reader and writer. -## 10. R4: the PDF core in Rust (in progress; R4.1 freeze 2026-10-18, export switched 2026-10-19) +## 10. R4: the PDF core in Rust (in progress; R4.1 freeze 2026-10-18, export and `report:compare` switched 2026-10-19) Plan and contract: `.claude/tasks/plans/2026-10-17-rust-r4-pdf-core.md` (revision 2, validated by the owner 2026-10-18 with the conditions of its @@ -1880,3 +1880,83 @@ Residual risks and open points (R4.6): frees the session). - The headless runs validate nothing on a device: J2 (`R4_J2.md`) on the owner's computer and iPhone, with the J1 import recette still due. + +### 10.9 R4.7: the archive check tool on the core (2026-10-19, `pnpm report:compare` switched) + +What runs where. `crates/core/src/pdf/compare` (feature `cli`, which enables +the reader and the writer; never enabled by the wasm facade, so the shipped +module is unchanged: 1,535,289 B raw, 519,795 B gzip -9) holds the tool +without its file system: `compare_bytes` reads a PDF's attachment with the R4 +reader (`read_attached_document`, no value cap, as the TypeScript had none), +parses it as the session does (`parse_json`), reads it through the layered +archive reader (`read_archive_document`, the R2 code the application +imports with), and prints it as `compare/report.ts` did (`lines.rs`: zone +and finding lines, `!! UNRESOLVED` with its reason, the migration lines of a +v2 archive, the header with the sheet check and the counts, the footer, the +`--full` texts), with `esc` on every untrusted string (V8's +`[\p{Cc}\p{Cf}

]`, the `SAFE_TEXT_DROPPED` table of R4.4); +`run.rs` is `main` (`parse_args`, the loop over the entries, `--summary`, +`--diff`, stdout, stderr and the exit code) over a `Files` trait. The binary +`src/bin/report_compare.rs` (the one file of the crate allowed `std::fs` +and `std::env`) loads the seed draft sheet (`sheet-ref.json`, +`document.json`, `report-migration-map.json`; no drawing: the comparison +needs none), resolves the arguments against `INIT_CWD` (the folder pnpm was +typed in, as the TypeScript did) and implements `Files` (a folder stands for +its `.pdf` files sorted; a missing, unreadable, irregular or oversized entry +is a failed entry). `pnpm report:compare` runs `scripts/report-compare.sh` +(`rust-env.sh`, then `cargo run --release --locked -q --features cli --bin +report-compare`): the first run compiles the binary (25 s on the development +machine), the next ones start at once. The TypeScript CLI +(`apps/app/scripts/compare-reports.ts`, `compare/*`) and its vitest suite stay +until R4.8 (the O10 generator still runs them), with the app's own +`report:compare` script no longer behind the root one. + +Parity (cargo `pdf_compare`, 15 tests, 1.6 s): the 230 invocations of the +O10 oracle (`tests/oracles/r4/compare/compare.json`: every fixture PDF, the +step6 folder with and without options, the argument errors, the 22 runs of +the 11 `scripts.compare.test.ts` cases on their scenario PDFs) replayed +in-process through `run` on the seed sheet. The 158 rows the plan does not +declare match exactly (stdout, stderr, exit code); the 11 cases have one cargo +test each (the ledger targets, `pdf_compare::`, 0 pending rows left of +301). The real binary is spawned on the six `cli` and `folder` rows and on two +scenario rows: byte-equal to the pure run and to the oracle. The 72 declared +rows (§A.3) print one closed line each, exit 2: + +| TypeScript printed | Rows | Rust prints | +|---|---|---| +| a ZodError dump (the issues as JSON, escaped) | 63 | `report document invalid: ` (62: `missing field \`id\`` on the scan and read corpora, `missing field \`conclusion\``, `invalid type: null, expected f64` for the overflowing vertebra index); 1 decoy-first file whose decoy holds a lone surrogate: the refusal below, before any schema | +| the comparison, a lone surrogate kept in the content or a key, or overwritten by a duplicate (OD-4) | 6 | `attachment refused: lone UTF-16 surrogate escape at byte N` | +| `report version Infinity cannot be read` (2) and the report under `archive v1 (snapshot rejected: unsupported_envelope)` (1) | 3 | `report version null cannot be read`: the overflow reads as `null` in the core (`rewrite_overflow_numbers`), as the R3 application path already saw it (O8 `no_attachment` for the same three files) | + +Nothing else differs: the labels, the migration lines, the `__proto__` ids, +the control characters escaped (`\u001b`, `\u000a`, `\u009b`, `‮`, +`
`, five-digit code points as `1`), the `--diff` of the +placeholder and `--full` forms, the usage and option errors on stderr with +exit 1, the failed entries with exit 2. + +Time and memory (release binary, the development machine): one PDF 7 ms +(the seed read and indexed, the file compared), the three fixture folders +(72 entries, the R3 real-raster PDFs and the tolerance corpus included) +185 ms, the `r4/read` corpus (131 files) 135 ms; 6 MiB resident on the step6 +folder. Through `pnpm report:compare` the launch is cargo's (about 250 ms +per invocation, the binary already built); the TypeScript CLI took 270 ms +for one PDF and 410 ms for the three folders under tsx. + +Declared behaviour (plan §A.3, no other): the three lines above. The +output of every other row is the TypeScript's. `--locale`, `--summary`, +`--full`, `--diff` and the folder expansion keep their meaning; the exit +codes keep theirs (0, 1 usage or option, 2 failed entries). + +Residual risks and open points (R4.7): + +- The tool reads the seed from the repository checkout + (`CARGO_MANIFEST_DIR/../../supabase/seed/…`), as the TypeScript read it + from its script folder: the binary is not relocatable. A copied binary + prints the missing seed path and exits 1. +- The `report document invalid` line carries the core's serde detail + (`missing field …`, `invalid type …`, `unknown variant …`): the field name + or the offending value of the file, escaped; the TypeScript's ZodError + dump carried the same and more. +- A PDF over 50 MiB is refused by the entry check (`file too large`) before + the reader, as before; the reader's own cap would say `not a readable + PDF`.