docs(docs): RUST.md §10.9 R4.7 record (tool on the core, O10 parity 158/72, A.3 lines as printed, timing), CLAUDE.md map and command, PDF-FORMAT; TODO R4.7 closed; memory BDR-063, journal

This commit is contained in:
bchanot committed 2026-10-19 11:13:42 +02:00
1 parent 02ebf05195
commit 77e11a53e4
6 files changed
+106 -11

No files matched your search

+6
View File
@@ -475,3 +475,9 @@ Status: active.
Context: owner decision 2026-10-19 on the R4.5 report (RUST.md §10.7). R4.6 brief: ExportSession + app driver, rasters as bytes one by one, OD-2 buffers documented, `view_raster_failed` on raster failure, parity + restoration (Rust → Rust, Rust → TS reader, N-1), E3/E4 three engines same data, shipped W1 measured, TS engine kept until R4.8, no merge, stop at R4.6.
Decision (design, measured): session in main instance per OD-2. File written in chunks never reallocated (target 1 MiB doubling to 8 MiB), image object written at `add_raster` (PNG + planes freed in the call), logos kept until branding batch completes (slot order as engine), catalog + page tree objects written last, file handed to JS in parts → `new Blob(parts)`, self-check scans attachment chunk behind header. R4 digests (own bytes, not TS oracle) re-frozen once for it; O2 suite + qpdf + poppler re-run green. Hex strings upper-case as pdf-lib (B4). Alternatives measured and dropped: single `Vec` doubling (125 MiB for 44 MiB file, realloc spike), two buffers (2 × file at finish), 1 MiB fixed chunks (0.12 MB fragmentation per view).
Status: active; owner checkpoint R4.6 pending.
## BDR-063 | 2026-10-19 | R4.6 accepted provisionally; R4.7 report-compare authorised; Chromium 10 MiB series = open reserve
Context: owner decision 2026-10-19 on the R4.6 report (RUST.md §10.8). Chromium per-export RSS growth on L (≈ 20 MiB, same in TS control, plateau after 8) accepted provisionally: not a leak validation, kept as open reserve; reports must split wasm linear memory / browser global / canvas-blob-URL retention / transient vs persistent; thresholds never changed, no FAIL turned PASS. J1 + J2 on real devices mandatory before any merge; NON TESTÉ kept. R4.7 per plan, no scope widening; S2 export worker not engaged without proof + new decision.
Decision (R4.7 design): `pdf::compare` pure module behind feature `cli` (= pdf-read + pdf-write, never in wasm), `Files` trait for fs; binary `src/bin/report_compare.rs` only std::fs file; seed sheet read at run time from `CARGO_MANIFEST_DIR/../../supabase/seed/.../draft` (no drawing); root `report:compare` → `scripts/report-compare.sh` (rust-env + cargo run --release --features cli). A.3 closed lines chosen: `report document invalid: <serde detail>` (ZodError dumps), `attachment refused: lone UTF-16 surrogate escape at byte N` (surrogates kept/overwritten), `report version null cannot be read` (overflow → null, incl. envelopeVersion overflow where TS printed a rejected-envelope header). Alternatives dropped: `include_str!` of the seed (bakes content, rebuild on seed change); spawning the binary per O10 row in tests (230 spawns, slow): in-process `run` + 8 spawns instead.
Status: active; owner checkpoint R4.7 pending.
+1
View File
@@ -293,3 +293,4 @@ Owner validated R4.0 plan with conditions (BDR-057, plan §A.6, LRN-059/060). R3
- 2026-10-18 owner validated R4.4; R4.5 go (BDR-061). Subtasks R4.5a-f in TODO. Tooling present: qpdf 12.3.2, poppler 26.01 (pdftoppm, pdfdetach), crates.io reachable.
- 2026-10-18 R4.5 done (d420f67..b44dd80): content emitter (60 streams byte-equal, numberToString 703 + V8 pow10 quirk), PNG gate + UPNG decoder (O6 108/108), own object writer (pdf-writer dropped: numberToString everywhere, no dep), serialiser, export + self-check, from_envelope; O2 14 runs equal (streams after B3, planes, links, Info, attachment), qpdf clean, poppler 0 px, O11 + N-1 + TS reader read R4 PDFs, wasm = native digests; Node TS→Rust demo 198→36 ms, L 772→57, CAP 5.7 s→0.22 s; writer linked 531.6 KB gzip (proofs build), shipped 426 KB. Gate PASS 4 LOW → hardenings. Open: one-shot path holds PNGs twice (R4.6 session must stream), L real rasters 5 s native (deflate 6), E3 judged at R4.6. STOP before R4.6.
- 2026-10-19 owner validated R4.5; R4.6 go (BDR-062). R4.6 done (9e688b6..): ExportSession (decode on arrival, chunked never-realloc file 1→8 MiB, catalog last, parts → Blob, self-check on attachment chunk), views.rs (compose_highlight requests, raster_size_of O4 57/57), wasm ExportSession + CoreExportSession, app export.ts on session (abort on unmount, User Timing), labels.ts; app suites read content streams (pdfRead.ts), app-path bytes = R4 digests (re-frozen: object order B2, hex upper-case B4), Step 6 goldens via app path, 14 O2 runs through module = digests, request SVG = TS compose text, export Rust → import Rust / TS reader / N-1 ok. E3 Chromium demo 0.67, L 0.74 (Firefox 0.66/0.67, WebKit 0.91/0.95); E4 CAP completes 3 engines (TS stopped by ceiling); peaks lower (L Chromium 805→470 MiB). Linear memory L real 62 MiB flat over 7 exports; Chromium per-export RSS growth ≈ 20 MiB on L same as TS control, plateau after 8. Shipped 519.7 KB gzip (W1). rust-check 0, pnpm check 1,099 tests, ledger 11 pending (R4.7). J2 kit + procedure. STOP before R4.7.
- 2026-10-19 owner accepted R4.6 provisionally (Chromium series = open reserve); R4.7 go (BDR-063). R4.7 done: `pdf::compare` (feature `cli`), `report-compare` bin, `scripts/report-compare.sh` behind root `report:compare`, cargo `pdf_compare` 15 tests: O10 230 rows (158 exact, 72 declared A.3 → 62 `report document invalid`, 7 surrogate refusals, 3 `version null`), 11 scenario cases = ledger targets (301 rows, 0 pending), binary spawned 8×. Release binary 7 ms/PDF, 6 MiB RSS; module unchanged 1,535,289 B / 519,795 gzip. RUST.md §10.9, CLAUDE.md, PDF-FORMAT. STOP before R4.8.
+5 -1
View File
@@ -250,7 +250,11 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo
- [x] R4.6f browsers (scripts/browser): Rust vs TS composed-view rasters pixel-identical (Chromium, Firefox, WebKit); E3/E4 harness demo/L/CAP, same data, TS vs Rust: total, raster, serialise, linear memory, JS heap, RSS; release proof on success / raster error / cancel (unmount) / recovery (trap mid-session); W1 shipped module with the writer linked (gzip -9; report > 550 KB)
- [x] R4.6g docs: RUST.md §10.8 (buffers ownership and lifetimes, allocations, release proofs, E3/E4 tables, W1, deviations: catalog at the end, decode at arrival, logo batch order), DEPLOY.md if the module size line moves, fixtures README, ledger rows (pdf.goldens, pdf.roundtrip targets filled), memory (BDR, LRN, journal)
- [x] R4.6h (done 2026-10-19; STOP before R4.7) owner recette procedure (visual + functional on the owner's devices: FR/EN, logos, annotations, real rasters, links, L on the phone), report, STOP before R4.7
- [ ] R4.7 report-compare in Rust
- [ ] R4.7 report-compare in Rust (owner go 2026-10-19, R4.6 accepted provisionally; stop before R4.8)
- [x] R4.7a core `pdf/compare/{mod,lines,run}.rs` behind feature `cli` (= pdf-read + pdf-write): `esc` on the V8 `[\p{Cc}\p{Cf}\u2028\u2029]` table (`SAFE_TEXT_DROPPED`), `Comparison` (header, v3 and migrated observation lines, footer, normalized, counts), `render_comparison` / `render_summary` / `diff_lines`, `parse_args`, `run(sheet, argv, &dyn Files)` → stdout, stderr, exit code; closed failure wording (A.3): `report document invalid: <core detail>` for the Zod dumps, `attachment refused: lone UTF-16 surrogate escape at byte N` for the kept surrogates, `report version null cannot be read` for an overflow read as null; `as_str` on Severity, SeverityV2, LateralSide, SnapshotRejection
- [x] R4.7b `src/bin/report_compare.rs` (feature `cli`, the only std::fs file): seed sheet from `supabase/seed/anatomical-sheets/equine/draft` (ref, document, map; no asset), arguments relative to `INIT_CWD`, folders expanded, size pre-check, exit code; `scripts/report-compare.sh` (rust-env + cargo run --release --features cli) behind the root `report:compare`; rust-check feature lists widened to `cli`
- [x] R4.7c cargo `tests/pdf_compare.rs`: the 230 O10 invocations in-process (158 undeclared rows exact: stdout, stderr, exit; 72 declared rows against the Rust wording per A.3 class), scenario PDF digests = `inputs`, the 11 scenario cases named for the ledger, one spawn of the real binary (usage, INIT_CWD, output = in-process); ledger rows 135–145 targets filled, `scripts/r4-ledger.mjs` regenerated
- [x] R4.7d (done 2026-10-19; STOP before R4.8) docs: RUST.md §10.9 (binary, parity 158/72, A.3 lines as printed, timing), CLAUDE.md (command, map), fixtures README if touched, memory journal; rust-check + pnpm check green; report, STOP before R4.8
- [ ] R4.8 deletions + docs + ledger census
- [ ] R4.9 verifier + security + owner final acceptance → merge on owner signal
- [ ] outside R4 (owner 2026-10-17, non-blocking): worker container ignores SIGTERM (Docker kill after 10 s; chore, e.g. `init: true`); README test counter stale (367 vs 1,015)
+4 -4
View File
@@ -118,7 +118,7 @@ pnpm db:types:check # CI diff
pnpm build:preview # SHEET_BUNDLE=draft build (manual validation, preview deployments)
pnpm preview:draft # same, then serves it
pnpm import:equine # regenerate legacy-import/ and draft/ (document, migration map, review drawings)
pnpm report:compare <pdf|dir>... # archive check: attachment vs its v3 reading, unresolved items flagged (docs/PDF-FORMAT.md)
pnpm report:compare <pdf|dir>... # archive check: attachment vs its v3 reading, unresolved items flagged (docs/PDF-FORMAT.md); the core's `report-compare` binary since R4.7 (scripts/report-compare.sh, first run compiles)
# Rust (docs/development/RUST.md) — native toolchain (user-level rustup, channel of rust-toolchain.toml) first
pnpm rust:check # scripts/rust-check.sh: fmt --check, clippy -D warnings, cargo test (core, wasm, worker unit), ts-rs export, cargo deny
@@ -191,7 +191,7 @@ faunosteo/
├─ apps/app/ ← @faunosteo/app (everything below until packages/)
├─ public/ ← favicons, env.js, site header logo only (PDF logos live in src/assets/brand; no anatomy asset)
├─ vite/sheet-bundle.ts ← Vite plugin: `virtual:faunosteo-sheet` (SHEET_BUNDLE=draft|none)
├─ scripts/compare-reports.ts ← `pnpm report:compare`: archive PDF vs v3 reading (pure part in scripts/compare/report.ts)
├─ scripts/compare-reports.ts ← the frozen TS archive check CLI (pure part in scripts/compare/report.ts): O10 reference until R4.8, no longer behind `pnpm report:compare`
├─ src/
│ ├─ assets/brand/ ← the four PDF branding logos, bundled by Vite (?inline / ?raw / ?url), never fetched
│ ├─ app/
@@ -280,10 +280,10 @@ faunosteo/
├─ packages/README.md ← shared packages index
├─ packages/core-wasm/ ← @faunosteo/core-wasm: CoreRuntime (lazy, poison, recovery) + CoreExportSession (R4.6) + Node source, pkg/ shipped and pkg-proofs/ (built, git-ignored, BUILD_STAMP), ts-rs types (committed), frozen N-1 reader (tests/n-1), tests
├─ crates/ ← Rust workspace (Cargo.toml, rust-toolchain.toml, deny.toml, clippy.toml, .cargo/config.toml at the root)
│ ├─ core/ ← faunosteo-core: domain/{js,sheet,report,archive,editor,session} (contracts and rules), anatomy/{svg/{prescan,tree,sanitize,refs,number,serialize},layer,compose} (R3: the one SVG reader, the layer, canvas + highlight composition), ops.rs + facade.rs (`call`, `call_with_sheet`), json (JS number printing); tests/{oracles,conformance,fixtures} frozen from the TypeScript (oracles/r3, fixtures/svg/hostile); no I/O, no clock (lint-enforced)
│ ├─ core/ ← faunosteo-core: domain/{js,sheet,report,archive,editor,session} (contracts and rules), anatomy/{svg/{prescan,tree,sanitize,refs,number,serialize},layer,compose} (R3: the one SVG reader, the layer, canvas + highlight composition), pdf/{read,json_js,import,write,compare} (R4: reader, writer and session, the archive check tool behind feature `cli`), ops.rs + facade.rs (`call`, `call_with_sheet`), json (JS number printing), bin/report_compare.rs (`pnpm report:compare`, the one file allowed std::fs); tests/{oracles,conformance,fixtures,pdf/LEDGER.tsv} frozen from the TypeScript (oracles/r3, oracles/r4, fixtures/svg/hostile); no I/O, no clock (lint-enforced)
│ ├─ wasm/ ← faunosteo-wasm: wasm-bindgen facade (core_version, call, SheetHandle; trap exports in the proofs build only)
│ └─ worker/ ← faunosteo-worker: plain main; `decode` sandboxed child (rlimits, seccomp allowlist, image → libwebp), parent Spawner + RIFF check; R1 `run` loop: config, log (JSON lines, allowlist), keys, backend trait + http (ureq, PostgREST rpc + Storage objects), job, sweep, watchdog; Dockerfile, scripts/derive-seccomp.sh
├─ scripts/{rust-env,rust-check,wasm-build,check-all,avatar-e2e}.sh, wasm-stamp.mjs ← native Rust gate, wasm build (pkg + pkg-proofs + stamp), the pre-merge gate (`pnpm check:all`), the real-stack avatar e2e
├─ scripts/{rust-env,rust-check,wasm-build,check-all,avatar-e2e,report-compare}.sh, wasm-stamp.mjs ← native Rust gate, wasm build (pkg + pkg-proofs + stamp), the pre-merge gate (`pnpm check:all`), the real-stack avatar e2e, the archive check tool (`pnpm report:compare`)
├─ scripts/{dev-env,supabase,dev}.sh ← the DEV origin (LAN address) in one place; CLI wrapper; `pnpm dev` on that origin
├─ docker-compose.yml ← `app` service; `worker` service under the `worker` profile (read-only, cap_drop ALL, external Supabase network, `.env.worker`)
├─ docker/rust-builder.Dockerfile, rust-run.sh, csp-proof/ ← reproducible Rust builder (alternative) + the CSP proofs (R0 module, R2 app image: app-image.sh)
+9 -5
View File
@@ -267,10 +267,11 @@ fallback text and the export reports `logo_failed`.
### Known limits
- pdf-lib is out of the import path (BLK-009): the browser and the
`report:compare` CLI read the attachment by byte scan, so a crafted object
stream is never inflated (tested with a 64 MiB bomb: milliseconds, no
memory growth). pdf-lib stays on the export side, on trusted inputs, and
- pdf-lib is out of the import path (BLK-009): the browser (the core's
reader in a worker since R4.3) and the `report:compare` tool (the same
reader, natively, since R4.7) read the attachment by byte scan, so a
crafted object stream is never inflated (tested with a 64 MiB bomb:
milliseconds, no memory growth). pdf-lib stays on the export side, on trusted inputs, and
in tests that inspect exported files. What remains before external users
(`docs/development/PDF_IMPORT_HARDENING.md` §5): a Worker or server-side
budget for any path that still parses PDF objects, the pdf-lib dependency
@@ -290,7 +291,10 @@ fallback text and the export reports `logo_failed`.
`pnpm report:compare <file.pdf | folder>...` reads the attachment of one or
more PDFs and prints it next to its v3 reading on the bundled reference
sheet. It writes no output file:
sheet (the core's `report-compare` binary since R4.7, `RUST.md` §10.9: the
R4 reader and the R2 layered reader, the same lines as the TypeScript tool
it replaces, three failure lines reworded as declared there). It writes no
output file:
- an envelope prints a header line `archive v1 (snapshot kept)` or
`archive v1 (snapshot rejected: <reason>)`;
+81 -1
View File
@@ -905,7 +905,7 @@ imports it and restyles `data-state` in place as before.
needs it.
- WASM budget: 396 KB of 600 KB used; R4 adds the PDF reader and writer.
## 10. R4: the PDF core in Rust (in progress; R4.1 freeze 2026-10-18, export switched 2026-10-19)
## 10. R4: the PDF core in Rust (in progress; R4.1 freeze 2026-10-18, export and `report:compare` switched 2026-10-19)
Plan and contract: `.claude/tasks/plans/2026-10-17-rust-r4-pdf-core.md`
(revision 2, validated by the owner 2026-10-18 with the conditions of its
@@ -1880,3 +1880,83 @@ Residual risks and open points (R4.6):
frees the session).
- The headless runs validate nothing on a device: J2 (`R4_J2.md`) on the
owner's computer and iPhone, with the J1 import recette still due.
### 10.9 R4.7: the archive check tool on the core (2026-10-19, `pnpm report:compare` switched)
What runs where. `crates/core/src/pdf/compare` (feature `cli`, which enables
the reader and the writer; never enabled by the wasm facade, so the shipped
module is unchanged: 1,535,289 B raw, 519,795 B gzip -9) holds the tool
without its file system: `compare_bytes` reads a PDF's attachment with the R4
reader (`read_attached_document`, no value cap, as the TypeScript had none),
parses it as the session does (`parse_json`), reads it through the layered
archive reader (`read_archive_document`, the R2 code the application
imports with), and prints it as `compare/report.ts` did (`lines.rs`: zone
and finding lines, `!! UNRESOLVED` with its reason, the migration lines of a
v2 archive, the header with the sheet check and the counts, the footer, the
`--full` texts), with `esc` on every untrusted string (V8's
`[\p{Cc}\p{Cf}

]`, the `SAFE_TEXT_DROPPED` table of R4.4);
`run.rs` is `main` (`parse_args`, the loop over the entries, `--summary`,
`--diff`, stdout, stderr and the exit code) over a `Files` trait. The binary
`src/bin/report_compare.rs` (the one file of the crate allowed `std::fs`
and `std::env`) loads the seed draft sheet (`sheet-ref.json`,
`document.json`, `report-migration-map.json`; no drawing: the comparison
needs none), resolves the arguments against `INIT_CWD` (the folder pnpm was
typed in, as the TypeScript did) and implements `Files` (a folder stands for
its `.pdf` files sorted; a missing, unreadable, irregular or oversized entry
is a failed entry). `pnpm report:compare` runs `scripts/report-compare.sh`
(`rust-env.sh`, then `cargo run --release --locked -q --features cli --bin
report-compare`): the first run compiles the binary (25 s on the development
machine), the next ones start at once. The TypeScript CLI
(`apps/app/scripts/compare-reports.ts`, `compare/*`) and its vitest suite stay
until R4.8 (the O10 generator still runs them), with the app's own
`report:compare` script no longer behind the root one.
Parity (cargo `pdf_compare`, 15 tests, 1.6 s): the 230 invocations of the
O10 oracle (`tests/oracles/r4/compare/compare.json`: every fixture PDF, the
step6 folder with and without options, the argument errors, the 22 runs of
the 11 `scripts.compare.test.ts` cases on their scenario PDFs) replayed
in-process through `run` on the seed sheet. The 158 rows the plan does not
declare match exactly (stdout, stderr, exit code); the 11 cases have one cargo
test each (the ledger targets, `pdf_compare::<case>`, 0 pending rows left of
301). The real binary is spawned on the six `cli` and `folder` rows and on two
scenario rows: byte-equal to the pure run and to the oracle. The 72 declared
rows (§A.3) print one closed line each, exit 2:
| TypeScript printed | Rows | Rust prints |
|---|---|---|
| a ZodError dump (the issues as JSON, escaped) | 63 | `report document invalid: <the core's detail>` (62: `missing field \`id\`` on the scan and read corpora, `missing field \`conclusion\``, `invalid type: null, expected f64` for the overflowing vertebra index); 1 decoy-first file whose decoy holds a lone surrogate: the refusal below, before any schema |
| the comparison, a lone surrogate kept in the content or a key, or overwritten by a duplicate (OD-4) | 6 | `attachment refused: lone UTF-16 surrogate escape at byte N` |
| `report version Infinity cannot be read` (2) and the report under `archive v1 (snapshot rejected: unsupported_envelope)` (1) | 3 | `report version null cannot be read`: the overflow reads as `null` in the core (`rewrite_overflow_numbers`), as the R3 application path already saw it (O8 `no_attachment` for the same three files) |
Nothing else differs: the labels, the migration lines, the `__proto__` ids,
the control characters escaped (`\u001b`, `\u000a`, `\u009b`, `‮`,
`
`, five-digit code points as `1`), the `--diff` of the
placeholder and `--full` forms, the usage and option errors on stderr with
exit 1, the failed entries with exit 2.
Time and memory (release binary, the development machine): one PDF 7 ms
(the seed read and indexed, the file compared), the three fixture folders
(72 entries, the R3 real-raster PDFs and the tolerance corpus included)
185 ms, the `r4/read` corpus (131 files) 135 ms; 6 MiB resident on the step6
folder. Through `pnpm report:compare` the launch is cargo's (about 250 ms
per invocation, the binary already built); the TypeScript CLI took 270 ms
for one PDF and 410 ms for the three folders under tsx.
Declared behaviour (plan §A.3, no other): the three lines above. The
output of every other row is the TypeScript's. `--locale`, `--summary`,
`--full`, `--diff` and the folder expansion keep their meaning; the exit
codes keep theirs (0, 1 usage or option, 2 failed entries).
Residual risks and open points (R4.7):
- The tool reads the seed from the repository checkout
(`CARGO_MANIFEST_DIR/../../supabase/seed/…`), as the TypeScript read it
from its script folder: the binary is not relocatable. A copied binary
prints the missing seed path and exits 1.
- The `report document invalid` line carries the core's serde detail
(`missing field …`, `invalid type …`, `unknown variant …`): the field name
or the offending value of the file, escaped; the TypeScript's ZodError
dump carried the same and more.
- A PDF over 50 MiB is refused by the entry check (`file too large`) before
the reader, as before; the reader's own cap would say `not a readable
PDF`.