Files
faunosteo/docs/development/PDF_IMPORT_HARDENING.md
bchanot ac60b24267 docs(docs): realign active docs on develop for P0
Roadmap section and step statuses in the plan, D-R9 Rust services in
place of Edge Functions and a Node worker, platform roles per BDR-026,
equine sheet draft-only, curation-only authoring until Step 17,
OPEN_QUESTIONS statuses and conflict resolutions, dated snapshot
notices on the Step 0 documents.
2026-10-25 17:12:37 +01:00

20 KiB
Raw Permalink Blame History

PDF import hardening — BLK-009 analysis

Status 2026-10-11: analysis requested by the user after Step 6. §4 landed the same day on feature/step-06-pdf-archives (options A + B, see §4 for the commit); §5 lists what must land before the import is opened to external users.

1. The problem

readReportArchive loads the whole PDF with pdf-lib (PDFDocument.load) before looking for the attachment. pdf-lib's parser inflates every object stream (/ObjStm) and cross-reference stream eagerly with its own, uncapped inflater. Measured by the security audit: a 408 KB PDF expanded to 400 MB in 1.4 s (468 MB RSS); a 131 KB PDF to 128 MB. The ratio is about 1000:1, so the 50 MiB file cap still allows tens of gigabytes of expansion. Our 16 MiB caps protect the attachment stream only, which the reader inflates itself.

Effect: a crafted PDF freezes the practitioner's tab or kills it (out of memory). No code execution, no data access. A file size limit cannot help: the amplification lives inside the compression.

Why /ObjStm cannot simply be refused: pdf-lib's save() writes object streams by default (useObjectStreams: true), so every Faunosteo PDF produced so far carries them, and refusing them would refuse the archive.

2. What the reader actually needs from the file

One embedded file stream, found through its /Filespec, and the document keyword stamp. A PDF stream object can never live inside an object stream: the EmbeddedFile dictionary and its data are always in clear text in the file (<< /Type /EmbeddedFile /Filter /FlateDecode /Length N … >> stream … endstream). Only the small dictionary objects (Filespec, Info, catalog) may be packed into object streams.

So the attachment can be located by scanning the bytes, without parsing the PDF object graph at all.

3. Options

Option Protection Cost Verdict
A. Byte-scan reader: find /Type /EmbeddedFile streams, read /Length (direct integer, as pdf-lib writes it; fall back to endstream search), inflate with the existing capped inflateFlate, parse JSON; no PDFDocument.load on import removes pdf-lib from the import path entirely: the only inflation is our capped one one module (~120 lines) + tests on every fixture (v2, bare v3, envelope, object streams on/off, several streams, foreign PDF) recommended, can land now
B. Export with useObjectStreams: false the Filespec name, the Info keywords and the Filespec→EF link become visible to a byte scan, so the filename-first rule and the attachment_stripped detection keep working without the object graph; files a few KB larger one save option; older PDFs keep their object streams, which A tolerates recommended with A
C. Load in a Web Worker with a timeout and a memory budget contains the damage (the tab survives, the worker dies) worker bundling, message protocol, no memory cap API in browsers (only termination on timeout) defence in depth, later
D. Pre-scan declared /Length of /ObjStm streams and refuse above a sum partial: declared lengths are the compressed sizes, the expansion is unknown small insufficient alone
E. Replace pdf-lib's parser (fork @cantoo/pdf-lib, or pdf.js for reading) maintained parser; pdf.js has its own limits dependency change, larger bundle evaluate for Step 14 (server-side import)

With A, the import path reads bytes, finds at most a few streams, inflates one under 16 MiB, parses JSON under the existing schema caps. pdf-lib stays on the export side (trusted inputs) and in the report:compare CLI (operator tool, Node, acceptable risk but same fix applies).

Edge cases A must handle: several EmbeddedFile streams (pick the one whose Filespec name starts with osteo-report when visible, else the first whose inflated content parses as a report or an envelope); /Length as an indirect reference (pdf-lib writes direct lengths; other producers may not: fall back to scanning for endstream); \r\n versus \n after stream; encrypted PDFs (today loaded with ignoreEncryption: the attachment stream of an encrypted file is encrypted too and would fail to inflate → no_attachment, same outcome as today in practice); the 50 MiB file cap stays as the scan bound.

4. Done 2026-10-11 (feature/step-06-pdf-archives, commit: see commit 60ee712)

  1. Option A in packages/pdf/src/attachment/scan.ts (findEmbeddedFiles, readAttachmentBytes, hasArchiveStamp), used by readReportArchive; readAttachedDocument(bytes) takes the file bytes and the report:compare CLI runs on the scanner too. No PDFDocument.load remains under packages/pdf/src or apps/app/scripts.
  2. Option B on export (pdfDoc.save({ useObjectStreams: false })): no visible change; goldens unchanged (they record drawn text, images, links); the envelope roundtrip tests prove the reader finds the attachment in both layouts.
  3. Tests (packages/pdf/tests/scan.test.ts): the three frozen fixtures (object streams) and a fresh export; two embedded files with a decoy first; indirect and wrong /Length; \r\n after stream; /Type/EmbeddedFile without a space; hex and literal names and keywords; garbage inputs and the 32-candidate cap; a 64 MiB object-stream bomb next to the real attachment (report in milliseconds, RSS flat) and an EmbeddedFile bomb (inflation cancelled at 16 MiB, no_attachment).
  4. Docs: docs/PDF-FORMAT.md "Reading an archive" and "Known limits".

Known trade-off: a stamped PDF saved before this change (object streams) hides its /Keywords, so a stripped attachment on such a file reads as no_attachment instead of attachment_stripped.

5. Prerequisites before external users (Step 7+ sync; cloud import at Step 15, through the Rust worker or a Rust service)

  • Option A merged and the CLI on the scanner; no PDFDocument.load on any untrusted input in the browser.
  • Done at R4 (2026-10-18, docs/development/RUST.md §10.3 to §10.5): the reader runs in Rust inside a disposable worker; the inflate bound and the hostile classes were measured in Chromium 156, Firefox 157 and WebKit 27.2 (headless): a 16 MiB + 1 bomb costs 63 MiB of linear memory and is refused, 32 bombs 68, a 16 MiB token-dense document 16 MiB under the 400,000-value cap (784 MiB with it off), the 1 GiB module ceiling traps; the application build answers the same outcome and message as the TypeScript reader on 666 of 678 (file, engine) pairs, the other 12 being the four declared hostile files. Device run (J1, R4_J1.md): the owner reports the functional checks executed on 2026-10-18 with no failure; the device list, versions, scenario coverage, timings and memory observations are recorded in R4_J1_RESULTS.md as they are confirmed. The gate below stays open: imports are not offered to external users.
  • Done: the candidate CPU/byte budget on the scanner: one 16 MiB budget of retained and parsed content shared by every candidate (a cancelled inflation is not charged, so up to 32 oversized streams may each cost one capped inflation of CPU, bounded, no memory retained), a monotonic endstream search over candidates sorted by data offset and bounded to the cap, and no JSON parse for a candidate not opening on { or nesting brackets deeper than 64 levels (linear byte pre-scan, strings skipped).
  • Option C or a server-side import (Step 15) with a hard memory and time budget per file, for any path that still parses PDF objects.
  • Dependency decision on pdf-lib (unmaintained since 2021): closed at R4.8 (2026-10-19): pdf-lib left the repository with the TypeScript engine; the core's writer (RUST.md §10.7, §10.8) generates the PDFs and its reader reads them; no PDF library remains in the application, its tests or the tools (apps/app/tests/unit/r4.census.test.ts fails on a return).
  • Report collection caps on read (observations, zones, findings per observation) so a hostile report cannot freeze the editor before the export's own caps refuse it.
  • The remaining low findings of the Step 6 audits: <use> nesting budget in the sanitiser (today bounded by the rasterizer on the PDF path; since R3 the Rust reader refuses reference cycles and expansions over 50,000 elements, docs/development/RUST.md §9.2, and the engine gets it when R4 switches it to that reader: still open for the PDF path), the decoder memory envelope of PNG logos (bounded by the 4096² cap, about 430 MB worst case for four logos).

6. R4.9 classification (2026-10-20, feature/rust-r4-pdf-core, before the merge)

Every open point of §5 and of the R4 decisions BDR-058, BDR-059 and BDR-063, in one of four classes: fixed with proof (a test or a measure in the repository), accepted temporarily (bounded, with the justification and what closes it), blocking before merge (the owner's own gate), before public exposure (or before the upload it concerns is activated). The validated refusals of hostile files stay as they are: no cap, threshold or message was relaxed at R4.9, and no path rewrites, drops or alters a user's file, draft or archive on an error (the file is read through the File API only; a refused import leaves the tab's texts in place: app suites routes.HomeRoute.*, routes.coreStopped, state.coreFailures).

# Point Origin Class Proof, justification, or what closes it
1 PDF object graph parsed by pdf-lib on untrusted input (the 1000:1 object-stream expansion, §1) §5 item 1 fixed with proof no PDF library left (R4.8): the Rust byte scanner locates the attachment (crates/core/src/pdf/read/probe.rs), cargo pdf_scan, pdf_robustness; apps/app/tests/unit/r4.census.test.ts fails on a return of pdf-lib or a TypeScript scanner
2 Inflation bound and the hostile classes (bombs, 32 candidates, token-dense 16 MiB, nested copies) §5 item 2, BDR-058 fixed with proof one 16 MiB budget shared by the candidates, MAX_JSON_VALUES 400,000, depth 64, --max-memory 1 GiB trap; cargo pdf_budget, pdf_inflate, pdf_json_js; the browser run of the classes (RUST.md §10.4, three engines: ≤ 68 MiB of linear memory, the tab alive)
3 The same classes on the owner's devices (iPhone, low-end Android) BDR-058, BDR-060, plan OD-6 blocking before merge J1.2, J1.10 on the iPhone and the Android phone (R4_J1.md); the computer run of 2026-10-18 passed with no failure; a device not run stays NON TESTÉ
4 The 1 GiB module ceiling is a safety net, not a tab guarantee BDR-058 accepted temporarily documented as such (RUST.md §10.3); the worker is terminated on timeout or crash, so the main instance never holds the hostile tree; closes with the device run (J1.9, J1.10: no reload, no memory warning)
5 WebKit, a 50 MiB file: 145 MiB of content growth against 76 for the deleted TypeScript reader BDR-059 accepted temporarily the owner kept the 50 MiB cap and refused a streaming scanner at R4.3; the worker reads the file in 1 MiB chunks into linear memory (one copy; the 324 MiB worker-read figure is gone); closes with J1.8 on the iPhone (the tab survives, time noted) or a later streaming scanner
6 WebKit leak criterion (≤ 10 MiB recurring over ≥ 5 imports) not demonstrable headless BDR-058, BDR-059 accepted temporarily headless WebKit has no forced GC; the one-page series levels off and a no-import control grows the same way with navigation (RUST.md §10.5); closes with J1.9 on the iPhone (six L imports, no reload)
7 Chromium per-export RSS growth ≈ 20 MiB on L (R4.6, same figures in the TypeScript control, plateau after 8) BDR-063 accepted temporarily measured at R4.9 (RUST.md §10.11): the module's linear memory is flat after the init export, every canvas, Blob and object URL is released; a raster-only control with no core call grows the same 20 MiB per step to the same plateau (436 against 437 MiB after 16 steps, last five within 10 MiB): the browser's raster path, bounded; closes with J2.6 / J2.7 on the phones; correction only if a device shows it (one reused canvas, measured first)
8 A hidden tab pauses the task timer: a hostile file keeps the worker busy in the background RUST.md §10.5 accepted temporarily by design (plan §B.1.5 item 3: a phone that switches apps must not lose a legitimate import); the core's own bounds (item 2) cap the work; the worker dies with the tab
9 Timers (10 s start, 20 s task) are hypotheses RUST.md §10.5, plan §B.3 blocking before merge J1.12 on each device gives the slowest legitimate import; the rule (≥ 10 × that) confirms or moves the value before the merge
10 OD-4: an overwritten duplicate key holding a lone surrogate escape is refused where JSON.stringify once dropped it BDR-057, BDR-058 fixed with proof declared (plan §A.3), refused with error_lone_surrogate, the file untouched; cargo pdf_import (O8 declared rows), app routes.HomeRoute.messages; no Faunosteo build writes duplicate keys
11 Worker crash, timeout and resource overflow told apart from a corrupt file BDR-058, OD-5 fixed with proof closed outcome kinds (core/worker/outcome.ts), messages pinned by routes.HomeRoute.import and routes.HomeRoute.rescueCopy; a technical failure never says the PDF is not a Faunosteo file
12 Report collection caps on read (observations, zones, findings per observation) §5 item "collection caps" before public exposure out of R4 (plan A11); today bounded by MAX_JSON_VALUES (400,000 values, 10 × the CAP envelope of 500 observations: a report of about 5,000 observations at the CAP density at most) and by the export's caps (500 × 8 views, closed kind); a dense hostile report slows the editor but cannot exhaust it
13 <use> nesting and reference budget on the PDF path §5 "remaining low findings" fixed with proof every SVG the export composes went through the core's one reader (R3: cycles refused, expansion > 50,000 elements refused, prescan caps); ExportInput runs read_svg per asset (R4.4); cargo svg_sanitiser, pdf_layout
14 PNG logo decoder envelope (4096² cap, four logos) §5 "remaining low findings" fixed with proof the application exports the bundled logos only (src/assets/brand, never the archive's branding: export_from_envelope is cargo-only); the gate refuses above 8192 px a side, 4096² pixels or 16 MiB of raw scanlines before any allocation (PNG_MAX_*, cargo pdf_png 108 cases + budget cases); the session frees the planes at add_raster
15 Option C or a server-side import for any path that still parses PDF objects §5 item 4 before public exposure no such path exists today; applies to Step 15 if a server-side reader is introduced
16 pdf-lib (unmaintained since 2021) §5 item 5 fixed with proof left the repository at R4.8; r4.census
17 Stamped PDFs saved with object streams (before Step 6) read a stripped attachment as no_attachment §4 trade-off accepted temporarily pre-Step-6 files only, same outcome as a foreign PDF, no data at stake; documented in PDF-FORMAT
18 Imports offered to external users §5 gate before public exposure the gate stays closed: the import is a local-tool feature; opening it needs items 3, 9 and 12 closed and a fresh audit of the worker path
19 J2 on the owner's devices (export FR / EN, views and markers, attachment, round trip, L on the phone, repetition) BDR-063, BDR-060, plan A.4 blocking before merge R4_J2.md J2.1 to J2.14 on the OD-6 list; every cell NON TESTÉ today; headless runs (RUST.md §10.8, §10.11) never fill a cell
20 WebKit one-off worker error event seen headless at R4.2 / R4.3 BDR-058 accepted temporarily not blocking per BDR-058: kept in the record, never masked (the client classifies every error event as a crash, core/worker/client.ts), the repetition series showed no recurrence (RUST.md §10.5); closes with J1.9 / J1.10 on the iPhone (Safari iOS)

Owner reading at the R4.9 checkpoint: no item in this table is a code change pending in R4; items 3, 9 and 19 are the device runs (J1 and J2), items 12, 15 and 18 are the public-exposure gate, item 7 is measured and bounded (the browser's raster path), its device check is J2.6 / J2.7.

7. R4 close (2026-10-21, merged into develop)

The owner ran J1 and J2 on the OD-6 devices (computer with Chrome, Firefox and Brave; Android with its browser; a real iPhone with Safari iOS) and reports both recettes as passed with no blocking anomaly (R4_J1_RESULTS.md, R4_J2_RESULTS.md, docs/development/RUST.md §10.15). Effect on the §6 table, nothing else changed:

  • Items 3 and 19 (the device runs) no longer block the merge: every device was run. Their per-check detail (J1.2, J1.10, J2.1 to J2.14 on each device) was declared as a whole, not individually.
  • Items 5, 6, 7, 9 and 20 stay accepted temporarily: what closes them is a measure or an observation on the phone (the J1.8 time and tab survival, the J1.9 and J1.10 memory behaviour, the J2.6 and J2.7 growth, the J1.12 timings behind the timer rule, the worker error event over repeated imports), and none was reported. They close when the owner reports those observations, or at the next device recette.
  • Items 12, 15 and 18 (before public exposure) are untouched: the import stays a local-tool feature behind the gate of §5, and the merge of R4 opens nothing to external users.