Files
faunosteo/docs/development/PDF_IMPORT_HARDENING.md
T

168 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# PDF import hardening — BLK-009 analysis
Status 2026-10-11: analysis requested by the user after Step 6. §4 landed the
same day on `feature/step-06-pdf-archives` (options A + B, see §4 for the
commit); §5 lists what must land before the import is opened to external
users.
## 1. The problem
`readReportArchive` loads the whole PDF with pdf-lib (`PDFDocument.load`)
before looking for the attachment. pdf-lib's parser inflates every object
stream (`/ObjStm`) and cross-reference stream eagerly with its own, uncapped
inflater. Measured by the security audit: a 408 KB PDF expanded to 400 MB in
1.4 s (468 MB RSS); a 131 KB PDF to 128 MB. The ratio is about 1000:1, so the
50 MiB file cap still allows tens of gigabytes of expansion. Our 16 MiB caps
protect the attachment stream only, which the reader inflates itself.
Effect: a crafted PDF freezes the practitioner's tab or kills it (out of
memory). No code execution, no data access. A file size limit cannot help: the
amplification lives inside the compression.
Why `/ObjStm` cannot simply be refused: pdf-lib's `save()` writes object
streams by default (`useObjectStreams: true`), so every Faunosteo PDF
produced so far carries them, and refusing them would refuse the archive.
## 2. What the reader actually needs from the file
One embedded file stream, found through its `/Filespec`, and the document
keyword stamp. A PDF stream object can never live inside an object stream:
the `EmbeddedFile` dictionary and its data are always in clear text in the
file (`<< /Type /EmbeddedFile /Filter /FlateDecode /Length N … >> stream …
endstream`). Only the small dictionary objects (Filespec, Info, catalog) may
be packed into object streams.
So the attachment can be located by scanning the bytes, without parsing the
PDF object graph at all.
## 3. Options
| Option | Protection | Cost | Verdict |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| A. Byte-scan reader: find `/Type /EmbeddedFile` streams, read `/Length` (direct integer, as pdf-lib writes it; fall back to `endstream` search), inflate with the existing capped `inflateFlate`, parse JSON; no `PDFDocument.load` on import | removes pdf-lib from the import path entirely: the only inflation is our capped one | one module (~120 lines) + tests on every fixture (v2, bare v3, envelope, object streams on/off, several streams, foreign PDF) | **recommended, can land now** |
| B. Export with `useObjectStreams: false` | the Filespec name, the Info keywords and the Filespec→EF link become visible to a byte scan, so the filename-first rule and the `attachment_stripped` detection keep working without the object graph; files a few KB larger | one save option; older PDFs keep their object streams, which A tolerates | **recommended with A** |
| C. Load in a Web Worker with a timeout and a memory budget | contains the damage (the tab survives, the worker dies) | worker bundling, message protocol, no memory cap API in browsers (only termination on timeout) | defence in depth, later |
| D. Pre-scan declared `/Length` of `/ObjStm` streams and refuse above a sum | partial: declared lengths are the compressed sizes, the expansion is unknown | small | insufficient alone |
| E. Replace pdf-lib's parser (fork `@cantoo/pdf-lib`, or `pdf.js` for reading) | maintained parser; pdf.js has its own limits | dependency change, larger bundle | evaluate for Step 14 (server-side import) |
With A, the import path reads bytes, finds at most a few streams, inflates one
under 16 MiB, parses JSON under the existing schema caps. pdf-lib stays on
the export side (trusted inputs) and in the `report:compare` CLI (operator
tool, Node, acceptable risk but same fix applies).
Edge cases A must handle: several `EmbeddedFile` streams (pick the one whose
Filespec name starts with `osteo-report` when visible, else the first whose
inflated content parses as a report or an envelope); `/Length` as an indirect
reference (pdf-lib writes direct lengths; other producers may not: fall back
to scanning for `endstream`); `\r\n` versus `\n` after `stream`; encrypted
PDFs (today loaded with `ignoreEncryption`: the attachment stream of an
encrypted file is encrypted too and would fail to inflate → `no_attachment`,
same outcome as today in practice); the 50 MiB file cap stays as the scan
bound.
## 4. Done 2026-10-11 (feature/step-06-pdf-archives, commit: see commit 60ee712)
1. Option A in `packages/pdf/src/attachment/scan.ts`
(`findEmbeddedFiles`, `readAttachmentBytes`, `hasArchiveStamp`), used by
`readReportArchive`; `readAttachedDocument(bytes)` takes the file bytes
and the `report:compare` CLI runs on the scanner too. No
`PDFDocument.load` remains under `packages/pdf/src` or `apps/app/scripts`.
2. Option B on export (`pdfDoc.save({ useObjectStreams: false })`): no visible
change; goldens unchanged (they record drawn text, images, links); the
envelope roundtrip tests prove the reader finds the attachment in both
layouts.
3. Tests (`packages/pdf/tests/scan.test.ts`): the three frozen fixtures (object
streams) and a fresh export; two embedded files with a decoy first;
indirect and wrong `/Length`; `\r\n` after `stream`; `/Type/EmbeddedFile`
without a space; hex and literal names and keywords; garbage inputs and
the 32-candidate cap; a 64 MiB object-stream bomb next to the real
attachment (report in milliseconds, RSS flat) and an `EmbeddedFile` bomb
(inflation cancelled at 16 MiB, `no_attachment`).
4. Docs: `docs/PDF-FORMAT.md` "Reading an archive" and "Known limits".
Known trade-off: a stamped PDF saved before this change (object streams)
hides its `/Keywords`, so a stripped attachment on such a file reads as
`no_attachment` instead of `attachment_stripped`.
## 5. Prerequisites before external users (Step 7+ sync, Step 14 worker)
- Option A merged and the CLI on the scanner; no `PDFDocument.load` on any
untrusted input in the browser.
- Done at R4 (2026-10-18, `docs/development/RUST.md` §10.3 to §10.5): the
reader runs in Rust inside a disposable worker; the inflate bound and the
hostile classes were measured in Chromium 156, Firefox 157 and WebKit 27.2
(headless): a 16 MiB + 1 bomb costs 63 MiB of linear memory and is
refused, 32 bombs 68, a 16 MiB token-dense document 16 MiB under the
400,000-value cap (784 MiB with it off), the 1 GiB module ceiling traps;
the application build answers the same outcome and message as the
TypeScript reader on 666 of 678 (file, engine) pairs, the other 12 being
the four declared hostile files. Device run (J1, `R4_J1.md`): the owner
reports the functional checks executed on 2026-10-18 with no failure;
the device list, versions, scenario coverage, timings and memory
observations are recorded in `R4_J1_RESULTS.md` as they are confirmed.
The gate below stays open: imports are not offered to external users.
- Done: the candidate CPU/byte budget on the scanner: one 16 MiB budget of
retained and parsed content shared by every candidate (a cancelled
inflation is not charged, so up to 32 oversized streams may each cost one
capped inflation of CPU, bounded, no memory retained), a monotonic
`endstream` search over candidates sorted by data offset and bounded to
the cap, and no JSON parse for a candidate not opening on `{` or nesting
brackets deeper than 64 levels (linear byte pre-scan, strings skipped).
- Option C or a server-side import (Step 14) with a hard memory and time
budget per file, for any path that still parses PDF objects.
- Dependency decision on pdf-lib (unmaintained since 2021): closed at R4.8
(2026-10-19): pdf-lib left the repository with the TypeScript engine; the
core's writer (`RUST.md` §10.7, §10.8) generates the PDFs and its reader
reads them; no PDF library remains in the application, its tests or the
tools (`apps/app/tests/unit/r4.census.test.ts` fails on a return).
- Report collection caps on read (observations, zones, findings per
observation) so a hostile report cannot freeze the editor before the
export's own caps refuse it.
- The remaining low findings of the Step 6 audits: `<use>` nesting budget in
the sanitiser (today bounded by the rasterizer on the PDF path; since R3 the
Rust reader refuses reference cycles and expansions over 50,000 elements,
`docs/development/RUST.md` §9.2, and the engine gets it when R4 switches it to
that reader: still open for the PDF path), the decoder memory
envelope of PNG logos (bounded by the 4096² cap, about 430 MB worst case
for four logos).
## 6. R4.9 classification (2026-10-20, `feature/rust-r4-pdf-core`, before the merge)
Every open point of §5 and of the R4 decisions BDR-058, BDR-059 and BDR-063,
in one of four classes: **fixed with proof** (a test or a measure in the
repository), **accepted temporarily** (bounded, with the justification and
what closes it), **blocking before merge** (the owner's own gate), **before
public exposure** (or before the upload it concerns is activated). The
validated refusals of hostile files stay as they are: no cap, threshold or
message was relaxed at R4.9, and no path rewrites, drops or alters a user's
file, draft or archive on an error (the file is read through the File API
only; a refused import leaves the tab's texts in place: app suites
`routes.HomeRoute.*`, `routes.coreStopped`, `state.coreFailures`).
| # | Point | Origin | Class | Proof, justification, or what closes it |
|---|---|---|---|---|
| 1 | PDF object graph parsed by pdf-lib on untrusted input (the 1000:1 object-stream expansion, §1) | §5 item 1 | fixed with proof | no PDF library left (R4.8): the Rust byte scanner locates the attachment (`crates/core/src/pdf/read/probe.rs`), cargo `pdf_scan`, `pdf_robustness`; `apps/app/tests/unit/r4.census.test.ts` fails on a return of pdf-lib or a TypeScript scanner |
| 2 | Inflation bound and the hostile classes (bombs, 32 candidates, token-dense 16 MiB, nested copies) | §5 item 2, BDR-058 | fixed with proof | one 16 MiB budget shared by the candidates, `MAX_JSON_VALUES` 400,000, depth 64, `--max-memory` 1 GiB trap; cargo `pdf_budget`, `pdf_inflate`, `pdf_json_js`; the browser run of the classes (RUST.md §10.4, three engines: ≤ 68 MiB of linear memory, the tab alive) |
| 3 | The same classes on the owner's devices (iPhone, low-end Android) | BDR-058, BDR-060, plan OD-6 | blocking before merge | J1.2, J1.10 on the iPhone and the Android phone (`R4_J1.md`); the computer run of 2026-10-18 passed with no failure; a device not run stays NON TESTÉ |
| 4 | The 1 GiB module ceiling is a safety net, not a tab guarantee | BDR-058 | accepted temporarily | documented as such (RUST.md §10.3); the worker is terminated on timeout or crash, so the main instance never holds the hostile tree; closes with the device run (J1.9, J1.10: no reload, no memory warning) |
| 5 | WebKit, a 50 MiB file: 145 MiB of content growth against 76 for the deleted TypeScript reader | BDR-059 | accepted temporarily | the owner kept the 50 MiB cap and refused a streaming scanner at R4.3; the worker reads the file in 1 MiB chunks into linear memory (one copy; the 324 MiB worker-read figure is gone); closes with J1.8 on the iPhone (the tab survives, time noted) or a later streaming scanner |
| 6 | WebKit leak criterion (≤ 10 MiB recurring over ≥ 5 imports) not demonstrable headless | BDR-058, BDR-059 | accepted temporarily | headless WebKit has no forced GC; the one-page series levels off and a no-import control grows the same way with navigation (RUST.md §10.5); closes with J1.9 on the iPhone (six L imports, no reload) |
| 7 | Chromium per-export RSS growth ≈ 20 MiB on L (R4.6, same figures in the TypeScript control, plateau after 8) | BDR-063 | accepted temporarily | measured at R4.9 (RUST.md §10.11): the module's linear memory is flat after the init export, every canvas, Blob and object URL is released; a raster-only control with no core call grows the same 20 MiB per step to the same plateau (436 against 437 MiB after 16 steps, last five within 10 MiB): the browser's raster path, bounded; closes with J2.6 / J2.7 on the phones; correction only if a device shows it (one reused canvas, measured first) |
| 8 | A hidden tab pauses the task timer: a hostile file keeps the worker busy in the background | RUST.md §10.5 | accepted temporarily | by design (plan §B.1.5 item 3: a phone that switches apps must not lose a legitimate import); the core's own bounds (item 2) cap the work; the worker dies with the tab |
| 9 | Timers (10 s start, 20 s task) are hypotheses | RUST.md §10.5, plan §B.3 | blocking before merge | J1.12 on each device gives the slowest legitimate import; the rule (≥ 10 × that) confirms or moves the value before the merge |
| 10 | OD-4: an overwritten duplicate key holding a lone surrogate escape is refused where JSON.stringify once dropped it | BDR-057, BDR-058 | fixed with proof | declared (plan §A.3), refused with `error_lone_surrogate`, the file untouched; cargo `pdf_import` (O8 declared rows), app `routes.HomeRoute.messages`; no Faunosteo build writes duplicate keys |
| 11 | Worker crash, timeout and resource overflow told apart from a corrupt file | BDR-058, OD-5 | fixed with proof | closed outcome kinds (`core/worker/outcome.ts`), messages pinned by `routes.HomeRoute.import` and `routes.HomeRoute.rescueCopy`; a technical failure never says the PDF is not a Faunosteo file |
| 12 | Report collection caps on read (observations, zones, findings per observation) | §5 item "collection caps" | before public exposure | out of R4 (plan A11); today bounded by `MAX_JSON_VALUES` (400,000 values, 10 × the CAP envelope of 500 observations: a report of about 5,000 observations at the CAP density at most) and by the export's caps (500 × 8 views, closed kind); a dense hostile report slows the editor but cannot exhaust it |
| 13 | `<use>` nesting and reference budget on the PDF path | §5 "remaining low findings" | fixed with proof | every SVG the export composes went through the core's one reader (R3: cycles refused, expansion > 50,000 elements refused, prescan caps); `ExportInput` runs `read_svg` per asset (R4.4); cargo `svg_sanitiser`, `pdf_layout` |
| 14 | PNG logo decoder envelope (4096² cap, four logos) | §5 "remaining low findings" | fixed with proof | the application exports the bundled logos only (`src/assets/brand`, never the archive's branding: `export_from_envelope` is cargo-only); the gate refuses above 8192 px a side, 4096² pixels or 16 MiB of raw scanlines before any allocation (`PNG_MAX_*`, cargo `pdf_png` 108 cases + budget cases); the session frees the planes at `add_raster` |
| 15 | Option C or a server-side import for any path that still parses PDF objects | §5 item 4 | before public exposure | no such path exists today; applies to Step 14 if a server-side reader is introduced |
| 16 | pdf-lib (unmaintained since 2021) | §5 item 5 | fixed with proof | left the repository at R4.8; `r4.census` |
| 17 | Stamped PDFs saved with object streams (before Step 6) read a stripped attachment as `no_attachment` | §4 trade-off | accepted temporarily | pre-Step-6 files only, same outcome as a foreign PDF, no data at stake; documented in PDF-FORMAT |
| 18 | Imports offered to external users | §5 gate | before public exposure | the gate stays closed: the import is a local-tool feature; opening it needs items 3, 9 and 12 closed and a fresh audit of the worker path |
| 19 | J2 on the owner's devices (export FR / EN, views and markers, attachment, round trip, L on the phone, repetition) | BDR-063, BDR-060, plan A.4 | blocking before merge | `R4_J2.md` J2.1 to J2.14 on the OD-6 list; every cell NON TESTÉ today; headless runs (RUST.md §10.8, §10.11) never fill a cell |
| 20 | WebKit one-off worker `error` event seen headless at R4.2 / R4.3 | BDR-058 | accepted temporarily | not blocking per BDR-058: kept in the record, never masked (the client classifies every `error` event as a crash, `core/worker/client.ts`), the repetition series showed no recurrence (RUST.md §10.5); closes with J1.9 / J1.10 on the iPhone (Safari iOS) |
Owner reading at the R4.9 checkpoint: no item in this table is a code
change pending in R4; items 3, 9 and 19 are the device runs (J1 and J2),
items 12, 15 and 18 are the public-exposure gate, item 7 is measured and bounded (the browser's raster path), its device check is J2.6 / J2.7.