168 lines
19 KiB
Markdown
168 lines
19 KiB
Markdown
# PDF import hardening — BLK-009 analysis
|
||
|
||
Status 2026-10-11: analysis requested by the user after Step 6. §4 landed the
|
||
same day on `feature/step-06-pdf-archives` (options A + B, see §4 for the
|
||
commit); §5 lists what must land before the import is opened to external
|
||
users.
|
||
|
||
## 1. The problem
|
||
|
||
`readReportArchive` loads the whole PDF with pdf-lib (`PDFDocument.load`)
|
||
before looking for the attachment. pdf-lib's parser inflates every object
|
||
stream (`/ObjStm`) and cross-reference stream eagerly with its own, uncapped
|
||
inflater. Measured by the security audit: a 408 KB PDF expanded to 400 MB in
|
||
1.4 s (468 MB RSS); a 131 KB PDF to 128 MB. The ratio is about 1000:1, so the
|
||
50 MiB file cap still allows tens of gigabytes of expansion. Our 16 MiB caps
|
||
protect the attachment stream only, which the reader inflates itself.
|
||
|
||
Effect: a crafted PDF freezes the practitioner's tab or kills it (out of
|
||
memory). No code execution, no data access. A file size limit cannot help: the
|
||
amplification lives inside the compression.
|
||
|
||
Why `/ObjStm` cannot simply be refused: pdf-lib's `save()` writes object
|
||
streams by default (`useObjectStreams: true`), so every Faunosteo PDF
|
||
produced so far carries them, and refusing them would refuse the archive.
|
||
|
||
## 2. What the reader actually needs from the file
|
||
|
||
One embedded file stream, found through its `/Filespec`, and the document
|
||
keyword stamp. A PDF stream object can never live inside an object stream:
|
||
the `EmbeddedFile` dictionary and its data are always in clear text in the
|
||
file (`<< /Type /EmbeddedFile /Filter /FlateDecode /Length N … >> stream …
|
||
endstream`). Only the small dictionary objects (Filespec, Info, catalog) may
|
||
be packed into object streams.
|
||
|
||
So the attachment can be located by scanning the bytes, without parsing the
|
||
PDF object graph at all.
|
||
|
||
## 3. Options
|
||
|
||
| Option | Protection | Cost | Verdict |
|
||
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
|
||
| A. Byte-scan reader: find `/Type /EmbeddedFile` streams, read `/Length` (direct integer, as pdf-lib writes it; fall back to `endstream` search), inflate with the existing capped `inflateFlate`, parse JSON; no `PDFDocument.load` on import | removes pdf-lib from the import path entirely: the only inflation is our capped one | one module (~120 lines) + tests on every fixture (v2, bare v3, envelope, object streams on/off, several streams, foreign PDF) | **recommended, can land now** |
|
||
| B. Export with `useObjectStreams: false` | the Filespec name, the Info keywords and the Filespec→EF link become visible to a byte scan, so the filename-first rule and the `attachment_stripped` detection keep working without the object graph; files a few KB larger | one save option; older PDFs keep their object streams, which A tolerates | **recommended with A** |
|
||
| C. Load in a Web Worker with a timeout and a memory budget | contains the damage (the tab survives, the worker dies) | worker bundling, message protocol, no memory cap API in browsers (only termination on timeout) | defence in depth, later |
|
||
| D. Pre-scan declared `/Length` of `/ObjStm` streams and refuse above a sum | partial: declared lengths are the compressed sizes, the expansion is unknown | small | insufficient alone |
|
||
| E. Replace pdf-lib's parser (fork `@cantoo/pdf-lib`, or `pdf.js` for reading) | maintained parser; pdf.js has its own limits | dependency change, larger bundle | evaluate for Step 14 (server-side import) |
|
||
|
||
With A, the import path reads bytes, finds at most a few streams, inflates one
|
||
under 16 MiB, parses JSON under the existing schema caps. pdf-lib stays on
|
||
the export side (trusted inputs) and in the `report:compare` CLI (operator
|
||
tool, Node, acceptable risk but same fix applies).
|
||
|
||
Edge cases A must handle: several `EmbeddedFile` streams (pick the one whose
|
||
Filespec name starts with `osteo-report` when visible, else the first whose
|
||
inflated content parses as a report or an envelope); `/Length` as an indirect
|
||
reference (pdf-lib writes direct lengths; other producers may not: fall back
|
||
to scanning for `endstream`); `\r\n` versus `\n` after `stream`; encrypted
|
||
PDFs (today loaded with `ignoreEncryption`: the attachment stream of an
|
||
encrypted file is encrypted too and would fail to inflate → `no_attachment`,
|
||
same outcome as today in practice); the 50 MiB file cap stays as the scan
|
||
bound.
|
||
|
||
## 4. Done 2026-10-11 (feature/step-06-pdf-archives, commit: see commit 60ee712)
|
||
|
||
1. Option A in `packages/pdf/src/attachment/scan.ts`
|
||
(`findEmbeddedFiles`, `readAttachmentBytes`, `hasArchiveStamp`), used by
|
||
`readReportArchive`; `readAttachedDocument(bytes)` takes the file bytes
|
||
and the `report:compare` CLI runs on the scanner too. No
|
||
`PDFDocument.load` remains under `packages/pdf/src` or `apps/app/scripts`.
|
||
2. Option B on export (`pdfDoc.save({ useObjectStreams: false })`): no visible
|
||
change; goldens unchanged (they record drawn text, images, links); the
|
||
envelope roundtrip tests prove the reader finds the attachment in both
|
||
layouts.
|
||
3. Tests (`packages/pdf/tests/scan.test.ts`): the three frozen fixtures (object
|
||
streams) and a fresh export; two embedded files with a decoy first;
|
||
indirect and wrong `/Length`; `\r\n` after `stream`; `/Type/EmbeddedFile`
|
||
without a space; hex and literal names and keywords; garbage inputs and
|
||
the 32-candidate cap; a 64 MiB object-stream bomb next to the real
|
||
attachment (report in milliseconds, RSS flat) and an `EmbeddedFile` bomb
|
||
(inflation cancelled at 16 MiB, `no_attachment`).
|
||
4. Docs: `docs/PDF-FORMAT.md` "Reading an archive" and "Known limits".
|
||
|
||
Known trade-off: a stamped PDF saved before this change (object streams)
|
||
hides its `/Keywords`, so a stripped attachment on such a file reads as
|
||
`no_attachment` instead of `attachment_stripped`.
|
||
|
||
## 5. Prerequisites before external users (Step 7+ sync, Step 14 worker)
|
||
|
||
- Option A merged and the CLI on the scanner; no `PDFDocument.load` on any
|
||
untrusted input in the browser.
|
||
- Done at R4 (2026-10-18, `docs/development/RUST.md` §10.3 to §10.5): the
|
||
reader runs in Rust inside a disposable worker; the inflate bound and the
|
||
hostile classes were measured in Chromium 156, Firefox 157 and WebKit 27.2
|
||
(headless): a 16 MiB + 1 bomb costs 63 MiB of linear memory and is
|
||
refused, 32 bombs 68, a 16 MiB token-dense document 16 MiB under the
|
||
400,000-value cap (784 MiB with it off), the 1 GiB module ceiling traps;
|
||
the application build answers the same outcome and message as the
|
||
TypeScript reader on 666 of 678 (file, engine) pairs, the other 12 being
|
||
the four declared hostile files. Device run (J1, `R4_J1.md`): the owner
|
||
reports the functional checks executed on 2026-10-18 with no failure;
|
||
the device list, versions, scenario coverage, timings and memory
|
||
observations are recorded in `R4_J1_RESULTS.md` as they are confirmed.
|
||
The gate below stays open: imports are not offered to external users.
|
||
- Done: the candidate CPU/byte budget on the scanner: one 16 MiB budget of
|
||
retained and parsed content shared by every candidate (a cancelled
|
||
inflation is not charged, so up to 32 oversized streams may each cost one
|
||
capped inflation of CPU, bounded, no memory retained), a monotonic
|
||
`endstream` search over candidates sorted by data offset and bounded to
|
||
the cap, and no JSON parse for a candidate not opening on `{` or nesting
|
||
brackets deeper than 64 levels (linear byte pre-scan, strings skipped).
|
||
- Option C or a server-side import (Step 14) with a hard memory and time
|
||
budget per file, for any path that still parses PDF objects.
|
||
- Dependency decision on pdf-lib (unmaintained since 2021): closed at R4.8
|
||
(2026-10-19): pdf-lib left the repository with the TypeScript engine; the
|
||
core's writer (`RUST.md` §10.7, §10.8) generates the PDFs and its reader
|
||
reads them; no PDF library remains in the application, its tests or the
|
||
tools (`apps/app/tests/unit/r4.census.test.ts` fails on a return).
|
||
- Report collection caps on read (observations, zones, findings per
|
||
observation) so a hostile report cannot freeze the editor before the
|
||
export's own caps refuse it.
|
||
- The remaining low findings of the Step 6 audits: `<use>` nesting budget in
|
||
the sanitiser (today bounded by the rasterizer on the PDF path; since R3 the
|
||
Rust reader refuses reference cycles and expansions over 50,000 elements,
|
||
`docs/development/RUST.md` §9.2, and the engine gets it when R4 switches it to
|
||
that reader: still open for the PDF path), the decoder memory
|
||
envelope of PNG logos (bounded by the 4096² cap, about 430 MB worst case
|
||
for four logos).
|
||
|
||
## 6. R4.9 classification (2026-10-20, `feature/rust-r4-pdf-core`, before the merge)
|
||
|
||
Every open point of §5 and of the R4 decisions BDR-058, BDR-059 and BDR-063,
|
||
in one of four classes: **fixed with proof** (a test or a measure in the
|
||
repository), **accepted temporarily** (bounded, with the justification and
|
||
what closes it), **blocking before merge** (the owner's own gate), **before
|
||
public exposure** (or before the upload it concerns is activated). The
|
||
validated refusals of hostile files stay as they are: no cap, threshold or
|
||
message was relaxed at R4.9, and no path rewrites, drops or alters a user's
|
||
file, draft or archive on an error (the file is read through the File API
|
||
only; a refused import leaves the tab's texts in place: app suites
|
||
`routes.HomeRoute.*`, `routes.coreStopped`, `state.coreFailures`).
|
||
|
||
| # | Point | Origin | Class | Proof, justification, or what closes it |
|
||
|---|---|---|---|---|
|
||
| 1 | PDF object graph parsed by pdf-lib on untrusted input (the 1000:1 object-stream expansion, §1) | §5 item 1 | fixed with proof | no PDF library left (R4.8): the Rust byte scanner locates the attachment (`crates/core/src/pdf/read/probe.rs`), cargo `pdf_scan`, `pdf_robustness`; `apps/app/tests/unit/r4.census.test.ts` fails on a return of pdf-lib or a TypeScript scanner |
|
||
| 2 | Inflation bound and the hostile classes (bombs, 32 candidates, token-dense 16 MiB, nested copies) | §5 item 2, BDR-058 | fixed with proof | one 16 MiB budget shared by the candidates, `MAX_JSON_VALUES` 400,000, depth 64, `--max-memory` 1 GiB trap; cargo `pdf_budget`, `pdf_inflate`, `pdf_json_js`; the browser run of the classes (RUST.md §10.4, three engines: ≤ 68 MiB of linear memory, the tab alive) |
|
||
| 3 | The same classes on the owner's devices (iPhone, low-end Android) | BDR-058, BDR-060, plan OD-6 | blocking before merge | J1.2, J1.10 on the iPhone and the Android phone (`R4_J1.md`); the computer run of 2026-10-18 passed with no failure; a device not run stays NON TESTÉ |
|
||
| 4 | The 1 GiB module ceiling is a safety net, not a tab guarantee | BDR-058 | accepted temporarily | documented as such (RUST.md §10.3); the worker is terminated on timeout or crash, so the main instance never holds the hostile tree; closes with the device run (J1.9, J1.10: no reload, no memory warning) |
|
||
| 5 | WebKit, a 50 MiB file: 145 MiB of content growth against 76 for the deleted TypeScript reader | BDR-059 | accepted temporarily | the owner kept the 50 MiB cap and refused a streaming scanner at R4.3; the worker reads the file in 1 MiB chunks into linear memory (one copy; the 324 MiB worker-read figure is gone); closes with J1.8 on the iPhone (the tab survives, time noted) or a later streaming scanner |
|
||
| 6 | WebKit leak criterion (≤ 10 MiB recurring over ≥ 5 imports) not demonstrable headless | BDR-058, BDR-059 | accepted temporarily | headless WebKit has no forced GC; the one-page series levels off and a no-import control grows the same way with navigation (RUST.md §10.5); closes with J1.9 on the iPhone (six L imports, no reload) |
|
||
| 7 | Chromium per-export RSS growth ≈ 20 MiB on L (R4.6, same figures in the TypeScript control, plateau after 8) | BDR-063 | accepted temporarily | measured at R4.9 (RUST.md §10.11): the module's linear memory is flat after the init export, every canvas, Blob and object URL is released; a raster-only control with no core call grows the same 20 MiB per step to the same plateau (436 against 437 MiB after 16 steps, last five within 10 MiB): the browser's raster path, bounded; closes with J2.6 / J2.7 on the phones; correction only if a device shows it (one reused canvas, measured first) |
|
||
| 8 | A hidden tab pauses the task timer: a hostile file keeps the worker busy in the background | RUST.md §10.5 | accepted temporarily | by design (plan §B.1.5 item 3: a phone that switches apps must not lose a legitimate import); the core's own bounds (item 2) cap the work; the worker dies with the tab |
|
||
| 9 | Timers (10 s start, 20 s task) are hypotheses | RUST.md §10.5, plan §B.3 | blocking before merge | J1.12 on each device gives the slowest legitimate import; the rule (≥ 10 × that) confirms or moves the value before the merge |
|
||
| 10 | OD-4: an overwritten duplicate key holding a lone surrogate escape is refused where JSON.stringify once dropped it | BDR-057, BDR-058 | fixed with proof | declared (plan §A.3), refused with `error_lone_surrogate`, the file untouched; cargo `pdf_import` (O8 declared rows), app `routes.HomeRoute.messages`; no Faunosteo build writes duplicate keys |
|
||
| 11 | Worker crash, timeout and resource overflow told apart from a corrupt file | BDR-058, OD-5 | fixed with proof | closed outcome kinds (`core/worker/outcome.ts`), messages pinned by `routes.HomeRoute.import` and `routes.HomeRoute.rescueCopy`; a technical failure never says the PDF is not a Faunosteo file |
|
||
| 12 | Report collection caps on read (observations, zones, findings per observation) | §5 item "collection caps" | before public exposure | out of R4 (plan A11); today bounded by `MAX_JSON_VALUES` (400,000 values, 10 × the CAP envelope of 500 observations: a report of about 5,000 observations at the CAP density at most) and by the export's caps (500 × 8 views, closed kind); a dense hostile report slows the editor but cannot exhaust it |
|
||
| 13 | `<use>` nesting and reference budget on the PDF path | §5 "remaining low findings" | fixed with proof | every SVG the export composes went through the core's one reader (R3: cycles refused, expansion > 50,000 elements refused, prescan caps); `ExportInput` runs `read_svg` per asset (R4.4); cargo `svg_sanitiser`, `pdf_layout` |
|
||
| 14 | PNG logo decoder envelope (4096² cap, four logos) | §5 "remaining low findings" | fixed with proof | the application exports the bundled logos only (`src/assets/brand`, never the archive's branding: `export_from_envelope` is cargo-only); the gate refuses above 8192 px a side, 4096² pixels or 16 MiB of raw scanlines before any allocation (`PNG_MAX_*`, cargo `pdf_png` 108 cases + budget cases); the session frees the planes at `add_raster` |
|
||
| 15 | Option C or a server-side import for any path that still parses PDF objects | §5 item 4 | before public exposure | no such path exists today; applies to Step 14 if a server-side reader is introduced |
|
||
| 16 | pdf-lib (unmaintained since 2021) | §5 item 5 | fixed with proof | left the repository at R4.8; `r4.census` |
|
||
| 17 | Stamped PDFs saved with object streams (before Step 6) read a stripped attachment as `no_attachment` | §4 trade-off | accepted temporarily | pre-Step-6 files only, same outcome as a foreign PDF, no data at stake; documented in PDF-FORMAT |
|
||
| 18 | Imports offered to external users | §5 gate | before public exposure | the gate stays closed: the import is a local-tool feature; opening it needs items 3, 9 and 12 closed and a fresh audit of the worker path |
|
||
| 19 | J2 on the owner's devices (export FR / EN, views and markers, attachment, round trip, L on the phone, repetition) | BDR-063, BDR-060, plan A.4 | blocking before merge | `R4_J2.md` J2.1 to J2.14 on the OD-6 list; every cell NON TESTÉ today; headless runs (RUST.md §10.8, §10.11) never fill a cell |
|
||
| 20 | WebKit one-off worker `error` event seen headless at R4.2 / R4.3 | BDR-058 | accepted temporarily | not blocking per BDR-058: kept in the record, never masked (the client classifies every `error` event as a crash, `core/worker/client.ts`), the repetition series showed no recurrence (RUST.md §10.5); closes with J1.9 / J1.10 on the iPhone (Safari iOS) |
|
||
|
||
Owner reading at the R4.9 checkpoint: no item in this table is a code
|
||
change pending in R4; items 3, 9 and 19 are the device runs (J1 and J2),
|
||
items 12, 15 and 18 are the public-exposure gate, item 7 is measured and bounded (the browser's raster path), its device check is J2.6 / J2.7.
|