fix(pdf): bounded attachment extraction by byte scan — no pdf-lib parse on import, export without object streams (BLK-009)

This commit is contained in:
bchanot committed 2026-10-11 16:42:45 +02:00
1 parent 62b8a43e09
commit 60ee71208f
14 files changed
+1054 -198

No files matched your search

+86 -45
View File
@@ -35,6 +35,10 @@ branding have changed since the export. Design record:
- Document information: title, author, subject, dates = `exportedAt`, and
`Keywords: faunosteo-archive/1` so a PDF whose attachment was stripped by a
viewer is still recognised.
- Saved without object streams (`useObjectStreams: false`) since BLK-009: the
Filespec, its file name and the Info dictionary stay in clear, where the
byte-scan reader below can see them. Files produced earlier pack them into
an `/ObjStm`; their attachment stream is still in clear and still imports.
## Attachment: the report archive envelope (v1)
@@ -53,18 +57,38 @@ schema):
"report": { "version": 3, "...": "Report v3 exactly as exported" },
"sheetSnapshot": {
"status": "draft",
"ref": { "sheetId": "…", "versionId": null, "documentSha256": "…", "schemaVersion": 2, "speciesId": "equine" },
"document": { "schema_version": 2, "...": "lean, valid anatomical sheet document" },
"assets": [ { "svgAssetId": "…", "sha256": "…", "svg": "<svg …>" } ]
"ref": {
"sheetId": "…",
"versionId": null,
"documentSha256": "…",
"schemaVersion": 2,
"speciesId": "equine"
},
"document": {
"schema_version": 2,
"...": "lean, valid anatomical sheet document"
},
"assets": [{ "svgAssetId": "…", "sha256": "…", "svg": "<svg …>" }]
},
"branding": {
"siteUrl": "https://…",
"headerLogo": { "kind": "png", "base64": "…" },
"footerLogo": { "kind": "png", "base64": "…" },
"poweredBy": { "logo": { "kind": "png", "base64": "…" }, "url": "https://…", "fallbackText": "…" },
"partner": { "logo": { "kind": "svg", "svg": "<svg …>" }, "url": "https://…", "fallbackText": "…" }
"poweredBy": {
"logo": { "kind": "png", "base64": "…" },
"url": "https://…",
"fallbackText": "…"
},
"partner": {
"logo": { "kind": "svg", "svg": "<svg …>" },
"url": "https://…",
"fallbackText": "…"
}
},
"labels": { "fr": { "pdf.section.zones": "ZONES", "...": "25 keys" }, "en": { "...": "" } }
"labels": {
"fr": { "pdf.section.zones": "ZONES", "...": "25 keys" },
"en": { "...": "" }
}
}
```
@@ -74,14 +98,14 @@ The snapshot is the sheet the bilan was built on, reduced to what the bilan
needs, as a **valid** anatomical sheet document (schema v2, same integrity
rules as the catalog):
| Part | Content |
|---|---|
| `ref` | sheet id, version id, sha256 of the full catalog document (provenance pointer), schema version, species |
| `document.views`, `annotations`, `categories`, `vertebral_config` | all of them: geometry, colours, names, anatomical relations (`finding_zone_id`, `vertebral_region_code`, `opens_view_id`, laterality) |
| `document.finding_zones` | every zone with its name; for the zones the report's findings use (resolved or `unresolved`), the categories pruned to the referenced paths with the name and vet flag of every node on the path; other zones keep `categories: []` |
| `document.references` | emptied (bibliography) |
| `document.name`, `description`, `scope_disclaimer`, `attribution` | kept |
| `assets` | the SVG text of every asset a view reaches, with the catalog's content hash when known |
| Part | Content |
| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ref` | sheet id, version id, sha256 of the full catalog document (provenance pointer), schema version, species |
| `document.views`, `annotations`, `categories`, `vertebral_config` | all of them: geometry, colours, names, anatomical relations (`finding_zone_id`, `vertebral_region_code`, `opens_view_id`, laterality) |
| `document.finding_zones` | every zone with its name; for the zones the report's findings use (resolved or `unresolved`), the categories pruned to the referenced paths with the name and vet flag of every node on the path; other zones keep `categories: []` |
| `document.references` | emptied (bibliography) |
| `document.name`, `description`, `scope_disclaimer`, `attribution` | kept |
| `assets` | the SVG text of every asset a view reaches, with the catalog's content hash when known |
`branding` is the exact set of images, URLs and credit texts used at export;
the application builds it from logos bundled at build time
@@ -98,10 +122,10 @@ Invariant: `report.sheet.documentSha256 === sheetSnapshot.ref.documentSha256`
Measured on the demo report (6 observations, both views, equine draft):
| Branding | Stored in the PDF (Flate) | Raw JSON |
|---|---|---|
| no logos | 55 KB | 189 KB |
| full branding (3 PNG logos + partner SVG) | 136 KB | 306 KB |
| Branding | Stored in the PDF (Flate) | Raw JSON |
| ----------------------------------------- | ------------------------- | -------- |
| no logos | 55 KB | 189 KB |
| full branding (3 PNG logos + partner SVG) | 136 KB | 306 KB |
The drawing is the largest part (47 KB stored); the archived PNG logos add
about 80 KB stored. The 250 KB target applies to the stored size.
@@ -111,11 +135,22 @@ about 80 KB stored. The 250 KB target applies to the stored size.
`readReportArchive(pdfBytes, migrationMap)` reads in layers so the report is
never lost because of its wrapper:
1. The PDF must load (`invalid_pdf`, also above 50 MiB).
2. The attachment named `osteo-report*` is taken first, else the first
embedded file; Flate streams are inflated under a 16 MiB cap. No
attachment: `attachment_stripped` when the PDF carries the Faunosteo
keyword, else `no_attachment`.
1. The file must weigh at most 50 MiB and carry `%PDF-` within its first
KiB (`invalid_pdf`). Nothing else of the PDF syntax is required.
2. The attachment is located by a byte scan, never by parsing the PDF object
graph (BLK-009): every `/Type /EmbeddedFile` stream dictionary in clear
(a stream object can never sit inside an object stream) is a candidate,
with its direct `/Length` or, failing that, the distance to `endstream`;
a `/Filespec` dictionary in clear whose `/EF /F` points at the stream
gives it a name. Candidates named `osteo-report*` come first, then file
order; Flate data is inflated by the platform inflater under a 16 MiB cap
(cancelled beyond it), and the first candidate whose text is a JSON
object with a numeric `envelopeVersion` or `version` wins. Object
streams, cross-reference streams and page contents are never inflated.
No candidate: `attachment_stripped` when a `/Keywords` string in clear
contains `faunosteo-archive`, else `no_attachment`. A stamped file saved
with object streams (before BLK-009) hides its stamp and reads as
`no_attachment`.
3. The report is located (`data.report` inside an envelope, `data` for a
bare attachment) and read with the report reader: v3 as is, v2 migrated
through the sheet's `report-migration-map.json` (items the map cannot
@@ -128,12 +163,12 @@ never lost because of its wrapper:
closed reason (`schema`, `caps`, `integrity`, `sheet_mismatch`) and keeps
the report.
| Archive | Produced by | Read as |
|---|---|---|
| bare Report v2 | application before Step 5 | `origin 'v2'`, migrated, unresolved counts |
| bare Report v3 | Step 5 | `origin 'bare_v3'`, no archive |
| envelope v1 | Step 6 | `origin 'envelope_v1'`, archive kept or `snapshotRejected` |
| envelope v≥2 | future | report kept when its version is known, snapshot dropped |
| Archive | Produced by | Read as |
| -------------- | ------------------------- | ---------------------------------------------------------- |
| bare Report v2 | application before Step 5 | `origin 'v2'`, migrated, unresolved counts |
| bare Report v3 | Step 5 | `origin 'bare_v3'`, no archive |
| envelope v1 | Step 6 | `origin 'envelope_v1'`, archive kept or `snapshotRejected` |
| envelope v≥2 | future | report kept when its version is known, snapshot dropped |
Readers never drop a schema version: sheet document schema v2 stays readable
as long as envelope v1 archives exist, as Report v2 does today. A rollback to
@@ -156,17 +191,17 @@ Two operations are distinct:
## Limits and sanitising
| Item | Cap |
|---|---|
| PDF file | 50 MiB |
| attachment, raw and inflated | 16 MiB |
| snapshot assets | ≤ 4, ≤ 2 MiB each, ≤ 6 MiB total |
| snapshot document | ≤ 64 views, ≤ 1,024 annotations, ≤ 8,192 finding nodes, depth ≤ 8, names ≤ 512 chars |
| branding | PNG base64 ≤ 512 KiB, SVG ≤ 256 KiB, http(s) URLs ≤ 2,048 chars |
| branding URL | `http(s)://` then printable ASCII only, without `"` `'` `(` `)` `<` `>` `\`, no space or control character |
| branding PNG | valid signature, one IHDR (13 bytes) first with a colour type and bit depth the spec allows, compression and filter method 0, no interlace; each side 1 to 8,192 px, ≤ 16,777,216 px (4096²), expected raw scanline size ≤ 16 MiB; then only `PLTE` `tRNS` `IDAT` `gAMA` `cHRM` `sRGB` `sBIT` `pHYs` `bKGD`, no chunk past the buffer, an empty `IEND` as the last bytes; the IDAT stream inflated natively and yielding exactly the raw scanline size |
| report at export | ≤ 500 observations, ≤ 8 views per observation |
| SVG sanitiser | ≤ 20,000 elements, nesting ≤ 64, ≤ 256 `<use>`, blur `stdDeviation` ≤ 100 |
| Item | Cap |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| PDF file | 50 MiB |
| attachment, raw and inflated | 16 MiB |
| snapshot assets | ≤ 4, ≤ 2 MiB each, ≤ 6 MiB total |
| snapshot document | ≤ 64 views, ≤ 1,024 annotations, ≤ 8,192 finding nodes, depth ≤ 8, names ≤ 512 chars |
| branding | PNG base64 ≤ 512 KiB, SVG ≤ 256 KiB, http(s) URLs ≤ 2,048 chars |
| branding URL | `http(s)://` then printable ASCII only, without `"` `'` `(` `)` `<` `>` `\`, no space or control character |
| branding PNG | valid signature, one IHDR (13 bytes) first with a colour type and bit depth the spec allows, compression and filter method 0, no interlace; each side 1 to 8,192 px, ≤ 16,777,216 px (4096²), expected raw scanline size ≤ 16 MiB; then only `PLTE` `tRNS` `IDAT` `gAMA` `cHRM` `sRGB` `sBIT` `pHYs` `bKGD`, no chunk past the buffer, an empty `IEND` as the last bytes; the IDAT stream inflated natively and yielding exactly the raw scanline size |
| report at export | ≤ 500 observations, ≤ 8 views per observation |
| SVG sanitiser | ≤ 20,000 elements, nesting ≤ 64, ≤ 256 `<use>`, blur `stdDeviation` ≤ 100 |
Every SVG that reaches the engine (sheet asset or branding) is untrusted: a
text containing `<!DOCTYPE` or `<!ENTITY` is refused; the parsed tree is
@@ -223,11 +258,17 @@ fallback text and the export reports `logo_failed`.
### Known limits
- pdf-lib inflates PDF object streams without a cap when loading: a crafted
PDF of a few hundred KB can expand to hundreds of MB in the tab before the
attachment is even looked at. The 50 MiB file cap bounds the input, not
the expansion. Mitigation planned: parse in a Worker, or locate the
attachment without a full load.
- pdf-lib is out of the import path (BLK-009): the browser and the
`report:compare` CLI read the attachment by byte scan, so a crafted object
stream is never inflated (tested with a 64 MiB bomb: milliseconds, no
memory growth). pdf-lib stays on the export side, on trusted inputs, and
in tests that inspect exported files. What remains before external users
(`docs/development/PDF_IMPORT_HARDENING.md` §5): a Worker or server-side
budget for any path that still parses PDF objects, the pdf-lib dependency
decision, and read-side caps on report collections.
- The scanner reads what is in clear: an attachment whose stream dictionary
is malformed beyond `/Length` and `endstream`, or an encrypted file, yields
no candidate (`no_attachment`), as the previous reader did in practice.
- `<use>` expansion is bounded by the rasterizer, not by the sanitiser,
which only counts the `<use>` elements (≤ 256). resvg (tests) refuses an
oversized tree; the production path is the browser rasterizer
+30 -22
View File
@@ -1,8 +1,9 @@
# PDF import hardening — BLK-009 analysis
Status 2026-10-11: analysis requested by the user after Step 6. Nothing here is
implemented; §4 says what can land now and §5 what must land before the
import is opened to external users.
Status 2026-10-11: analysis requested by the user after Step 6. §4 landed the
same day on `feature/step-06-pdf-archives` (options A + B, see §4 for the
commit); §5 lists what must land before the import is opened to external
users.
## 1. The problem
@@ -36,13 +37,13 @@ PDF object graph at all.
## 3. Options
| Option | Protection | Cost | Verdict |
|---|---|---|---|
| A. Byte-scan reader: find `/Type /EmbeddedFile` streams, read `/Length` (direct integer, as pdf-lib writes it; fall back to `endstream` search), inflate with the existing capped `inflateFlate`, parse JSON; no `PDFDocument.load` on import | removes pdf-lib from the import path entirely: the only inflation is our capped one | one module (~120 lines) + tests on every fixture (v2, bare v3, envelope, object streams on/off, several streams, foreign PDF) | **recommended, can land now** |
| B. Export with `useObjectStreams: false` | the Filespec name, the Info keywords and the Filespec→EF link become visible to a byte scan, so the filename-first rule and the `attachment_stripped` detection keep working without the object graph; files a few KB larger | one save option; older PDFs keep their object streams, which A tolerates | **recommended with A** |
| C. Load in a Web Worker with a timeout and a memory budget | contains the damage (the tab survives, the worker dies) | worker bundling, message protocol, no memory cap API in browsers (only termination on timeout) | defence in depth, later |
| D. Pre-scan declared `/Length` of `/ObjStm` streams and refuse above a sum | partial: declared lengths are the compressed sizes, the expansion is unknown | small | insufficient alone |
| E. Replace pdf-lib's parser (fork `@cantoo/pdf-lib`, or `pdf.js` for reading) | maintained parser; pdf.js has its own limits | dependency change, larger bundle | evaluate for Step 14 (server-side import) |
| Option | Protection | Cost | Verdict |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| A. Byte-scan reader: find `/Type /EmbeddedFile` streams, read `/Length` (direct integer, as pdf-lib writes it; fall back to `endstream` search), inflate with the existing capped `inflateFlate`, parse JSON; no `PDFDocument.load` on import | removes pdf-lib from the import path entirely: the only inflation is our capped one | one module (~120 lines) + tests on every fixture (v2, bare v3, envelope, object streams on/off, several streams, foreign PDF) | **recommended, can land now** |
| B. Export with `useObjectStreams: false` | the Filespec name, the Info keywords and the Filespec→EF link become visible to a byte scan, so the filename-first rule and the `attachment_stripped` detection keep working without the object graph; files a few KB larger | one save option; older PDFs keep their object streams, which A tolerates | **recommended with A** |
| C. Load in a Web Worker with a timeout and a memory budget | contains the damage (the tab survives, the worker dies) | worker bundling, message protocol, no memory cap API in browsers (only termination on timeout) | defence in depth, later |
| D. Pre-scan declared `/Length` of `/ObjStm` streams and refuse above a sum | partial: declared lengths are the compressed sizes, the expansion is unknown | small | insufficient alone |
| E. Replace pdf-lib's parser (fork `@cantoo/pdf-lib`, or `pdf.js` for reading) | maintained parser; pdf.js has its own limits | dependency change, larger bundle | evaluate for Step 14 (server-side import) |
With A, the import path reads bytes, finds at most a few streams, inflates one
under 16 MiB, parses JSON under the existing schema caps. pdf-lib stays on
@@ -59,22 +60,29 @@ encrypted file is encrypted too and would fail to inflate → `no_attachment`,
same outcome as today in practice); the 50 MiB file cap stays as the scan
bound.
## 4. What can be done now (before merge of Step 6, or as the next chore)
## 4. Done 2026-10-11 (feature/step-06-pdf-archives, commit: see `git log -- packages/pdf/src/attachment/scan.ts`)
1. Option A in `packages/pdf/src/attachment/scan.ts`, used by
`readReportArchive`; `readAttachedDocument(pdfDoc)` kept for the CLI until
it moves to the scanner too.
1. Option A in `packages/pdf/src/attachment/scan.ts`
(`findEmbeddedFiles`, `readAttachmentBytes`, `hasArchiveStamp`), used by
`readReportArchive`; `readAttachedDocument(bytes)` takes the file bytes
and the `report:compare` CLI runs on the scanner too. No
`PDFDocument.load` remains under `packages/pdf/src` or `apps/app/scripts`.
2. Option B on export (`pdfDoc.save({ useObjectStreams: false })`): no visible
change; goldens unaffected (they record drawn text, images, links); the
envelope roundtrip tests prove the reader still finds the attachment in
both layouts.
3. Tests: every existing import fixture through the scanner; a crafted
object-stream bomb (the audit's recipe: a few hundred KB inflating to
hundreds of MB) imports in milliseconds without inflating it; a PDF with
two embedded files; a PDF whose `/Length` is indirect.
change; goldens unchanged (they record drawn text, images, links); the
envelope roundtrip tests prove the reader finds the attachment in both
layouts.
3. Tests (`packages/pdf/tests/scan.test.ts`): the three frozen fixtures (object
streams) and a fresh export; two embedded files with a decoy first;
indirect and wrong `/Length`; `\r\n` after `stream`; `/Type/EmbeddedFile`
without a space; hex and literal names and keywords; garbage inputs and
the 32-candidate cap; a 64 MiB object-stream bomb next to the real
attachment (report in milliseconds, RSS flat) and an `EmbeddedFile` bomb
(inflation cancelled at 16 MiB, `no_attachment`).
4. Docs: `docs/PDF-FORMAT.md` "Reading an archive" and "Known limits".
Estimated size: one executor phase, about 300 lines including tests.
Known trade-off: a stamped PDF saved before this change (object streams)
hides its `/Keywords`, so a stripped attachment on such a file reads as
`no_attachment` instead of `attachment_stripped`.
## 5. Prerequisites before external users (Step 7+ sync, Step 14 worker)