fix(pdf): bounded attachment extraction by byte scan — no pdf-lib parse on import, export without object streams (BLK-009)
This commit is contained in:
1 parent
62b8a43e09
commit
60ee71208f
14 files changed
+1054
-198
No files matched your search
+86
-45
@@ -35,6 +35,10 @@ branding have changed since the export. Design record:
|
||||
- Document information: title, author, subject, dates = `exportedAt`, and
|
||||
`Keywords: faunosteo-archive/1` so a PDF whose attachment was stripped by a
|
||||
viewer is still recognised.
|
||||
- Saved without object streams (`useObjectStreams: false`) since BLK-009: the
|
||||
Filespec, its file name and the Info dictionary stay in clear, where the
|
||||
byte-scan reader below can see them. Files produced earlier pack them into
|
||||
an `/ObjStm`; their attachment stream is still in clear and still imports.
|
||||
|
||||
## Attachment: the report archive envelope (v1)
|
||||
|
||||
@@ -53,18 +57,38 @@ schema):
|
||||
"report": { "version": 3, "...": "Report v3 exactly as exported" },
|
||||
"sheetSnapshot": {
|
||||
"status": "draft",
|
||||
"ref": { "sheetId": "…", "versionId": null, "documentSha256": "…", "schemaVersion": 2, "speciesId": "equine" },
|
||||
"document": { "schema_version": 2, "...": "lean, valid anatomical sheet document" },
|
||||
"assets": [ { "svgAssetId": "…", "sha256": "…", "svg": "<svg …>" } ]
|
||||
"ref": {
|
||||
"sheetId": "…",
|
||||
"versionId": null,
|
||||
"documentSha256": "…",
|
||||
"schemaVersion": 2,
|
||||
"speciesId": "equine"
|
||||
},
|
||||
"document": {
|
||||
"schema_version": 2,
|
||||
"...": "lean, valid anatomical sheet document"
|
||||
},
|
||||
"assets": [{ "svgAssetId": "…", "sha256": "…", "svg": "<svg …>" }]
|
||||
},
|
||||
"branding": {
|
||||
"siteUrl": "https://…",
|
||||
"headerLogo": { "kind": "png", "base64": "…" },
|
||||
"footerLogo": { "kind": "png", "base64": "…" },
|
||||
"poweredBy": { "logo": { "kind": "png", "base64": "…" }, "url": "https://…", "fallbackText": "…" },
|
||||
"partner": { "logo": { "kind": "svg", "svg": "<svg …>" }, "url": "https://…", "fallbackText": "…" }
|
||||
"poweredBy": {
|
||||
"logo": { "kind": "png", "base64": "…" },
|
||||
"url": "https://…",
|
||||
"fallbackText": "…"
|
||||
},
|
||||
"partner": {
|
||||
"logo": { "kind": "svg", "svg": "<svg …>" },
|
||||
"url": "https://…",
|
||||
"fallbackText": "…"
|
||||
}
|
||||
},
|
||||
"labels": { "fr": { "pdf.section.zones": "ZONES", "...": "25 keys" }, "en": { "...": "" } }
|
||||
"labels": {
|
||||
"fr": { "pdf.section.zones": "ZONES", "...": "25 keys" },
|
||||
"en": { "...": "" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -74,14 +98,14 @@ The snapshot is the sheet the bilan was built on, reduced to what the bilan
|
||||
needs, as a **valid** anatomical sheet document (schema v2, same integrity
|
||||
rules as the catalog):
|
||||
|
||||
| Part | Content |
|
||||
|---|---|
|
||||
| `ref` | sheet id, version id, sha256 of the full catalog document (provenance pointer), schema version, species |
|
||||
| `document.views`, `annotations`, `categories`, `vertebral_config` | all of them: geometry, colours, names, anatomical relations (`finding_zone_id`, `vertebral_region_code`, `opens_view_id`, laterality) |
|
||||
| `document.finding_zones` | every zone with its name; for the zones the report's findings use (resolved or `unresolved`), the categories pruned to the referenced paths with the name and vet flag of every node on the path; other zones keep `categories: []` |
|
||||
| `document.references` | emptied (bibliography) |
|
||||
| `document.name`, `description`, `scope_disclaimer`, `attribution` | kept |
|
||||
| `assets` | the SVG text of every asset a view reaches, with the catalog's content hash when known |
|
||||
| Part | Content |
|
||||
| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `ref` | sheet id, version id, sha256 of the full catalog document (provenance pointer), schema version, species |
|
||||
| `document.views`, `annotations`, `categories`, `vertebral_config` | all of them: geometry, colours, names, anatomical relations (`finding_zone_id`, `vertebral_region_code`, `opens_view_id`, laterality) |
|
||||
| `document.finding_zones` | every zone with its name; for the zones the report's findings use (resolved or `unresolved`), the categories pruned to the referenced paths with the name and vet flag of every node on the path; other zones keep `categories: []` |
|
||||
| `document.references` | emptied (bibliography) |
|
||||
| `document.name`, `description`, `scope_disclaimer`, `attribution` | kept |
|
||||
| `assets` | the SVG text of every asset a view reaches, with the catalog's content hash when known |
|
||||
|
||||
`branding` is the exact set of images, URLs and credit texts used at export;
|
||||
the application builds it from logos bundled at build time
|
||||
@@ -98,10 +122,10 @@ Invariant: `report.sheet.documentSha256 === sheetSnapshot.ref.documentSha256`
|
||||
|
||||
Measured on the demo report (6 observations, both views, equine draft):
|
||||
|
||||
| Branding | Stored in the PDF (Flate) | Raw JSON |
|
||||
|---|---|---|
|
||||
| no logos | 55 KB | 189 KB |
|
||||
| full branding (3 PNG logos + partner SVG) | 136 KB | 306 KB |
|
||||
| Branding | Stored in the PDF (Flate) | Raw JSON |
|
||||
| ----------------------------------------- | ------------------------- | -------- |
|
||||
| no logos | 55 KB | 189 KB |
|
||||
| full branding (3 PNG logos + partner SVG) | 136 KB | 306 KB |
|
||||
|
||||
The drawing is the largest part (47 KB stored); the archived PNG logos add
|
||||
about 80 KB stored. The 250 KB target applies to the stored size.
|
||||
@@ -111,11 +135,22 @@ about 80 KB stored. The 250 KB target applies to the stored size.
|
||||
`readReportArchive(pdfBytes, migrationMap)` reads in layers so the report is
|
||||
never lost because of its wrapper:
|
||||
|
||||
1. The PDF must load (`invalid_pdf`, also above 50 MiB).
|
||||
2. The attachment named `osteo-report*` is taken first, else the first
|
||||
embedded file; Flate streams are inflated under a 16 MiB cap. No
|
||||
attachment: `attachment_stripped` when the PDF carries the Faunosteo
|
||||
keyword, else `no_attachment`.
|
||||
1. The file must weigh at most 50 MiB and carry `%PDF-` within its first
|
||||
KiB (`invalid_pdf`). Nothing else of the PDF syntax is required.
|
||||
2. The attachment is located by a byte scan, never by parsing the PDF object
|
||||
graph (BLK-009): every `/Type /EmbeddedFile` stream dictionary in clear
|
||||
(a stream object can never sit inside an object stream) is a candidate,
|
||||
with its direct `/Length` or, failing that, the distance to `endstream`;
|
||||
a `/Filespec` dictionary in clear whose `/EF /F` points at the stream
|
||||
gives it a name. Candidates named `osteo-report*` come first, then file
|
||||
order; Flate data is inflated by the platform inflater under a 16 MiB cap
|
||||
(cancelled beyond it), and the first candidate whose text is a JSON
|
||||
object with a numeric `envelopeVersion` or `version` wins. Object
|
||||
streams, cross-reference streams and page contents are never inflated.
|
||||
No candidate: `attachment_stripped` when a `/Keywords` string in clear
|
||||
contains `faunosteo-archive`, else `no_attachment`. A stamped file saved
|
||||
with object streams (before BLK-009) hides its stamp and reads as
|
||||
`no_attachment`.
|
||||
3. The report is located (`data.report` inside an envelope, `data` for a
|
||||
bare attachment) and read with the report reader: v3 as is, v2 migrated
|
||||
through the sheet's `report-migration-map.json` (items the map cannot
|
||||
@@ -128,12 +163,12 @@ never lost because of its wrapper:
|
||||
closed reason (`schema`, `caps`, `integrity`, `sheet_mismatch`) and keeps
|
||||
the report.
|
||||
|
||||
| Archive | Produced by | Read as |
|
||||
|---|---|---|
|
||||
| bare Report v2 | application before Step 5 | `origin 'v2'`, migrated, unresolved counts |
|
||||
| bare Report v3 | Step 5 | `origin 'bare_v3'`, no archive |
|
||||
| envelope v1 | Step 6 | `origin 'envelope_v1'`, archive kept or `snapshotRejected` |
|
||||
| envelope v≥2 | future | report kept when its version is known, snapshot dropped |
|
||||
| Archive | Produced by | Read as |
|
||||
| -------------- | ------------------------- | ---------------------------------------------------------- |
|
||||
| bare Report v2 | application before Step 5 | `origin 'v2'`, migrated, unresolved counts |
|
||||
| bare Report v3 | Step 5 | `origin 'bare_v3'`, no archive |
|
||||
| envelope v1 | Step 6 | `origin 'envelope_v1'`, archive kept or `snapshotRejected` |
|
||||
| envelope v≥2 | future | report kept when its version is known, snapshot dropped |
|
||||
|
||||
Readers never drop a schema version: sheet document schema v2 stays readable
|
||||
as long as envelope v1 archives exist, as Report v2 does today. A rollback to
|
||||
@@ -156,17 +191,17 @@ Two operations are distinct:
|
||||
|
||||
## Limits and sanitising
|
||||
|
||||
| Item | Cap |
|
||||
|---|---|
|
||||
| PDF file | 50 MiB |
|
||||
| attachment, raw and inflated | 16 MiB |
|
||||
| snapshot assets | ≤ 4, ≤ 2 MiB each, ≤ 6 MiB total |
|
||||
| snapshot document | ≤ 64 views, ≤ 1,024 annotations, ≤ 8,192 finding nodes, depth ≤ 8, names ≤ 512 chars |
|
||||
| branding | PNG base64 ≤ 512 KiB, SVG ≤ 256 KiB, http(s) URLs ≤ 2,048 chars |
|
||||
| branding URL | `http(s)://` then printable ASCII only, without `"` `'` `(` `)` `<` `>` `\`, no space or control character |
|
||||
| branding PNG | valid signature, one IHDR (13 bytes) first with a colour type and bit depth the spec allows, compression and filter method 0, no interlace; each side 1 to 8,192 px, ≤ 16,777,216 px (4096²), expected raw scanline size ≤ 16 MiB; then only `PLTE` `tRNS` `IDAT` `gAMA` `cHRM` `sRGB` `sBIT` `pHYs` `bKGD`, no chunk past the buffer, an empty `IEND` as the last bytes; the IDAT stream inflated natively and yielding exactly the raw scanline size |
|
||||
| report at export | ≤ 500 observations, ≤ 8 views per observation |
|
||||
| SVG sanitiser | ≤ 20,000 elements, nesting ≤ 64, ≤ 256 `<use>`, blur `stdDeviation` ≤ 100 |
|
||||
| Item | Cap |
|
||||
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| PDF file | 50 MiB |
|
||||
| attachment, raw and inflated | 16 MiB |
|
||||
| snapshot assets | ≤ 4, ≤ 2 MiB each, ≤ 6 MiB total |
|
||||
| snapshot document | ≤ 64 views, ≤ 1,024 annotations, ≤ 8,192 finding nodes, depth ≤ 8, names ≤ 512 chars |
|
||||
| branding | PNG base64 ≤ 512 KiB, SVG ≤ 256 KiB, http(s) URLs ≤ 2,048 chars |
|
||||
| branding URL | `http(s)://` then printable ASCII only, without `"` `'` `(` `)` `<` `>` `\`, no space or control character |
|
||||
| branding PNG | valid signature, one IHDR (13 bytes) first with a colour type and bit depth the spec allows, compression and filter method 0, no interlace; each side 1 to 8,192 px, ≤ 16,777,216 px (4096²), expected raw scanline size ≤ 16 MiB; then only `PLTE` `tRNS` `IDAT` `gAMA` `cHRM` `sRGB` `sBIT` `pHYs` `bKGD`, no chunk past the buffer, an empty `IEND` as the last bytes; the IDAT stream inflated natively and yielding exactly the raw scanline size |
|
||||
| report at export | ≤ 500 observations, ≤ 8 views per observation |
|
||||
| SVG sanitiser | ≤ 20,000 elements, nesting ≤ 64, ≤ 256 `<use>`, blur `stdDeviation` ≤ 100 |
|
||||
|
||||
Every SVG that reaches the engine (sheet asset or branding) is untrusted: a
|
||||
text containing `<!DOCTYPE` or `<!ENTITY` is refused; the parsed tree is
|
||||
@@ -223,11 +258,17 @@ fallback text and the export reports `logo_failed`.
|
||||
|
||||
### Known limits
|
||||
|
||||
- pdf-lib inflates PDF object streams without a cap when loading: a crafted
|
||||
PDF of a few hundred KB can expand to hundreds of MB in the tab before the
|
||||
attachment is even looked at. The 50 MiB file cap bounds the input, not
|
||||
the expansion. Mitigation planned: parse in a Worker, or locate the
|
||||
attachment without a full load.
|
||||
- pdf-lib is out of the import path (BLK-009): the browser and the
|
||||
`report:compare` CLI read the attachment by byte scan, so a crafted object
|
||||
stream is never inflated (tested with a 64 MiB bomb: milliseconds, no
|
||||
memory growth). pdf-lib stays on the export side, on trusted inputs, and
|
||||
in tests that inspect exported files. What remains before external users
|
||||
(`docs/development/PDF_IMPORT_HARDENING.md` §5): a Worker or server-side
|
||||
budget for any path that still parses PDF objects, the pdf-lib dependency
|
||||
decision, and read-side caps on report collections.
|
||||
- The scanner reads what is in clear: an attachment whose stream dictionary
|
||||
is malformed beyond `/Length` and `endstream`, or an encrypted file, yields
|
||||
no candidate (`no_attachment`), as the previous reader did in practice.
|
||||
- `<use>` expansion is bounded by the rasterizer, not by the sanitiser,
|
||||
which only counts the `<use>` elements (≤ 256). resvg (tests) refuses an
|
||||
oversized tree; the production path is the browser rasterizer
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
# PDF import hardening — BLK-009 analysis
|
||||
|
||||
Status 2026-10-11: analysis requested by the user after Step 6. Nothing here is
|
||||
implemented; §4 says what can land now and §5 what must land before the
|
||||
import is opened to external users.
|
||||
Status 2026-10-11: analysis requested by the user after Step 6. §4 landed the
|
||||
same day on `feature/step-06-pdf-archives` (options A + B, see §4 for the
|
||||
commit); §5 lists what must land before the import is opened to external
|
||||
users.
|
||||
|
||||
## 1. The problem
|
||||
|
||||
@@ -36,13 +37,13 @@ PDF object graph at all.
|
||||
|
||||
## 3. Options
|
||||
|
||||
| Option | Protection | Cost | Verdict |
|
||||
|---|---|---|---|
|
||||
| A. Byte-scan reader: find `/Type /EmbeddedFile` streams, read `/Length` (direct integer, as pdf-lib writes it; fall back to `endstream` search), inflate with the existing capped `inflateFlate`, parse JSON; no `PDFDocument.load` on import | removes pdf-lib from the import path entirely: the only inflation is our capped one | one module (~120 lines) + tests on every fixture (v2, bare v3, envelope, object streams on/off, several streams, foreign PDF) | **recommended, can land now** |
|
||||
| B. Export with `useObjectStreams: false` | the Filespec name, the Info keywords and the Filespec→EF link become visible to a byte scan, so the filename-first rule and the `attachment_stripped` detection keep working without the object graph; files a few KB larger | one save option; older PDFs keep their object streams, which A tolerates | **recommended with A** |
|
||||
| C. Load in a Web Worker with a timeout and a memory budget | contains the damage (the tab survives, the worker dies) | worker bundling, message protocol, no memory cap API in browsers (only termination on timeout) | defence in depth, later |
|
||||
| D. Pre-scan declared `/Length` of `/ObjStm` streams and refuse above a sum | partial: declared lengths are the compressed sizes, the expansion is unknown | small | insufficient alone |
|
||||
| E. Replace pdf-lib's parser (fork `@cantoo/pdf-lib`, or `pdf.js` for reading) | maintained parser; pdf.js has its own limits | dependency change, larger bundle | evaluate for Step 14 (server-side import) |
|
||||
| Option | Protection | Cost | Verdict |
|
||||
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
|
||||
| A. Byte-scan reader: find `/Type /EmbeddedFile` streams, read `/Length` (direct integer, as pdf-lib writes it; fall back to `endstream` search), inflate with the existing capped `inflateFlate`, parse JSON; no `PDFDocument.load` on import | removes pdf-lib from the import path entirely: the only inflation is our capped one | one module (~120 lines) + tests on every fixture (v2, bare v3, envelope, object streams on/off, several streams, foreign PDF) | **recommended, can land now** |
|
||||
| B. Export with `useObjectStreams: false` | the Filespec name, the Info keywords and the Filespec→EF link become visible to a byte scan, so the filename-first rule and the `attachment_stripped` detection keep working without the object graph; files a few KB larger | one save option; older PDFs keep their object streams, which A tolerates | **recommended with A** |
|
||||
| C. Load in a Web Worker with a timeout and a memory budget | contains the damage (the tab survives, the worker dies) | worker bundling, message protocol, no memory cap API in browsers (only termination on timeout) | defence in depth, later |
|
||||
| D. Pre-scan declared `/Length` of `/ObjStm` streams and refuse above a sum | partial: declared lengths are the compressed sizes, the expansion is unknown | small | insufficient alone |
|
||||
| E. Replace pdf-lib's parser (fork `@cantoo/pdf-lib`, or `pdf.js` for reading) | maintained parser; pdf.js has its own limits | dependency change, larger bundle | evaluate for Step 14 (server-side import) |
|
||||
|
||||
With A, the import path reads bytes, finds at most a few streams, inflates one
|
||||
under 16 MiB, parses JSON under the existing schema caps. pdf-lib stays on
|
||||
@@ -59,22 +60,29 @@ encrypted file is encrypted too and would fail to inflate → `no_attachment`,
|
||||
same outcome as today in practice); the 50 MiB file cap stays as the scan
|
||||
bound.
|
||||
|
||||
## 4. What can be done now (before merge of Step 6, or as the next chore)
|
||||
## 4. Done 2026-10-11 (feature/step-06-pdf-archives, commit: see `git log -- packages/pdf/src/attachment/scan.ts`)
|
||||
|
||||
1. Option A in `packages/pdf/src/attachment/scan.ts`, used by
|
||||
`readReportArchive`; `readAttachedDocument(pdfDoc)` kept for the CLI until
|
||||
it moves to the scanner too.
|
||||
1. Option A in `packages/pdf/src/attachment/scan.ts`
|
||||
(`findEmbeddedFiles`, `readAttachmentBytes`, `hasArchiveStamp`), used by
|
||||
`readReportArchive`; `readAttachedDocument(bytes)` takes the file bytes
|
||||
and the `report:compare` CLI runs on the scanner too. No
|
||||
`PDFDocument.load` remains under `packages/pdf/src` or `apps/app/scripts`.
|
||||
2. Option B on export (`pdfDoc.save({ useObjectStreams: false })`): no visible
|
||||
change; goldens unaffected (they record drawn text, images, links); the
|
||||
envelope roundtrip tests prove the reader still finds the attachment in
|
||||
both layouts.
|
||||
3. Tests: every existing import fixture through the scanner; a crafted
|
||||
object-stream bomb (the audit's recipe: a few hundred KB inflating to
|
||||
hundreds of MB) imports in milliseconds without inflating it; a PDF with
|
||||
two embedded files; a PDF whose `/Length` is indirect.
|
||||
change; goldens unchanged (they record drawn text, images, links); the
|
||||
envelope roundtrip tests prove the reader finds the attachment in both
|
||||
layouts.
|
||||
3. Tests (`packages/pdf/tests/scan.test.ts`): the three frozen fixtures (object
|
||||
streams) and a fresh export; two embedded files with a decoy first;
|
||||
indirect and wrong `/Length`; `\r\n` after `stream`; `/Type/EmbeddedFile`
|
||||
without a space; hex and literal names and keywords; garbage inputs and
|
||||
the 32-candidate cap; a 64 MiB object-stream bomb next to the real
|
||||
attachment (report in milliseconds, RSS flat) and an `EmbeddedFile` bomb
|
||||
(inflation cancelled at 16 MiB, `no_attachment`).
|
||||
4. Docs: `docs/PDF-FORMAT.md` "Reading an archive" and "Known limits".
|
||||
|
||||
Estimated size: one executor phase, about 300 lines including tests.
|
||||
Known trade-off: a stamped PDF saved before this change (object streams)
|
||||
hides its `/Keywords`, so a stripped attachment on such a file reads as
|
||||
`no_attachment` instead of `attachment_stripped`.
|
||||
|
||||
## 5. Prerequisites before external users (Step 7+ sync, Step 14 worker)
|
||||
|
||||
|
||||
Reference in new issue
Block a user