fix(pdf): candidates sorted by data offset, JSON depth pre-scan before parse, budget wording; chore(memory): BLK-009 mitigated with gate, EVAL-014, journal, TODO, CLAUDE.md import notes

This commit is contained in:
bchanot committed 2026-10-11 17:31:07 +02:00
1 parent 3a6e92625a
commit 62b2b9d86a
9 files changed
+188 -30

No files matched your search

+1 -1
View File
@@ -80,4 +80,4 @@ Status: open (no real data yet; treat before Step 7+ persistence of user reports
## BLK-009 | 2026-10-11 | pdf-lib inflates PDF object streams without cap at `PDFDocument.load` (import path) ## BLK-009 | 2026-10-11 | pdf-lib inflates PDF object streams without cap at `PDFDocument.load` (import path)
Security audit Step 6 (MEDIUM, pre-existing since Step 1 import): `PDFDocument.load` decodes every `/ObjStm` and xref stream eagerly with its own uncapped FlateStream; measured 408 KB PDF → 400 MB in 1.4 s, 468 MB RSS; `MAX_PDF_BYTES` 50 MiB allows ~50 GB. Attachment cap (16 MiB) covers the attachment only. Impact: tab freeze/OOM for the practitioner on a crafted PDF, no code execution. Cannot refuse `/ObjStm` (pdf-lib `save()` emits object streams in our own PDFs). Options: parse in a Web Worker (bounded blast radius), or locate the attachment by scanning bytes without a full load, or a capped custom decoder. pdf-lib 1.17.1 unmaintained since 2021 (LOW in the same audit). Security audit Step 6 (MEDIUM, pre-existing since Step 1 import): `PDFDocument.load` decodes every `/ObjStm` and xref stream eagerly with its own uncapped FlateStream; measured 408 KB PDF → 400 MB in 1.4 s, 468 MB RSS; `MAX_PDF_BYTES` 50 MiB allows ~50 GB. Attachment cap (16 MiB) covers the attachment only. Impact: tab freeze/OOM for the practitioner on a crafted PDF, no code execution. Cannot refuse `/ObjStm` (pdf-lib `save()` emits object streams in our own PDFs). Options: parse in a Web Worker (bounded blast radius), or locate the attachment by scanning bytes without a full load, or a capped custom decoder. pdf-lib 1.17.1 unmaintained since 2021 (LOW in the same audit).
Status: open; analysis + plan in docs/development/PDF_IMPORT_HARDENING.md (2026-10-11): byte-scan reader (no pdf-lib load on import) + `useObjectStreams: false` at export can land now; worker/server import + pdf-lib replacement are prerequisites before external users. Status: MITIGATED on the browser import path 2026-10-11 (60ee712: byte-scan attachment reader, no `PDFDocument.load` on untrusted bytes, export without object streams; ObjStm bomb read in 5 ms, +0.5 MiB RSS). Then 3a6e926: inflater fed in 16 KiB slices (Chromium read-ahead defeated the cancel: 4 GiB bomb +2.2 GB → +192 MB measured), shared 16 MiB candidate budget, bounded endstream search; final loop: candidates sorted by offset, JSON depth pre-scan. Audits ×3 PASS; worst measured costs on crafted files: +192 MB single bomb, ~0.5 s / ≤0.4 GB for 32 failing inflations, ~1 s deep JSON (before the pre-scan). GATE STILL OPEN before any external-user import (BDR-040): Firefox/Safari inflater read-ahead unmeasured (no browser test runner), Worker/server import with time+memory budget, pdf-lib replacement decision (unmaintained), report collection caps on read, `<use>` nesting budget, low-end phone measurement. Plan: docs/development/PDF_IMPORT_HARDENING.md §5. Never close on passing tests alone.
+9
View File
@@ -106,3 +106,12 @@ Method: analysis (3 analyzers + measurements) challenged by 3 plan-challengers (
Result: Phase 0 froze 39 goldens (text/images/links per locale × logo mode, view facts) + 2 route characterisation tests with the OLD code; engine reproduced 36 goldens with 0 mismatch (views: data-* 2→0 and hidden-marker titles declared relaxed). Verifier pass 1 ECARTS(31) = all oracle/scope defects of MY contract (grep on parametrised table, xmldom filter, narrow FILE SCOPE, 3 deleted tests with moved coverage) → contract amended → pass 2 CONFORME 13/13. Security pass 1 BLOCK(1) HIGH real + PoC: archived branding URL written unescaped into `/URI` (PDF object injection via `exportFromEnvelope`), +3 MEDIUM/4 LOW → executor fixed (hex URI, strict URL schema, export re-validation, PNG IHDR budget, CSS-escape rules, NaN guard, file.size) → pass 2 PASS but 2 MEDIUM incl. a regression the fix introduced (UTF-16 BOM URI dead in poppler) + PNG double-IHDR bypass → executor → pass 3 PASS + 1 MEDIUM (UPNG nextZero spin on tEXt) → executor (chunk allowlist + IEND) → pass 4 PASS + 3 MEDIUM (UPNG `inflateRaw` spins on truncated IDAT with allowlisted chunks only; IHDR fields unvalidated; zero dimension) → executor: native bounded `inflateFlate` of the IDAT against the exact raw size, IHDR fields, sides ≥ 1, no interlace, raw ≤ 16 MiB → pass 5 focused. Lesson: a third-party decoder on untrusted bytes needs a native oracle in front, not a chunk allowlist. Final: 656 tests (194/21/130/88/223), typecheck, lint, build, docker ×4 green; goldens byte-identical since 8b86b92. Result: Phase 0 froze 39 goldens (text/images/links per locale × logo mode, view facts) + 2 route characterisation tests with the OLD code; engine reproduced 36 goldens with 0 mismatch (views: data-* 2→0 and hidden-marker titles declared relaxed). Verifier pass 1 ECARTS(31) = all oracle/scope defects of MY contract (grep on parametrised table, xmldom filter, narrow FILE SCOPE, 3 deleted tests with moved coverage) → contract amended → pass 2 CONFORME 13/13. Security pass 1 BLOCK(1) HIGH real + PoC: archived branding URL written unescaped into `/URI` (PDF object injection via `exportFromEnvelope`), +3 MEDIUM/4 LOW → executor fixed (hex URI, strict URL schema, export re-validation, PNG IHDR budget, CSS-escape rules, NaN guard, file.size) → pass 2 PASS but 2 MEDIUM incl. a regression the fix introduced (UTF-16 BOM URI dead in poppler) + PNG double-IHDR bypass → executor → pass 3 PASS + 1 MEDIUM (UPNG nextZero spin on tEXt) → executor (chunk allowlist + IEND) → pass 4 PASS + 3 MEDIUM (UPNG `inflateRaw` spins on truncated IDAT with allowlisted chunks only; IHDR fields unvalidated; zero dimension) → executor: native bounded `inflateFlate` of the IDAT against the exact raw size, IHDR fields, sides ≥ 1, no interlace, raw ≤ 16 MiB → pass 5 focused. Lesson: a third-party decoder on untrusted bytes needs a native oracle in front, not a chunk allowlist. Final: 656 tests (194/21/130/88/223), typecheck, lint, build, docker ×4 green; goldens byte-identical since 8b86b92.
Anomalies: contract oracles written fast = 2 verifier passes lost (lesson: run every CHECK once before dispatching the verifier); security fix introduced a functional regression invisible to tests because the test reader decoded the BOM (lesson: assert raw bytes, cross-check with a real reader like poppler); executors denied `rm`/`git rm` (orchestrator deletes); one executor used `git stash` despite "no git state" (now forbidden explicitly in briefs); demo envelope with archived branding = 136 KB stored / 306 KB raw (raw > 250 KB, stored < 250 KB: reported, user said data first). Anomalies: contract oracles written fast = 2 verifier passes lost (lesson: run every CHECK once before dispatching the verifier); security fix introduced a functional regression invisible to tests because the test reader decoded the BOM (lesson: assert raw bytes, cross-check with a real reader like poppler); executors denied `rm`/`git rm` (orchestrator deletes); one executor used `git stash` despite "no git state" (now forbidden explicitly in briefs); demo envelope with archived branding = 136 KB stored / 306 KB raw (raw > 250 KB, stored < 250 KB: reported, user said data first).
Action: await user manual validation + UX decision on re-export of the kept archive; BLK-009 (pdf-lib ObjStm inflation) before external users; react-router advisories (pre-existing) to bump in a chore. Action: await user manual validation + UX decision on re-export of the kept archive; BLK-009 (pdf-lib ObjStm inflation) before external users; react-router advisories (pre-existing) to bump in a chore.
---
## EVAL-014 | 2026-10-11 | Step 6 closure round — manual-test feedback, BLK-009 scanner, final audits
Method: user manual tests round 1 (logos absent, picker blurry) → reproduction attempts (headless Chromium dev/preview/Docker, Firefox 157 via esbuild harness + raw WebDriver BiDi client, CDP LayerTree, LCD-fringe metric, headed runs on the Wayland session) → fixes by executors → user round 2 confirmed both OK → closure decisions → BLK-009 implemented (byte-scan reader, no object streams) → verifier + 2 security audits + 1 fix loop.
Result: neither visual defect reproduced here (4 environments, 2 engines, 2 PDF renderers); shipped the environment-independent fixes (branding bundled, no fetch; warnings surfaced) and the user confirmed logos + crisp text. Scanner: verifier CONFORME 7/7; audit 1 PASS + 2 MEDIUM (Chromium DecompressionStream read-ahead defeats the cancel: +2.2 GB on a 4 GiB bomb; ×32 candidate CPU) → executor (16 KiB slice feeding, shared 16 MiB budget, monotonic bounded endstream search, `{` pre-check; 133 pdf tests) → audit 2 focused. Workspace 710 tests, build, docker green.
Anomalies: the blur fix of the previous round (will-change) had been marked PARTIALLY MET "desktop by user" — the user's "still blurry" came from that unverified half; this round could not verify real-GPU text rendering either (headless engines render grayscale AA, headed captures failed) → lesson: a screen-rendering defect needs the user's own capture + an A/B recipe early, not days of headless measurement. The Node vitest suite cannot observe browser stream read-ahead: record browser-only bounds as owed manual checks (PDF_IMPORT_HARDENING §5).
Action: BLK-009 stays "mitigated, gate open" (BDR-040); Step 6 finished into develop; release/0.2.0 prepped for the user's go.
+4
View File
@@ -207,3 +207,7 @@ User validated the Step 6 analysis with D7 (branding archived), D8 (imported sna
## 2026-10-11 (Step 6 manual-test feedback) ## 2026-10-11 (Step 6 manual-test feedback)
User: step not merged; findings-picker text still blurry; footer logos absent in their PDFs; decisions: re-export from archive wanted (UX to propose), restore-rebase must be visible, exact labels to validate; BLK-009 analysis. Done on feature/step-06-pdf-archives: logos NOT reproduced (real export in headless Chromium dev/preview/Docker image + Firefox 157 engine harness: 4 logos embedded, 0 warnings, poppler + pdf.js render them) → branding bundled into the app (`src/assets/brand`, `?inline`/`?raw`, no fetch at all) + export returns warnings + `editor.export_warnings` alert (293a6e0); blur NOT reproduced (headless Chromium/Firefox, DPR 1/1.25/2, scrolling 9-leaf list, CDP LayerTree: no composited layer, no lingering animation/will-change; headed runs on the user's Wayland display failed to capture) → analysis + A/B recipe for the user; restore rebase alert `editor.draft_rebased` (e8f1734); docs/development/PDF_IMPORT_HARDENING.md (byte-scan reader + no object streams now; worker/parser later) (5ed4a4b). Root check 660 tests, build, docker green. Awaiting user: browser/zoom for the blur, UX choice for re-export, label validation. User: step not merged; findings-picker text still blurry; footer logos absent in their PDFs; decisions: re-export from archive wanted (UX to propose), restore-rebase must be visible, exact labels to validate; BLK-009 analysis. Done on feature/step-06-pdf-archives: logos NOT reproduced (real export in headless Chromium dev/preview/Docker image + Firefox 157 engine harness: 4 logos embedded, 0 warnings, poppler + pdf.js render them) → branding bundled into the app (`src/assets/brand`, `?inline`/`?raw`, no fetch at all) + export returns warnings + `editor.export_warnings` alert (293a6e0); blur NOT reproduced (headless Chromium/Firefox, DPR 1/1.25/2, scrolling 9-leaf list, CDP LayerTree: no composited layer, no lingering animation/will-change; headed runs on the user's Wayland display failed to capture) → analysis + A/B recipe for the user; restore rebase alert `editor.draft_rebased` (e8f1734); docs/development/PDF_IMPORT_HARDENING.md (byte-scan reader + no object streams now; worker/parser later) (5ed4a4b). Root check 660 tests, build, docker green. Awaiting user: browser/zoom for the blur, UX choice for re-export, label validation.
## 2026-10-11 (Step 6 closure)
User round 2: logos OK, picker text crisp, flows OK → STEP 6 VALIDATED. Decisions BDR-040: re-export-archive button refused for good; restore-rebase alert approved; import headlines full/partial (62b8a43). BLK-009: byte-scan attachment reader + export without object streams (60ee712), verifier CONFORME, audit PASS+2 MEDIUM → slice-fed inflater + candidate budget (3a6e926), focused re-audit. EVAL-014. Then: gitflow finish into develop, release/0.2.0 prep (Milestone A) up to the human go; Step 7 not started (UX/design to discuss with ChatGPT first).
+3 -1
View File
@@ -108,7 +108,9 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo
- [x] 6.4 Phase 3 packages/pdf (c179560: 53 engine tests, 36 goldens exact, resvg raster, demo envelope 55 KB / 136 KB stored): scaffold (no DOM lib, restricted globals, vitest node, Dockerfile, CI, README), svg/sanitize allowlist + horse.svg zero-removal, render moved, Branding/Rasterizer/exportedAt/safeText/dict, exportReportArchive (T3 both logo modes), readReportArchive (filename-first, layered, attachment_stripped; T4-T6, T8), T1/T7/T9/T10 resvg/T11, xmldom bundle size ≤ 50 KB gz - [x] 6.4 Phase 3 packages/pdf (c179560: 53 engine tests, 36 goldens exact, resvg raster, demo envelope 55 KB / 136 KB stored): scaffold (no DOM lib, restricted globals, vitest node, Dockerfile, CI, README), svg/sanitize allowlist + horse.svg zero-removal, render moved, Branding/Rasterizer/exportedAt/safeText/dict, exportReportArchive (T3 both logo modes), readReportArchive (filename-first, layered, attachment_stripped; T4-T6, T8), T1/T7/T9/T10 resvg/T11, xmldom bundle size ≤ 50 KB gz
- [x] 6.5 Phase 4 app (9bd5173: 222 app tests, archive kept in session, rebase + alerts, legacy PDF modules removed, xmldom ≈ 31 KB gz in the main chunk): browserRasterizer, branding.ts, export/import on the engine, rebase on import + draft restore, 2 new messages, old render/attachment/highlight code + pdf.* keys removed, report:compare on engine API, T2/T14/T15, suite migrated 1:1 - [x] 6.5 Phase 4 app (9bd5173: 222 app tests, archive kept in session, rebase + alerts, legacy PDF modules removed, xmldom ≈ 31 KB gz in the main chunk): browserRasterizer, branding.ts, export/import on the engine, rebase on import + draft restore, 2 new messages, old render/attachment/highlight code + pdf.* keys removed, report:compare on engine API, T2/T14/T15, suite migrated 1:1
- [x] 6.6 Phase 5 docs + gates (docs synced; verifier ECARTS(31 oracle/scope) → CONFORME 13/13; security BLOCK(1) → 4 fix loops → PASS; 656 tests, build, docker ×4 green) - [x] 6.6 Phase 5 docs + gates (docs synced; verifier ECARTS(31 oracle/scope) → CONFORME 13/13; security BLOCK(1) → 4 fix loops → PASS; 656 tests, build, docker ×4 green)
- [ ] 6.8 user manual tests round 1 (2026-10-11): logos absent in their PDFs → not reproduced in 4 environments; branding bundled, export warnings surfaced (293a6e0); blur in the findings picker → not reproduced (headless, no layer/animation left); restore-rebase alert added (e8f1734); PDF_IMPORT_HARDENING.md written. AWAITING USER: browser + zoom + screenshot for the blur (A/B recipe in the report), UX choice for "re-export the archive", validation of the 6 message labels; then merge decision - [x] 6.8 user manual tests round 1 (2026-10-11): logos and picker text confirmed OK by the user after branding bundled (293a6e0) and the investigation; restore-rebase alert (e8f1734); STEP 6 VALIDATED BY USER 2026-10-11
- [x] 6.9 closure decisions (BDR-040): re-export-archive button REFUSED for good; import headlines full/partial (62b8a43); BLK-009 byte-scan reader + no object streams (60ee712), fresh security audit + verifier; gate before external users kept in PDF_IMPORT_HARDENING.md §5
- [ ] 6.10 finish into develop, release/0.2.0 prepped (Milestone A) — tag/merge to main on the user's go; Step 7 NOT started (UX/design discussion first)
- [ ] 6.7 follow-ups recorded for Step 13: catalog asset cap aligned with the archive cap (2 MiB), allowlist sanitiser at upload, app canvas sanitiser (EVAL-011) - [ ] 6.7 follow-ups recorded for Step 13: catalog asset cap aligned with the archive cap (2 MiB), allowlist sanitiser at upload, app canvas sanitiser (EVAL-011)
- [ ] 5 follow-up: allowlist SVG sanitiser before Storage-served assets (Step 13); orphan findings UX (D5) at the report UX step; report:compare residuals (ZodError reason genericised, v3 lines computed once, long template lines); 16 MiB attachment cap without direct test - [ ] 5 follow-up: allowlist SVG sanitiser before Storage-served assets (Step 13); orphan findings UX (D5) at the report UX step; report:compare residuals (ZodError reason genericised, v3 lines computed once, long template lines); 16 MiB attachment cap without direct test
+3 -1
View File
@@ -269,7 +269,8 @@ PDF export (app side: apps/app/src/pdf, engine: packages/pdf)
PDF import PDF import
→ importReportFromPdf(bytes, migrationMap) → readReportArchive (engine) → importReportFromPdf(bytes, migrationMap) → readReportArchive (engine)
→ attachment osteo-report* first → layered reading: report (v3 as is, v2 migrated, else incompatible_version), → byte scan for the EmbeddedFile stream (no pdf-lib parse on untrusted bytes, BLK-009): named osteo-report* first,
capped native inflation → layered reading: report (v3 as is, v2 migrated, else incompatible_version),
envelope metadata tolerant, snapshot/branding/labels strict → archive kept or snapshotRejected (closed reason) envelope metadata tolerant, snapshot/branding/labels strict → archive kept or snapshotRejected (closed reason)
→ rebaseReportOnSheet(report, index, archivedSnapshot) when the loaded sheet differs (unresolved flags, labels kept) → rebaseReportOnSheet(report, index, archivedSnapshot) when the loaded sheet differs (unresolved flags, labels kept)
→ hydrateImport({ report, archive }) → one alert (archive_ok / snapshot_rejected / rebased / migrated) → navigate /editor → hydrateImport({ report, archive }) → one alert (archive_ok / snapshot_rejected / rebased / migrated) → navigate /editor
@@ -430,6 +431,7 @@ Database (Supabase)
- Never modify the legacy sources under `supabase/seed/anatomical-sheets/equine/legacy-sources/` — they are the - Never modify the legacy sources under `supabase/seed/anatomical-sheets/equine/legacy-sources/` — they are the
importer's input (the user's historical files). importer's input (the user's historical files).
- Ask before any change to `readReportArchive`, `readArchiveDocument`, `readAttachedDocument` or `rebaseReportOnSheet` — they handle user data; archive readers never drop a schema version (envelope v1, Report v2/v3, sheet document v2). - Ask before any change to `readReportArchive`, `readArchiveDocument`, `readAttachedDocument` or `rebaseReportOnSheet` — they handle user data; archive readers never drop a schema version (envelope v1, Report v2/v3, sheet document v2).
- Import never parses untrusted PDF object graphs: the attachment is located by byte scan and inflated under the 16 MiB cap (BLK-009). Opening the import to external users is gated by `docs/development/PDF_IMPORT_HARDENING.md` §5 (BDR-040).
- After PDF roundtrip tests fail: check attachment bytes, then the layered reader (report first, envelope metadata, snapshot/branding/labels), then DecompressionStream inflation. Goldens under `apps/app/tests/fixtures/goldens/step6` are never regenerated without a user decision. - After PDF roundtrip tests fail: check attachment bytes, then the layered reader (report first, envelope metadata, snapshot/branding/labels), then DecompressionStream inflation. Goldens under `apps/app/tests/fixtures/goldens/step6` are never regenerated without a user decision.
- Node 24 / pnpm 10.13.1: `corepack enable` or `corepack pnpm`; never a global install. - Node 24 / pnpm 10.13.1: `corepack enable` or `corepack pnpm`; never a global install.
- SQL change: `pnpm --filter @faunosteo/supabase test:run` (PGlite) before commit; `pnpm db:reset && pnpm db:test` - SQL change: `pnpm --filter @faunosteo/supabase test:run` (PGlite) before commit; `pnpm db:reset && pnpm db:test`
+7 -5
View File
@@ -145,11 +145,13 @@ never lost because of its wrapper:
gives it a name. Candidates named `osteo-report*` come first, then file gives it a name. Candidates named `osteo-report*` come first, then file
order; Flate data is fed to the platform inflater in 16 KiB slices, so order; Flate data is fed to the platform inflater in 16 KiB slices, so
memory stays bounded in browsers too, under a 16 MiB cap (cancelled memory stays bounded in browsers too, under a 16 MiB cap (cancelled
beyond it); at most 16 MiB of candidate content is examined in all, a beyond it); at most 16 MiB of candidate content is retained and parsed in
candidate not opening on `{` is skipped unparsed, and the first one whose all (a cancelled inflation is not charged, so up to 32 oversized streams
text is a JSON object with a numeric `envelopeVersion` or `version` may each cost one capped inflation of CPU, bounded, no memory retained),
wins. Object streams, cross-reference streams and page contents are never a candidate not opening on `{` or nesting brackets deeper than 64 levels
inflated. is skipped unparsed, and the first one whose text is a JSON object with a
numeric `envelopeVersion` or `version` wins. Object streams,
cross-reference streams and page contents are never inflated.
No candidate: `attachment_stripped` when a `/Keywords` string in clear No candidate: `attachment_stripped` when a `/Keywords` string in clear
contains `faunosteo-archive`, else `no_attachment`. A stamped file saved contains `faunosteo-archive`, else `no_attachment`. A stamped file saved
with object streams (before BLK-009) hides its stamp and reads as with object streams (before BLK-009) hides its stamp and reads as
+7 -3
View File
@@ -93,9 +93,13 @@ hides its `/Keywords`, so a stripped attachment on such a file reads as
The audit of 60ee712 measured +2.2 GB RSS in Chromium on a 4 MB / 4 GiB The audit of 60ee712 measured +2.2 GB RSS in Chromium on a 4 MB / 4 GiB
bomb before the inflater was fed in 16 KiB slices (+215 MB after); that bomb before the inflater was fed in 16 KiB slices (+215 MB after); that
measurement is manual until a browser runner exists. measurement is manual until a browser runner exists.
- Done: the candidate CPU/byte budget on the scanner, one 16 MiB budget of - Done: the candidate CPU/byte budget on the scanner: one 16 MiB budget of
examined content shared by every candidate, a monotonic `endstream` search retained and parsed content shared by every candidate (a cancelled
bounded to the cap, and no JSON parse for a candidate not opening on `{`. inflation is not charged, so up to 32 oversized streams may each cost one
capped inflation of CPU, bounded, no memory retained), a monotonic
`endstream` search over candidates sorted by data offset and bounded to
the cap, and no JSON parse for a candidate not opening on `{` or nesting
brackets deeper than 64 levels (linear byte pre-scan, strings skipped).
- Option C or a server-side import (Step 14) with a hard memory and time - Option C or a server-side import (Step 14) with a hard memory and time
budget per file, for any path that still parses PDF objects. budget per file, for any path that still parses PDF objects.
- Dependency decision on pdf-lib (unmaintained since 2021): keep it for - Dependency decision on pdf-lib (unmaintained since 2021): keep it for
+71 -17
View File
@@ -261,9 +261,10 @@ type EndstreamFinder = (offset: number) => number
/** /**
* Searches `endstream` once over the file instead of once per candidate: * Searches `endstream` once over the file instead of once per candidate:
* candidates come in offset order, so the last hit serves every offset * candidates come in offset order (`embeddedStreamHeads` sorts them), so the
* before it and a miss clears the bytes it covered for the next search. An * last hit serves every offset before it and a miss clears the bytes it
* offset out of order (never produced today) just restarts the search. * covered for the next search. An offset out of order (never produced since
* the sort) restarts the search: a defensive fallback, never the normal path.
*/ */
function endstreamFinder(bytes: Uint8Array, end: number): EndstreamFinder { function endstreamFinder(bytes: Uint8Array, end: number): EndstreamFinder {
let lastHit = -1 let lastHit = -1
@@ -302,14 +303,37 @@ function filterOf(dict: string): EmbeddedFileCandidate['filter'] {
return /\/Filter\s*(\[\s*)?\/FlateDecode\b/.test(dict) ? 'FlateDecode' : null return /\/Filter\s*(\[\s*)?\/FlateDecode\b/.test(dict) ? 'FlateDecode' : null
} }
/** An `EmbeddedFile` dictionary followed by `stream`, before its data is measured. */
interface StreamHead {
dict: DictSpan
offset: number
}
/**
* Every `/Type /EmbeddedFile` dictionary followed by `stream`, sorted by data
* offset (stable). Nested dictionaries list outermost first, so their offsets
* come out descending and each one would reset the `endstream` finder: 32 of
* them rescanned 16 MiB each (audit of 3a6e926, finding 1). Sorted, the
* finder never restarts.
*/
function embeddedStreamHeads(bytes: Uint8Array, end: number): StreamHead[] {
const heads: StreamHead[] = []
for (const pos of findTypedDicts(bytes, EMBEDDED_FILE, end)) {
const dict = dictAround(bytes, pos, end)
if (!dict) continue
const offset = streamDataOffset(bytes, dict.end, end)
if (offset < 0) continue
heads.push({ dict, offset })
}
return heads.sort((a, b) => a.offset - b.offset)
}
function candidateOf( function candidateOf(
bytes: Uint8Array, bytes: Uint8Array,
dict: DictSpan, { dict, offset }: StreamHead,
end: number, end: number,
findEndstream: EndstreamFinder, findEndstream: EndstreamFinder,
): Omit<EmbeddedFileCandidate, 'name'> | null { ): Omit<EmbeddedFileCandidate, 'name'> | null {
const offset = streamDataOffset(bytes, dict.end, end)
if (offset < 0) return null
const length = const length =
declaredLength(bytes, dict.text, offset, end) ?? declaredLength(bytes, dict.text, offset, end) ??
measuredLength(bytes, offset, findEndstream) measuredLength(bytes, offset, findEndstream)
@@ -392,20 +416,19 @@ export function hasPdfHeader(bytes: Uint8Array): boolean {
/** /**
* Every `/Type /EmbeddedFile` stream of the file, named when its Filespec * Every `/Type /EmbeddedFile` stream of the file, named when its Filespec
* is in clear (export without object streams); `osteo-report*` names first, * is in clear (export without object streams); `osteo-report*` names first,
* then file order. At most MAX_CANDIDATES, within MAX_PDF_BYTES. * then data offset order. At most MAX_CANDIDATES, within MAX_PDF_BYTES.
*/ */
export function findEmbeddedFiles(bytes: Uint8Array): EmbeddedFileCandidate[] { export function findEmbeddedFiles(bytes: Uint8Array): EmbeddedFileCandidate[] {
const end = Math.min(bytes.byteLength, MAX_PDF_BYTES) const end = Math.min(bytes.byteLength, MAX_PDF_BYTES)
const names = filespecNames(bytes, end) const names = filespecNames(bytes, end)
const findEndstream = endstreamFinder(bytes, end) const findEndstream = endstreamFinder(bytes, end)
const found: EmbeddedFileCandidate[] = [] const found: EmbeddedFileCandidate[] = []
for (const pos of findTypedDicts(bytes, EMBEDDED_FILE, end)) { for (const head of embeddedStreamHeads(bytes, end)) {
const dict = dictAround(bytes, pos, end) const candidate = candidateOf(bytes, head, end, findEndstream)
if (!dict) continue
const candidate = candidateOf(bytes, dict, end, findEndstream)
if (!candidate) continue if (!candidate) continue
const { objectNumber } = head.dict
const name = const name =
dict.objectNumber === null ? null : (names.get(dict.objectNumber) ?? null) objectNumber === null ? null : (names.get(objectNumber) ?? null)
found.push({ ...candidate, name }) found.push({ ...candidate, name })
} }
const named = (c: EmbeddedFileCandidate) => const named = (c: EmbeddedFileCandidate) =>
@@ -440,6 +463,34 @@ function startsJsonObject(data: Uint8Array): boolean {
return data[i] === 0x7b return data[i] === 0x7b
} }
/** The report and its envelope nest under 16 levels; deeper is not ours. */
const MAX_JSON_DEPTH = 64
/**
* Linear bracket-depth count over the raw bytes before `JSON.parse`: a 16 MiB
* document nested thousands deep costs the parser a second and a gigabyte
* (audit of 3a6e926, finding 3). String literals are skipped, their escapes
* honoured; UTF-8 continuation bytes never collide with the ASCII tokens.
*/
function withinJsonDepth(data: Uint8Array): boolean {
let depth = 0
let inString = false
for (let i = 0; i < data.byteLength; i += 1) {
const byte = data[i]!
if (inString) {
if (byte === 0x5c) i += 1 // backslash: the escaped byte is skipped
else if (byte === 0x22) inString = false
continue
}
if (byte === 0x22) inString = true
else if (byte === 0x7b || byte === 0x5b) {
depth += 1
if (depth > MAX_JSON_DEPTH) return false
} else if (byte === 0x7d || byte === 0x5d) depth -= 1
}
return true
}
/** JSON with a numeric `envelopeVersion` or `version`; undefined otherwise. */ /** JSON with a numeric `envelopeVersion` or `version`; undefined otherwise. */
function parseVersioned(data: Uint8Array): unknown { function parseVersioned(data: Uint8Array): unknown {
let parsed: unknown let parsed: unknown
@@ -464,10 +515,13 @@ export interface LocatedAttachment {
/** /**
* The first candidate whose data is a versioned JSON document, decoded once. * The first candidate whose data is a versioned JSON document, decoded once.
* None: `attachment_stripped` when the stamp is visible, else `no_attachment`. * None: `attachment_stripped` when the stamp is visible, else `no_attachment`.
* One MAX_ATTACHMENT_BYTES budget of examined content (raw or inflated) is * One MAX_ATTACHMENT_BYTES budget of content (raw or inflated) is shared by
* shared by every candidate: a candidate that does not fit what is left ends * every candidate: a candidate that does not fit what is left ends the
* the search, so 32 crafted streams cost at most one attachment's worth of * search, so at most 16 MiB of candidate content is retained and parsed. A
* inflation and parsing. A stream cancelled mid-inflation is not charged. * cancelled inflation is not charged (audit of 3a6e926, finding 2), so up to
* MAX_CANDIDATES oversized streams may each cost one capped inflation of CPU
* (bounded, no memory retained); charging them would let bombs starve a real
* unnamed attachment placed after them.
*/ */
export async function locateAttachment( export async function locateAttachment(
bytes: Uint8Array, bytes: Uint8Array,
@@ -478,7 +532,7 @@ export async function locateAttachment(
const data = await candidateBytes(bytes, candidate, budget) const data = await candidateBytes(bytes, candidate, budget)
if (!data) continue if (!data) continue
budget -= data.byteLength budget -= data.byteLength
if (!startsJsonObject(data)) continue if (!startsJsonObject(data) || !withinJsonDepth(data)) continue
const parsed = parseVersioned(data) const parsed = parseVersioned(data)
if (parsed !== undefined) return { bytes: data, data: parsed } if (parsed !== undefined) return { bytes: data, data: parsed }
} }
+83 -2
View File
@@ -3,7 +3,8 @@
* on the frozen fixtures (object-stream layout), crafted layouts (two files, * on the frozen fixtures (object-stream layout), crafted layouts (two files,
* indirect `/Length`, `\r\n`, no space after `/Type`), garbage, the keyword * indirect `/Length`, `\r\n`, no space after `/Type`), garbage, the keyword
* stamp, the object-stream bomb the reader never inflates, and the candidate * stamp, the object-stream bomb the reader never inflates, and the candidate
* cost bounds (monotonic `endstream` search, shared 16 MiB budget, `{` check). * cost bounds (monotonic `endstream` search over offset-sorted candidates,
* shared 16 MiB budget, `{` check, bracket-depth pre-scan before JSON.parse).
*/ */
import { deflateSync } from 'node:zlib' import { deflateSync } from 'node:zlib'
import { beforeAll, describe, expect, it, vi } from 'vitest' import { beforeAll, describe, expect, it, vi } from 'vitest'
@@ -19,7 +20,11 @@ import {
type EmbeddedFileCandidate, type EmbeddedFileCandidate,
} from '../src' } from '../src'
import { inflateFlate } from '../src/attachment/inflate' import { inflateFlate } from '../src/attachment/inflate'
import { decodePdfString, hasArchiveStamp } from '../src/attachment/scan' import {
decodePdfString,
hasArchiveStamp,
locateAttachment,
} from '../src/attachment/scan'
import { readFixtureBytes } from './helpers/draft' import { readFixtureBytes } from './helpers/draft'
import { import {
FIXED_EXPORTED_AT, FIXED_EXPORTED_AT,
@@ -449,6 +454,25 @@ describe('candidate cost bounds (audit M2)', () => {
expect(elapsed).toBeLessThan(1000) expect(elapsed).toBeLessThan(1000)
}) })
it('32 nested EmbeddedFile dicts (descending offsets) without endstream on a 20 MiB file: no candidate, no finder reset', async () => {
// Outermost `/Type` first in the file, its `stream` last: offsets come out
// reversed; the sort by offset keeps the endstream search to one pass.
let nested: Uint8Array = ascii('<< /Type /EmbeddedFile >>\nstream\n')
for (let level = 1; level < 32; level += 1) {
nested = concat([
ascii('<< /Type /EmbeddedFile /A '),
nested,
ascii(' >>\nstream\n'),
])
}
const pdf = craftPdf([ascii('2 0 obj\n'), nested, filled('x', 20 * MiB)])
const { result, elapsed } = await timed('32 nested, no endstream', () =>
findEmbeddedFiles(pdf),
)
expect(result).toEqual([])
expect(elapsed).toBeLessThan(500)
})
it('several indirect /Length streams in a row: each measured to its own endstream', () => { it('several indirect /Length streams in a row: each measured to its own endstream', () => {
const payloads = ['{"version":3}', '{"version":4,"x":1}', '{"v":5}'] const payloads = ['{"version":3}', '{"version":4,"x":1}', '{"v":5}']
const pdf = craftPdf( const pdf = craftPdf(
@@ -501,6 +525,63 @@ describe('candidate cost bounds (audit M2)', () => {
} }
}) })
describe('bracket-depth pre-scan before JSON.parse (audit finding 3)', () => {
const flateObject = (num: number, raw: Uint8Array) => {
const data = new Uint8Array(deflateSync(raw))
return streamObject({
num,
dict: `/Type /EmbeddedFile /Filter /FlateDecode /Length ${data.byteLength}`,
data,
})
}
const withSpy = async (run: () => Promise<void>) => {
const parse = vi.spyOn(JSON, 'parse')
try {
await run()
return parse.mock.calls.length
} finally {
parse.mockRestore()
}
}
it('a 16 KB Flate candidate opening 8 million "[" deep: skipped unparsed, no_attachment fast', async () => {
const raw = concat([ascii('{"version":3,"x":'), filled('[', 8_000_000)])
const object = flateObject(2, raw)
expect(object.byteLength).toBeLessThan(16 * 1024)
const calls = await withSpy(async () => {
const { elapsed } = await timed('8M-deep candidate', () =>
rejects(craftPdf([object]), 'no_attachment'),
)
expect(elapsed).toBeLessThan(100)
})
expect(calls).toBe(0)
})
it('2000 "[" inside a string with escaped quotes and backslashes: not rejected, parsed once', async () => {
const text = `{"version":3,"note":"\\"${'['.repeat(2000)}\\\\ \\" [[[["}`
const pdf = craftPdf([
streamObject({
num: 2,
dict: `/Type /EmbeddedFile /Length ${text.length}`,
data: ascii(text),
}),
])
const calls = await withSpy(async () => {
const located = await locateAttachment(pdf)
expect(located.data).toMatchObject({ version: 3 })
})
expect(calls).toBe(1)
})
it('the real envelope passes the pre-scan and is parsed once', async () => {
const calls = await withSpy(async () => {
const located = await locateAttachment(exported)
expect(located.data).toMatchObject({ envelopeVersion: 1 })
})
expect(calls).toBe(1)
})
})
describe('one 16 MiB budget shared by every candidate', () => { describe('one 16 MiB budget shared by every candidate', () => {
const garbageObject = (num: number) => const garbageObject = (num: number) =>
streamObject({ streamObject({