From dc78a0083516e4b9c9ebedcdbd1a84701dac321b Mon Sep 17 00:00:00 2001 From: bchanot Date: Sat, 10 Oct 2026 15:44:20 +0200 Subject: [PATCH] feat(pdf): archive check tool pnpm report:compare (attachment vs its v3 reading, unresolved items flagged, --diff/--summary/--identity); readAttachedDocument split out of extractReportJson; stale scaffold:species script removed; docs --- .claude/tasks/TODO.md | 5 +- CHANGELOG.md | 2 +- CLAUDE.md | 7 +- README.md | 3 +- apps/app/package.json | 4 +- apps/app/scripts/compare-reports.ts | 135 ++++++++ apps/app/scripts/compare/report.ts | 330 ++++++++++++++++++++ apps/app/src/pdf/attachment.ts | 70 +++-- apps/app/tests/unit/scripts.compare.test.ts | 203 ++++++++++++ apps/app/tsconfig.json | 2 +- docs/PDF-FORMAT.md | 24 ++ package.json | 2 +- pnpm-lock.yaml | 3 + 13 files changed, 753 insertions(+), 37 deletions(-) create mode 100644 apps/app/scripts/compare-reports.ts create mode 100644 apps/app/scripts/compare/report.ts create mode 100644 apps/app/tests/unit/scripts.compare.test.ts diff --git a/.claude/tasks/TODO.md b/.claude/tasks/TODO.md index 5775836b..30f00fcf 100644 --- a/.claude/tasks/TODO.md +++ b/.claude/tasks/TODO.md @@ -95,7 +95,10 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo - [x] 5.5 PDF shim + attachment v3 + v2 import (gate D3) - [x] 5.6 delete legacy (registry, species/*, zoneMatch, spineZones, svgConvention, D4 tools), SVG convention doc replaced, CLAUDE.md map, CHANGELOG, plan note, memory - [x] 5.7 visual before/after (cmp renders), root check + docker + real stack green; verifier ×3 (gaps closed), security PASS (LOW fixed); EVAL-011. Manual FR/EN flow = user validation - - [ ] 5 follow-up: allowlist SVG sanitiser before Storage-served assets (Step 13); orphan findings UX (D5) at the report UX step; per-leaf adapter test over the 612 legacy leaves (map-level coverage today) + - [ ] 5.8 per-leaf migration (user ask 2026-10-10, during manual validation): every legacy leaf (612 = 442 identical, 141 moved, 29 removed) as a v2 finding → migrated → resolved to a draft leaf (anatomy, map file) and through sheetIndex.nodeAt + PDF roundtrip (app); the 27 legacy annotations as zones on their views + - [ ] 5.9 archive check tool `pnpm report:compare ...`: v2 attachment vs migrated v3 side by side, unresolved flagged (removed vs unknown), identity opt-in, --diff of two PDFs, --summary; attachment.ts split read/parse (additive, D3); docs PDF-FORMAT/CLAUDE/README + - [ ] 5.10 final controls on the last lot: stale root script scaffold:species removed; fresh verifier + security on the whole diff (criteria 16-18); EVAL-011 addendum + - [ ] 5 follow-up: allowlist SVG sanitiser before Storage-served assets (Step 13); orphan findings UX (D5) at the report UX step --- diff --git a/CHANGELOG.md b/CHANGELOG.md index 926907a1..d23809e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Versions follow [Semantic Versioning](https://semver.org/). - **Supabase foundation** — local stack (`supabase/config.toml`), migrations (schemas `private` / `audit`, `profiles`, platform roles with audited RPCs), pgTAP suite run on PGlite (`pnpm test:run`) and on the real stack (`pnpm db:test`), `@faunosteo/supabase` typed client + generated types. No UI change. - **Anatomical sheet model** — `@faunosteo/domain`: canonical sheet document (views tree, annotations with stable ids and explicit laterality, ordered vertebral regions, typed finding categories, explicit annotation → finding zone links, FR/EN labels) with integrity validation; tables `species`, `anatomical_sheets`, `anatomical_sheet_versions` (published rows immutable, one draft per sheet, deletion preconditions), `anatomical_sheet_assets`; lifecycle RPCs (create, draft, publish, deprecate, delete, fork with provenance, archive, assets) audited and under RLS. Spec: `docs/development/ANATOMICAL_SHEET_MODEL.md`. -- **Application on the canonical sheet (Step 5)** — the editor renders the bundled reference sheet (curated equine draft, `SHEET_BUNDLE=draft`; a default build carries no sheet): markers drawn from the document with the legacy look (`@faunosteo/anatomy/render`), head view as a crop of the same drawing, zone interaction service, pickers on the sheet's regions (usual count, extension to the documented range) and finding zones. Report document **v3** (`packages/domain/report`): zones by annotation and view ids, findings by zone and path, severity codes, exact sheet reference; v2 PDFs and drafts migrate through the generated `report-migration-map.json`, unresolved items kept with their labels. Removed: species registry, label matchers, SVG convention parser, `/editors`, `/svg-editor`, `pathology-editor.html`, `scaffold:species`; the legacy sources now live under `supabase/seed/anatomical-sheets/equine/legacy-sources/`. +- **Application on the canonical sheet (Step 5)** — the editor renders the bundled reference sheet (curated equine draft, `SHEET_BUNDLE=draft`; a default build carries no sheet): markers drawn from the document with the legacy look (`@faunosteo/anatomy/render`), head view as a crop of the same drawing, zone interaction service, pickers on the sheet's regions (usual count, extension to the documented range) and finding zones. Report document **v3** (`packages/domain/report`): zones by annotation and view ids, findings by zone and path, severity codes, exact sheet reference; v2 PDFs and drafts migrate through the generated `report-migration-map.json`, unresolved items kept with their labels. Removed: species registry, label matchers, SVG convention parser, `/editors`, `/svg-editor`, `pathology-editor.html`, `scaffold:species`; the legacy sources now live under `supabase/seed/anatomical-sheets/equine/legacy-sources/`. Archive check tool `pnpm report:compare` (attachment vs its v3 reading, unresolved items flagged); the 612 legacy catalog leaves are tested through the migration in both packages. - **Equine sheet curation (Step 4c)** — document schema v2 (references registry, `review_status` on findings and annotations, configurable vertebral `count_range`; migration `20261010100000`, pgTAP 008); `@faunosteo/anatomy` curation pipeline: authored changeset (`curation/changes.json`, levels A/B/C, before/after records) applied to the legacy baseline → working draft with `CHANGELOG.md`, `REVIEW-REPORT.md` (10 sections for the osteopath) and before/after review drawings; one cleaned SVG asset, head view as a crop. Content: research-backed audit of the horse sheet (44 changes, 48 verifiable references): sphenoid zone restructured, NAV terminology, FR/EN translations, vertebral ranges, ten SVG points relocated, carpus zone added, veterinary red flags, legal disclaimer anchored on R. 243-6 / R. 243-8, osteopathic-model status on unvalidated concepts, 11 questions for the professional. Still a private DRAFT, not published. Docker resources prefixed `faunosteo_` (Compose project and container). - **Equine import (Step 4b)** — `@faunosteo/anatomy` deterministic importer: `Horse.svg` + `Horse_head.svg` + `pathologies.json` → canonical document, legacy id map, French import report, idempotent seed creating the official sheet `faunosteo-cheval` as a private DRAFT (never published by code). Storage bucket `anatomical-sheet-assets` (content-addressed SVGs, read through the sheet, operator upload script). diff --git a/CLAUDE.md b/CLAUDE.md index 209d385b..d21c1166 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -88,6 +88,7 @@ pnpm db:types:check # CI diff pnpm build:preview # SHEET_BUNDLE=draft build (manual validation, preview deployments) pnpm preview:draft # same, then serves it pnpm import:equine # regenerate legacy-import/ and draft/ (document, migration map, review drawings) +pnpm report:compare ... # archive check: attachment vs its v3 reading, unresolved items flagged (docs/PDF-FORMAT.md) ``` ## Docker commands @@ -125,7 +126,8 @@ pnpm test # vitest watch — dev loop ``` Test files live in `apps/app/tests/unit/*.test.ts(x)`; golden fixtures in -`apps/app/tests/fixtures/`. Setup: `apps/app/tests/setup.ts` imports +`apps/app/tests/fixtures/`. The 612 legacy catalog leaves run through the v2 → v3 migration +in both `packages/anatomy` (against the draft document) and the app (sheet index + PDF roundtrip). Setup: `apps/app/tests/setup.ts` imports @testing-library/jest-dom. Suites run on the bundled reference sheet (`tests/helpers/sheet.ts`); frozen v2 fixtures under `tests/fixtures/`. SQL: `supabase/tests/*.sql` (pgTAP) run by `packages/supabase/tests/db/pgtap.test.ts` on PGlite (part of `pnpm test:run`) and by `pnpm db:test` on the real stack. @@ -148,6 +150,7 @@ faunosteo/ ├─ apps/app/ ← @faunosteo/app (everything below until packages/) ├─ public/ ← logos only (no anatomy asset: the sheet is bundled) ├─ vite/sheet-bundle.ts ← Vite plugin: `virtual:faunosteo-sheet` (SHEET_BUNDLE=draft|none) +├─ scripts/compare-reports.ts ← `pnpm report:compare`: archive PDF vs v3 reading (pure part in scripts/compare/report.ts) ├─ src/ │ ├─ app/ │ │ ├─ App.tsx ← root router @@ -414,7 +417,7 @@ Database (Supabase) - Annotation rendering changes require updating docs/ANNOTATION-RENDERING.md first. - Never modify the legacy sources under `supabase/seed/anatomical-sheets/equine/legacy-sources/` — they are the importer's input (the user's historical files). -- Ask before any change to importReportFromPdf or extractReportJson — these handle user data. +- Ask before any change to importReportFromPdf, extractReportJson or readAttachedDocument — these handle user data. - After PDF roundtrip tests fail: check attachment bytes, then Zod validation (v3, or v2 through the migration map), then DecompressionStream inflation. - Node 24 / pnpm 10.13.1: `corepack enable` or `corepack pnpm`; never a global install. - SQL change: `pnpm --filter @faunosteo/supabase test:run` (PGlite) before commit; `pnpm db:reset && pnpm db:test` diff --git a/README.md b/README.md index 2a19e2d1..c075f883 100644 --- a/README.md +++ b/README.md @@ -78,8 +78,9 @@ docker compose up --build ```bash pnpm typecheck # TypeScript strict, no emit pnpm lint # ESLint 9 flat config, 0 warnings -pnpm test:run # vitest run (app 278 + supabase 13, incl. 79 pgTAP checks on PGlite) +pnpm test:run # vitest run (domain, anatomy, supabase/PGlite, app) pnpm build # tsc + vite build → apps/app/dist/ +pnpm report:compare ... # archive PDF vs its v3 reading, unresolved items flagged ``` --- diff --git a/apps/app/package.json b/apps/app/package.json index 6e9b44a5..b90f8665 100644 --- a/apps/app/package.json +++ b/apps/app/package.json @@ -11,7 +11,8 @@ "typecheck": "tsc -p tsconfig.json --noEmit", "test": "vitest", "test:run": "vitest run", - "lint": "eslint . --max-warnings 0" + "lint": "eslint . --max-warnings 0", + "report:compare": "tsx scripts/compare-reports.ts" }, "dependencies": { "@faunosteo/anatomy": "workspace:*", @@ -39,6 +40,7 @@ "eslint-plugin-react-hooks": "^5.1.0", "globals": "^15.14.0", "happy-dom": "^20.10.6", + "tsx": "^4.23.15", "typescript": "^5.7.2", "typescript-eslint": "^8.19.0", "vite": "^7.3.6", diff --git a/apps/app/scripts/compare-reports.ts b/apps/app/scripts/compare-reports.ts new file mode 100644 index 00000000..80d30e17 --- /dev/null +++ b/apps/app/scripts/compare-reports.ts @@ -0,0 +1,135 @@ +/** + * Archive check: reads the report attached to one or more Faunosteo PDFs and + * prints it next to its v3 reading on the bundled reference sheet. + * + * pnpm report:compare ... [--summary] [--identity] + * [--locale fr|en] [--diff a.pdf b.pdf] + * + * A v2 archive PDF is shown legacy id → canonical id, line by line; a v3 PDF + * is checked against the sheet. Items the sheet cannot place are marked + * `!! UNRESOLVED` with the reason. `--diff` compares the v3 content of two + * PDFs (an archive and its re-export after import, typically). Names are + * printed only with `--identity`. Reads only; never writes a file. + */ +import { readFileSync, readdirSync, statSync } from 'node:fs' +import { basename, join, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { PDFDocument } from 'pdf-lib' +import { + readReportDocument, + REPORT_VERSION, + UnsupportedReportVersionError, +} from '@faunosteo/domain' +import { readAttachedDocument } from '../src/pdf/attachment' +import { MAX_PDF_BYTES } from '../src/pdf/import' +import { parseBundledSheet } from '../src/sheets/provider' +import { buildSheetIndex, type SheetIndex } from '../src/services/sheetIndex' +import { + compareReport, + diffLines, + parseArgs, + renderComparison, + renderSummary, + type CliArgs, + type Comparison, +} from './compare/report' + +const SEED_DIR = resolve( + fileURLToPath(import.meta.url), + '../../../../supabase/seed/anatomical-sheets/equine/draft', +) +/** pnpm runs the script from apps/app; paths are typed from the caller's folder. */ +const CWD = process.env.INIT_CWD ?? process.cwd() + +function loadSheet(): SheetIndex { + const json = (name: string) => + JSON.parse(readFileSync(join(SEED_DIR, name), 'utf8')) as never + // The comparison needs the document and the map, not the drawings. + return buildSheetIndex( + parseBundledSheet({ + status: 'draft', + ref: json('sheet-ref.json'), + document: json('document.json'), + migrationMap: json('report-migration-map.json'), + assets: {}, + }), + ) +} + +function expandFiles(args: string[]): string[] { + const out: string[] = [] + for (const arg of args) { + const path = resolve(CWD, arg) + if (statSync(path).isDirectory()) { + const pdfs = readdirSync(path) + .filter((f) => f.toLowerCase().endsWith('.pdf')) + .sort() + out.push(...pdfs.map((f) => join(path, f))) + } else out.push(path) + } + return out +} + +async function compareFile( + path: string, + sheet: SheetIndex, + args: CliArgs, +): Promise { + const bytes = new Uint8Array(readFileSync(path)) + if (bytes.byteLength > MAX_PDF_BYTES) throw new Error('file too large') + const pdf = await PDFDocument.load(bytes, { ignoreEncryption: true }) + const raw = await readAttachedDocument(pdf) + const version = (raw as { version: number }).version + try { + const outcome = readReportDocument(raw, sheet.sheet.migrationMap) + return compareReport(raw, outcome, version !== REPORT_VERSION, sheet, args) + } catch (err) { + if (err instanceof UnsupportedReportVersionError) { + throw new Error(`report version ${version} cannot be read`) + } + throw err + } +} + +async function main(argv: string[]): Promise { + const args = parseArgs(argv) + const files = expandFiles(args.files) + if (files.length === 0 || (args.diff && files.length !== 2)) { + console.error( + 'usage: pnpm report:compare ... [--summary] [--identity] [--locale fr|en] [--diff a.pdf b.pdf]', + ) + return 1 + } + const sheet = loadSheet() + const results: Comparison[] = [] + let failures = 0 + for (const file of files) { + const name = basename(file) + try { + const c = await compareFile(file, sheet, args) + results.push(c) + if (!args.diff) { + const lines = args.summary + ? [renderSummary(name, c)] + : [...renderComparison(name, c), ''] + console.log(lines.join('\n')) + } + } catch (err) { + failures += 1 + console.log(`!! ${name}: ${err instanceof Error ? err.message : err}`) + } + } + if (args.diff && results.length === 2) { + const diff = diffLines(results[0]!.normalized, results[1]!.normalized) + console.log(diff.length ? diff.join('\n') : 'identical v3 content') + } + return failures ? 2 : 0 +} + +main(process.argv.slice(2)).then( + (code) => process.exit(code), + (err) => { + console.error(err instanceof Error ? err.message : err) + process.exit(1) + }, +) diff --git a/apps/app/scripts/compare/report.ts b/apps/app/scripts/compare/report.ts new file mode 100644 index 00000000..451f6b1a --- /dev/null +++ b/apps/app/scripts/compare/report.ts @@ -0,0 +1,330 @@ +import { + ReportV2Schema, + findingPathKey, + type Finding, + type FindingV2, + type MigrationOutcome, + type ObservationZoneV2, + type Report, + type ReportZone, +} from '@faunosteo/domain' +import type { Locale } from '../../src/i18n/types' +import { pickLabel, type SheetIndex } from '../../src/services/sheetIndex' + +/** + * Pure comparison of a report document found in a PDF with its v3 reading: + * a v2 archive is shown legacy id → canonical id line by line, a v3 report + * is checked against the bundled sheet. Every item the sheet cannot place is + * marked `!! UNRESOLVED` with the reason, nothing is summarised away. + */ + +export const UNRESOLVED = '!! UNRESOLVED' + +export interface CompareOptions { + locale: Locale + /** Print practitioner, animal and owner names (off by default). */ + identity: boolean +} + +export interface Comparison { + /** `v2` when the attachment was migrated, `v3` when read as is. */ + source: 'v2' | 'v3' + header: string[] + observations: string[][] + footer: string[] + /** Lines in the v3 vocabulary only, for `--diff` between two PDFs. */ + normalized: string[] + counts: Counts +} + +export interface Counts { + observations: number + zones: number + unresolvedZones: number + findings: number + unresolvedFindings: number +} + +const q = (text: string) => `"${text}"` + +function zoneRef(z: { annotationId: string; viewId: string; side?: string }) { + return `${z.annotationId}@${z.viewId}${z.side ? ' ' + z.side : ''}` +} + +function findingRef(f: { findingZoneId: string; path: readonly string[] }) { + return findingPathKey(f.findingZoneId, f.path) +} + +function vertebraeOf(z: { vertebrae?: { region: string; index: number }[] }) { + return z.vertebrae?.length + ? ` [${z.vertebrae.map((v) => `${v.region}${v.index}`).join(', ')}]` + : '' +} + +/** A v3 zone against the sheet: current label, or why it cannot be shown. */ +function describeZone( + z: ReportZone, + sheet: SheetIndex, + locale: Locale, +): string { + const ref = `${zoneRef(z)}${vertebraeOf(z)}` + const anno = sheet.annotation(z.annotationId) + if (z.unresolved) { + return `${ref} ${q(z.labelFrSnapshot ?? '')} ${UNRESOLVED} legacy zone kept from the archive` + } + if (!anno || anno.view_id !== z.viewId) { + return `${ref} ${q(z.labelFrSnapshot ?? '')} !! not in the sheet (unflagged)` + } + return `${ref} ${q(pickLabel(sheet.name(anno), locale))}` +} + +/** A v3 finding against the sheet: current label, or why it cannot be shown. */ +function describeFinding( + f: Finding, + sheet: SheetIndex, + locale: Locale, +): string { + const snapshot = q(locale === 'en' ? f.labelEnSnapshot : f.labelFrSnapshot) + const vet = f.requiresVetReferral ? ' (!)' : '' + const side = f.side ? ` ${f.side}` : '' + const ref = `${findingRef(f)}${side}${vet}` + if (f.unresolved) { + return `${ref} ${snapshot} ${UNRESOLVED} kept with its snapshot` + } + const node = sheet.nodeAt(f.findingZoneId, f.path) + if (!node) return `${ref} ${snapshot} !! not in the sheet (unflagged)` + if (node.children.length > 0) { + return `${ref} ${snapshot} !! points at a category, not a leaf` + } + return `${ref} ${q(pickLabel(node.name, locale))}` +} + +/** Legacy zone → migrated zone. */ +function describeZoneMigration( + legacy: ObservationZoneV2, + out: ReportZone, + sheet: SheetIndex, + locale: Locale, +): string { + const side = legacy.side ? ` ${legacy.side}` : '' + const from = `${legacy.boneId}${legacy.detailSourceId ? '@' + legacy.detailSourceId : ''}${side}${vertebraeOf(legacy)}` + if (out.unresolved) { + return `${from} ${UNRESOLVED} unknown legacy id, kept as ${zoneRef(out)}` + } + const anno = sheet.annotation(out.annotationId) + const label = anno ? q(pickLabel(sheet.name(anno), locale)) : '!! missing' + return `${from} ${q(out.labelFrSnapshot ?? '')} → ${zoneRef(out)} ${label}` +} + +/** Legacy finding → migrated finding, with the fate the map decided. */ +function describeFindingMigration( + legacy: FindingV2, + out: Finding, + sheet: SheetIndex, + locale: Locale, +): string { + const map = sheet.sheet.migrationMap.finding_paths + const vet = legacy.requiresVetReferral ? ' (!)' : '' + const key = `${findingPathKey(legacy.zoneId, legacy.path)}${vet}` + const snapshot = q( + locale === 'en' ? legacy.labelEnSnapshot : legacy.labelFrSnapshot, + ) + if (out.unresolved) { + const why = Object.hasOwn(map, findingPathKey(legacy.zoneId, legacy.path)) + ? 'removed from the sheet, kept with its label' + : 'never in the legacy catalog, kept with its label' + return `${key} ${snapshot} ${UNRESOLVED} ${why}` + } + const node = sheet.nodeAt(out.findingZoneId, out.path) + const label = node ? q(pickLabel(node.name, locale)) : '!! missing' + const target = + findingRef(out) === findingPathKey(legacy.zoneId, legacy.path) + ? 'same' + : findingRef(out) + return `${key} ${snapshot} → ${target} ${label}` +} + +function observationLines( + obs: Report['observations'][number], + sheet: SheetIndex, + opts: CompareOptions, +): string[] { + const lines = [ + `[${obs.id}] severity ${obs.severity ?? 'none'}`, + ...obs.zones.map((z) => `zone ${describeZone(z, sheet, opts.locale)}`), + ...obs.findings.map( + (f) => `finding ${describeFinding(f, sheet, opts.locale)}`, + ), + ...obs.freeTextFindings.map((t) => `text ${q(t)}`), + ] + if (obs.comment) lines.push(`comment ${q(obs.comment)}`) + return lines +} + +function migratedObservationLines( + raw: unknown, + report: Report, + sheet: SheetIndex, + opts: CompareOptions, +): string[][] { + const v2 = ReportV2Schema.parse(raw) + return v2.observations.map((legacy, i) => { + const out = report.observations[i]! + const severity = legacy.severity + ? `${legacy.severity} → ${out.severity}` + : 'none' + const lines = [`[${legacy.id}] severity ${severity}`] + legacy.zones.forEach((z, j) => { + const line = describeZoneMigration(z, out.zones[j]!, sheet, opts.locale) + lines.push(`zone ${line}`) + }) + legacy.findings.forEach((f, j) => { + const line = describeFindingMigration( + f, + out.findings[j]!, + sheet, + opts.locale, + ) + lines.push(`finding ${line}`) + }) + lines.push(...legacy.freeTextFindings.map((t) => `text ${q(t)}`)) + if (legacy.comment) lines.push(`comment ${q(legacy.comment)}`) + return lines + }) +} + +function countsOf(report: Report): Counts { + const zones = report.observations.flatMap((o) => o.zones) + const findings = report.observations.flatMap((o) => o.findings) + return { + observations: report.observations.length, + zones: zones.length, + unresolvedZones: zones.filter((z) => z.unresolved).length, + findings: findings.length, + unresolvedFindings: findings.filter((f) => f.unresolved).length, + } +} + +function headerLines( + source: 'v2' | 'v3', + report: Report, + sheet: SheetIndex, + opts: CompareOptions, +): string[] { + const ref = sheet.sheet.ref + const sameSheet = + report.sheet.sheetId === ref.sheetId && + report.sheet.documentSha256 === ref.documentSha256 + const sheetLine = sameSheet + ? `sheet ${ref.sheetId} (${sheet.sheet.status}, matches the bundled sheet)` + : `sheet ${report.sheet.sheetId} !! written on another sheet document (sha ${report.sheet.documentSha256.slice(0, 12)}…)` + const c = countsOf(report) + const lines = [ + source === 'v2' ? 'v2 → v3 (migrated)' : 'v3 (read as is)', + sheetLine, + `id ${report.id} · date ${report.date} · ${c.observations} observations · ${c.zones} zones (${c.unresolvedZones} unresolved) · ${c.findings} findings (${c.unresolvedFindings} unresolved)`, + ] + if (opts.identity) { + lines.push( + `practitioner ${q(report.practitioner.name)} · animal ${q(report.animal.name)} · owner ${q(report.owner.name)}`, + ) + } + return lines +} + +function footerLines(report: Report): string[] { + const lines: string[] = [] + if (report.conclusion) lines.push(`conclusion ${q(report.conclusion)}`) + if (report.recommendations) { + lines.push(`recommendations ${q(report.recommendations)}`) + } + if (report.nextAppointment) { + lines.push(`next appointment ${JSON.stringify(report.nextAppointment)}`) + } + return lines +} + +/** + * `raw` is the attachment as found in the PDF, `outcome` its v3 reading + * (`readReportDocument`); `migrated` tells which vocabulary the lines use. + */ +export function compareReport( + raw: unknown, + outcome: MigrationOutcome, + migrated: boolean, + sheet: SheetIndex, + opts: CompareOptions, +): Comparison { + const { report } = outcome + const source = migrated ? 'v2' : 'v3' + const v3Lines = report.observations.map((o) => + observationLines(o, sheet, opts), + ) + return { + source, + header: headerLines(source, report, sheet, opts), + observations: migrated + ? migratedObservationLines(raw, report, sheet, opts) + : v3Lines, + footer: footerLines(report), + normalized: [...v3Lines.flat(), ...footerLines(report)], + counts: countsOf(report), + } +} + +export function renderComparison(name: string, c: Comparison): string[] { + return [ + `== ${name}`, + ...c.header.map((l) => ` ${l}`), + ...c.observations.flatMap((lines) => + lines.map((l, i) => (i === 0 ? ` ${l}` : ` ${l}`)), + ), + ...c.footer.map((l) => ` ${l}`), + ] +} + +export function renderSummary(name: string, c: Comparison): string { + const n = c.counts + return `${name} · ${c.source === 'v2' ? 'v2→v3' : 'v3'} · ${n.observations} obs · zones ${n.zones}/${n.unresolvedZones} unresolved · findings ${n.findings}/${n.unresolvedFindings} unresolved` +} + +/** Lines of A absent from B and lines of B absent from A, in order. */ +export function diffLines(a: string[], b: string[]): string[] { + const inB = new Set(b) + const inA = new Set(a) + return [ + ...a.filter((l) => !inB.has(l)).map((l) => `- ${l}`), + ...b.filter((l) => !inA.has(l)).map((l) => `+ ${l}`), + ] +} + +export interface CliArgs extends CompareOptions { + files: string[] + summary: boolean + diff: boolean +} + +export function parseArgs(argv: string[]): CliArgs { + const args: CliArgs = { + files: [], + locale: 'fr', + identity: false, + summary: false, + diff: false, + } + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i]! + if (arg === '--identity') args.identity = true + else if (arg === '--summary') args.summary = true + else if (arg === '--diff') args.diff = true + else if (arg === '--locale') { + const value = argv[++i] + if (value !== 'fr' && value !== 'en') { + throw new Error('--locale takes fr or en') + } + args.locale = value + } else if (arg.startsWith('--')) throw new Error(`unknown option ${arg}`) + else args.files.push(arg) + } + return args +} diff --git a/apps/app/src/pdf/attachment.ts b/apps/app/src/pdf/attachment.ts index 731d9ebd..446d2120 100644 --- a/apps/app/src/pdf/attachment.ts +++ b/apps/app/src/pdf/attachment.ts @@ -43,10 +43,45 @@ export interface ExtractedReport extends MigrationOutcome { migrated: boolean } +/** + * The raw report document carried by the PDF, of whatever version, before + * any migration: what the archive check tool shows next to the v3 result. + * Throws when the PDF carries no Faunosteo attachment or an oversized one. + */ +export async function readAttachedDocument( + pdfDoc: PDFDocument, +): Promise { + const bytes = await attachmentBytes(pdfDoc) + if (bytes.byteLength > MAX_ATTACHMENT_BYTES) { + throw new Error('attachment too large') + } + const data: unknown = JSON.parse(new TextDecoder().decode(bytes)) + // No numeric version at all: not a Faunosteo attachment (legacy message), not an old one. + if (versionOf(data) === null) { + throw new Error('attachment carries no report version') + } + return data +} + export async function extractReportJson( pdfDoc: PDFDocument, map: ReportMigrationMap, ): Promise { + const data = await readAttachedDocument(pdfDoc) + try { + return { + ...readReportDocument(data, map), + migrated: versionOf(data) !== REPORT_VERSION, + } + } catch (err) { + if (err instanceof UnsupportedReportVersionError) { + throw new IncompatibleReportVersionError(err.foundVersion) + } + throw err + } +} + +async function attachmentBytes(pdfDoc: PDFDocument): Promise { const maybeApi = pdfDoc as unknown as { getAttachments?: () => Array<{ name: string; data: Uint8Array }> } @@ -57,44 +92,21 @@ export async function extractReportJson( list.find( (a) => a.name.startsWith('osteo-report') && a.name.endsWith('.json'), ) ?? list[0] - return parseAttachment(hit.data, map) + return hit.data } } const fallback = await traverseForJsonAttachment(pdfDoc) if (!fallback) throw new Error('No osteo-report attachment found') - return parseAttachment(fallback, map) + return fallback } /** Hard ceiling on an attachment (inflated or not): a report is a few hundred KB at most. */ const MAX_ATTACHMENT_BYTES = 16 * 1024 * 1024 -function parseAttachment( - bytes: Uint8Array, - map: ReportMigrationMap, -): ExtractedReport { - if (bytes.byteLength > MAX_ATTACHMENT_BYTES) { - throw new Error('attachment too large') - } - const data: unknown = JSON.parse(new TextDecoder().decode(bytes)) - const version = - data !== null && - typeof data === 'object' && - typeof (data as { version?: unknown }).version === 'number' - ? (data as { version: number }).version - : null - // No numeric version at all: not a Faunosteo attachment (legacy message), not an old one. - if (version === null) throw new Error('attachment carries no report version') - try { - return { - ...readReportDocument(data, map), - migrated: version !== REPORT_VERSION, - } - } catch (err) { - if (err instanceof UnsupportedReportVersionError) { - throw new IncompatibleReportVersionError(err.foundVersion) - } - throw err - } +function versionOf(data: unknown): number | null { + if (data === null || typeof data !== 'object') return null + const v = (data as { version?: unknown }).version + return typeof v === 'number' ? v : null } interface RawObj { diff --git a/apps/app/tests/unit/scripts.compare.test.ts b/apps/app/tests/unit/scripts.compare.test.ts new file mode 100644 index 00000000..568337fb --- /dev/null +++ b/apps/app/tests/unit/scripts.compare.test.ts @@ -0,0 +1,203 @@ +/** + * The archive check tool (scripts/compare-reports.ts) on the frozen v2 PDF, + * on a synthetic v2 report with removed and unknown items, and on a v3 report. + */ +import { describe, it, expect, beforeAll } from 'vitest' +import { PDFDocument } from 'pdf-lib' +import { readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' +import { dirname, resolve } from 'node:path' +import { readReportDocument, type ReportV2 } from '@faunosteo/domain' +import { readAttachedDocument } from '@/pdf/attachment' +import type { SheetIndex } from '@/services/sheetIndex' +import { + compareReport, + diffLines, + parseArgs, + renderComparison, + renderSummary, + UNRESOLVED, +} from '../../scripts/compare/report' +import { loadTestSheet, minimalReport } from '../helpers/sheet' + +const HERE = dirname(fileURLToPath(import.meta.url)) +const OPTS = { locale: 'fr' as const, identity: false } + +let sheet: SheetIndex + +function minimalV2(): ReportV2 { + return JSON.parse( + readFileSync(resolve(HERE, '../fixtures/report-v2.minimal.json'), 'utf8'), + ) as ReportV2 +} + +function compareRaw(raw: unknown, identity = false) { + const outcome = readReportDocument(raw, sheet.sheet.migrationMap) + const migrated = (raw as { version: number }).version !== 3 + return compareReport(raw, outcome, migrated, sheet, { ...OPTS, identity }) +} + +beforeAll(async () => { + sheet = await loadTestSheet() +}) + +describe('compareReport on the frozen v2 archive PDF', () => { + it('shows every legacy zone and finding next to its canonical target', async () => { + const bytes = readFileSync(resolve(HERE, '../fixtures/report-v2.full.pdf')) + const pdf = await PDFDocument.load(new Uint8Array(bytes)) + const raw = await readAttachedDocument(pdf) + const c = compareRaw(raw) + const lines = renderComparison('full.pdf', c) + expect(c.source).toBe('v2') + expect(lines[0]).toBe('== full.pdf') + expect(lines).toContain(' v2 → v3 (migrated)') + expect(lines.join('\n')).toContain( + 'point-mpbwtst0 left "Hanche (coxo-fémorale)" → hip@main left "Hanche (coxo-fémorale)"', + ) + expect(lines.join('\n')).toContain( + 'cervical/cervical-observations/stumbling-proprioceptive (!) "Trébuchements / déficit proprioceptif rapporté" → cervical/cervical-consultation-reasons/stumbling-proprioceptive', + ) + expect(lines.join('\n')).toContain('severity modérée → moderate') + expect(lines.some((l) => l.includes(UNRESOLVED))).toBe(false) + expect(c.counts).toEqual({ + observations: 3, + zones: 5, + unresolvedZones: 0, + findings: 4, + unresolvedFindings: 0, + }) + expect(renderSummary('full.pdf', c)).toBe( + 'full.pdf · v2→v3 · 3 obs · zones 5/0 unresolved · findings 4/0 unresolved', + ) + expect(lines.join('\n')).not.toContain('practitioner') + expect(renderComparison('x', compareRaw(raw, true)).join('\n')).toContain( + 'practitioner "Dr. Claire Dupont"', + ) + }) +}) + +describe('compareReport on unresolved items', () => { + it('names a removed finding, a finding unknown to the catalog and an unknown zone', () => { + const v2: ReportV2 = { + ...minimalV2(), + observations: [ + { + id: 'o', + zones: [{ boneId: 'point-gone' }], + findings: [ + { + zoneId: 'ssb', + path: ['ssb-dysfunctions', 'sutures', 'sphenozygomatic'], + labelFrSnapshot: 'Suture sphéno-zygomatique', + labelEnSnapshot: 'Sphenozygomatic suture', + requiresVetReferral: true, + }, + { + zoneId: 'nowhere', + path: ['x'], + labelFrSnapshot: 'X', + labelEnSnapshot: 'X', + }, + ], + freeTextFindings: [], + comment: '', + }, + ], + } + const text = renderComparison('odd.pdf', compareRaw(v2)).join('\n') + expect(text).toContain( + `point-gone ${UNRESOLVED} unknown legacy id, kept as point-gone@main`, + ) + expect(text).toContain( + `ssb/ssb-dysfunctions/sutures/sphenozygomatic (!) "Suture sphéno-zygomatique" ${UNRESOLVED} removed from the sheet, kept with its label`, + ) + expect(text).toContain( + `nowhere/x "X" ${UNRESOLVED} never in the legacy catalog, kept with its label`, + ) + expect(text).toContain('1 zones (1 unresolved) · 2 findings (2 unresolved)') + }) + + it('reads a v3 report as is and flags what the sheet cannot place', () => { + const report = minimalReport(sheet, { + observations: [ + { + id: 'o3', + zones: [{ annotationId: 'hip', viewId: 'main', side: 'left' }], + findings: [ + { + findingZoneId: 'hip', + path: ['hip-dysfunctions', 'articular', 'restriction-flexion'], + labelFrSnapshot: 'Restriction en flexion', + labelEnSnapshot: 'Flexion restriction', + }, + { + findingZoneId: 'atm', + path: ['atm-observations', 'headshaking'], + labelFrSnapshot: 'Headshaking (secouement de tête)', + labelEnSnapshot: 'Headshaking', + unresolved: true, + }, + { + findingZoneId: 'hip', + path: ['hip-dysfunctions'], + labelFrSnapshot: 'Catégorie', + labelEnSnapshot: 'Category', + }, + ], + freeTextFindings: [], + comment: '', + }, + ], + }) + const c = compareRaw(report) + const text = renderComparison('v3.pdf', c).join('\n') + expect(c.source).toBe('v3') + expect(text).toContain('v3 (read as is)') + expect(text).toContain('hip@main left "Hanche (coxo-fémorale)"') + expect(text).toContain( + 'hip/hip-dysfunctions/articular/restriction-flexion "Restriction en flexion"', + ) + expect(text).toContain( + `atm/atm-observations/headshaking "Headshaking (secouement de tête)" ${UNRESOLVED} kept with its snapshot`, + ) + expect(text).toContain('!! points at a category, not a leaf') + expect(c.counts.unresolvedFindings).toBe(1) + }) + + it('warns when a v3 report was written on another sheet document', () => { + const report = minimalReport(sheet, { + sheet: { ...sheet.sheet.ref, documentSha256: 'b'.repeat(64) }, + }) + const text = renderComparison('other.pdf', compareRaw(report)).join('\n') + expect(text).toContain( + '!! written on another sheet document (sha bbbbbbbbbbbb…)', + ) + }) +}) + +describe('diff and arguments', () => { + it('diffs the v3 content of a migrated v2 report and its v3 re-reading', () => { + const v2 = minimalV2() + const migrated = readReportDocument(v2, sheet.sheet.migrationMap).report + const a = compareRaw(v2) + const b = compareRaw(migrated) + expect(diffLines(a.normalized, b.normalized)).toEqual([]) + const edited = { ...migrated, conclusion: 'changed' } + expect(diffLines(a.normalized, compareRaw(edited).normalized)).toEqual([ + '+ conclusion "changed"', + ]) + }) + + it('parses files and options, refuses unknown ones', () => { + expect(parseArgs(['a.pdf', '--summary', 'dir', '--locale', 'en'])).toEqual({ + files: ['a.pdf', 'dir'], + locale: 'en', + identity: false, + summary: true, + diff: false, + }) + expect(parseArgs(['--identity', '--diff', 'a', 'b']).identity).toBe(true) + expect(() => parseArgs(['--locale', 'de'])).toThrow('fr or en') + expect(() => parseArgs(['--json'])).toThrow('unknown option') + }) +}) diff --git a/apps/app/tsconfig.json b/apps/app/tsconfig.json index 5f0912a5..7bc3a25d 100644 --- a/apps/app/tsconfig.json +++ b/apps/app/tsconfig.json @@ -23,6 +23,6 @@ "outDir": "dist", "composite": false }, - "include": ["src", "tests"], + "include": ["src", "tests", "scripts"], "exclude": ["node_modules", "dist"] } diff --git a/docs/PDF-FORMAT.md b/docs/PDF-FORMAT.md index 02827d7d..1ffd04d2 100644 --- a/docs/PDF-FORMAT.md +++ b/docs/PDF-FORMAT.md @@ -3,6 +3,7 @@ Placeholder — to be written when implementing src/pdf/. Will cover: + - A4 portrait dimensions (595.28 × 841.89 pt) - Page layout: header / body / footer zones - Observation page structure @@ -27,3 +28,26 @@ Import (`importReportFromPdf(bytes, migrationMap)`) accepts: or finding the map cannot place is kept with its snapshots and flagged `unresolved: true`; the outcome reports `migrated` and the unresolved counts; - any other version: `ImportError('incompatible_version', foundVersion)`. + +## Checking an archive against its migration + +`pnpm report:compare ...` reads the attachment of one or +more PDFs and prints it next to its v3 reading on the bundled reference +sheet, without writing anything: + +- a v2 archive PDF: every legacy zone (`data-anno-id`, side, vertebrae) and + every legacy finding path is shown with its canonical target and the + current label of that target (`same` when the path did not change); +- a v3 PDF: every zone and finding is checked against the sheet, and the + header says whether the report was written on the bundled sheet document; +- anything the sheet cannot place is printed as `!! UNRESOLVED` with the + reason: unknown legacy zone id, finding removed from the sheet by the + curation, finding that never belonged to the legacy catalog, or a v3 + finding flagged `unresolved` in the report itself. + +Options: `--summary` (one line per file, counts and unresolved counts), +`--identity` (print the practitioner, animal and owner names, off by +default), `--locale fr|en` (labels), `--diff a.pdf b.pdf` (compare the v3 +content of two PDFs: an archive and the PDF exported after importing it +should print `identical v3 content`). The tool exits 2 when a file could not +be read (no attachment, unknown version, oversized) and prints the reason. diff --git a/package.json b/package.json index ae46981d..e8f3f99d 100644 --- a/package.json +++ b/package.json @@ -19,7 +19,7 @@ "lint": "pnpm -r lint", "format": "prettier --write .", "check": "pnpm typecheck && pnpm test:run && pnpm lint && pnpm build", - "scaffold:species": "pnpm --filter @faunosteo/app scaffold:species", + "report:compare": "pnpm --filter @faunosteo/app report:compare", "import:equine": "pnpm --filter @faunosteo/anatomy import:equine", "db:start": "supabase start", "db:stop": "supabase stop", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 61fdf406..11574827 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -90,6 +90,9 @@ importers: happy-dom: specifier: ^20.10.6 version: 20.10.6 + tsx: + specifier: ^4.23.15 + version: 4.23.15 typescript: specifier: ^5.7.2 version: 5.9.3