pnpm e2e:avatar is now an alias of pnpm db:sandbox avatar-e2e and the DEV script scripts/avatar-e2e.sh is removed; the suite fails on any target but the sandbox gateway and database; an occupied port 57800 stops the run before anything starts. The sandbox replaces the seed accounts' DEV password by a per-session secret.
16 KiB
Test strategy
Status: Step 0 deliverable, 2026-10-08. Current state: 32 files under
tests/unit/, 180 it( cases, vitest 3 + happy-dom, no browser, no e2e
(audit 13). All green per TODO as of 2026-07-12; not re-run in Step 0.
Revised 2026-10-25 (P0): counts on develop (2026-10-25 inventory): vitest
app 446, core-wasm 60, supabase 40; cargo core 383, worker 33 native + 16
Docker sandbox; 11 pgTAP files under the flat supabase/tests/. The locations
apps/app/tests/components, apps/app/tests/integration, apps/app/e2e,
supabase/tests/{db,rls} and supabase/seed/test/ in §1 and §3 are planned,
not created; the real-stack e2e lives in packages/supabase/tests/e2e. No
real-browser suite exists yet (TODO RR-8; V1-1 contract §12.5 proposes the
first one, decision D14). No CI has ever run (BDR-047): "CI" in the tables means the
prepared workflow.
V1-1.1 (2026-10-25): supabase vitest 44 (config test added), 12 pgTAP files
(012_workspaces.sql, 85 checks; 531 checks in all), also run on an isolated
real stack because the DEV stack stays untouched until the owner applies M1
(V1-1_OWNERS_ANIMALS.md §8.1).
V1-1.2 (2026-10-25): supabase vitest 53 (field-limits parity test added), 13
pgTAP files (013_clients.sql, 130 checks; 661 in all); the isolated stack is
a project tool (pnpm db:sandbox, SUPABASE.md §5.1) with its own suite under
packages/supabase/tests/sandbox (28 real-stack checks: HTTP per role,
concurrency, budget races, 20 000-client search timing).
V1-1.3 (2026-10-25): the first real-browser suites (D14,
docs/development/BROWSER_TESTS.md): Playwright 1.64.1 with axe 4.14.0 on
Chromium, desktop and 375 × 667, under apps/app/e2e. pnpm e2e:ui (no
backend) 26 cases, pnpm e2e:shell (isolated stack) 6 cases; vitest app 458
(Dialog and avatar confirmation wiring added).
V1-1.4 (2026-10-26): supabase vitest 59 (animal and relation parity cases
added), 14 pgTAP files (014_animals_relations.sql, 258 checks; 919 in all),
the sandbox suites 46 real-stack checks (animals.sandbox.ts 18: HTTP per
role, no lock outside the workspace, replays, relation races, caps and the
shared budget under concurrency, D9a through GoTrue admin, read timings on
20 000 animals), run in four migration states; 14 deliberate faults of M3
caught (V1-1_OWNERS_ANIMALS.md §8.3). The axe exceptions match rule,
selector, markup and route.
V1-1.5 (2026-10-26): supabase vitest 92 (access layer 33); sandbox suites 54
(clients-access.sandbox.ts 8: the access layer through supabase-js against
real GoTrue and PostgREST); app vitest 509 (client screens 22, form schema 20,
workspace and account change 7); e2e:shell 42 (client journeys 36, FR/EN,
desktop and 375 px, axe). The shell suite keeps no HTML report.
V1-1.6 (2026-10-26): supabase vitest 127 (animal and relation access 35);
sandbox suites 61 (animals-access.sandbox.ts 7: the same functions against
real GoTrue and PostgREST, replays, duplicates, end, void, archived, foreign
ids); app vitest 566 (animal screens 35, animal form schema and display rules
22); e2e:shell 82 (animal and relation journeys 40: ten journeys, FR/EN,
desktop and 375 px, axe). Counter-checks: 12 deliberate faults caught by the
unit suites, 2 by the browser journeys.
1. Layers
| Layer | Tool | Location | Runs in |
|---|---|---|---|
| Unit (pure TS, Zod, services) | vitest | apps/app/tests/unit, packages/*/tests |
CI on every PR |
| Component | vitest + @testing-library/react + happy-dom | apps/app/tests/components |
CI |
| Integration (app ↔ local Supabase) | vitest with supabase start, test users created via the admin API in a fixture |
apps/app/tests/integration |
CI job with the Supabase stack |
| Database (schema, functions, triggers) | pgTAP via supabase test db |
supabase/tests/db |
CI, before app tests |
| RLS | pgTAP, per table, per role | supabase/tests/rls |
CI, release gate |
| End-to-end | Playwright and axe on Chromium (desktop, 375 px) against a loopback Vite dev server: no backend (pnpm e2e:ui) or the isolated stack (pnpm e2e:shell), never DEV; request allowlist, fixture values kept out of URLs |
apps/app/e2e (docs/development/BROWSER_TESTS.md) |
e2e:ui in pnpm check:all when the pinned Chromium is installed (SKIPPED line otherwise); e2e:shell by hand with pnpm db:sandbox up; CI nightly + release gate planned |
| Migration | supabase db reset from zero + seed + regression fixtures |
CI | every PR touching supabase/ |
| PDF regression | cargo: the R4 oracles (O1 draw logs, O2 PDFs, O4 to O10) and the R4 digests of the writer's own bytes; vitest: the Step 6 goldens read back from the app export's file (tests/helpers/pdfFile.ts, no pdf-lib), the attachment roundtrip through the module's reader, the N-1 readers |
crates/core/tests/pdf_*.rs, apps/app/tests/unit/pdf.*.test.ts, packages/core-wasm/tests/pdf-*.test.ts |
CI |
| SQL/TS contract | generated types diff check | CI | every PR touching supabase/ |
| Rust core and facade | cargo test (unit + fixture tests reading the repository JSON in place), clippy -D warnings, cargo deny; ts-rs types drift check |
crates/*, pnpm rust:check (native, scripts/rust-check.sh; same file in the builder image), pnpm check:all |
before every Rust commit and every merge; CI job rust prepared, no runner yet |
| Rust/TS contract | vitest on the built module: byte equality with JSON.stringify(…, null, 2) on the real documents, ts-rs types against the Zod output types, the runtime (lazy load, trap → poison → offline re-instantiation, handle rebuild, cap, retry), panic and stack traps on pkg-proofs, served size budget and measurements |
packages/core-wasm/tests (refused, never skipped, when pkg/ or pkg-proofs/ is missing or stale) |
pnpm check:all (after the native wasm:build); CI check prepared |
| Freeze-then-switch oracles (R2) | outputs of the TypeScript frozen before the switch and checked by cargo: Zod output of the reference documents (byte-equal), 85 report / archive cases, 765 Zod verdicts, 49 editor cases, 26 sheet conformance fixtures (also inlined into the generated pgTAP 011, drift-checked), the three archive envelopes (deep-equal through the app path) |
crates/core/tests/{oracles,conformance,fixtures}, generators packages/domain/tests/r2-*.test.ts (deleted at R5.7, at 756b7ef; the oracles stay) and apps/app/tests/unit/archive.envelopes.test.ts (MAKE_R2_ORACLES / MAKE_R2_FREEZE, never regenerated without an owner decision) |
every Rust commit (pnpm rust:check), pnpm check:all |
| N-1 reader (permanent) | every draft and archive the core-fed application writes parses with the Zod schemas of the last TypeScript build (packages/core-wasm/tests/n-1, develop 1a7ee7d) |
apps/app/tests/unit/n-1.reader.test.ts, archive.envelopes.test.ts |
every commit (pnpm test:run) |
| CSP on the image (R2) | the real app image behind the DEPLOY.md CSP in headless Chromium: with 'wasm-unsafe-eval' the core runs the editor path, without it the degraded state and an untouched draft; no module request on /login, /app/account, / |
docker/csp-proof/app-image.sh |
before a merge touching the core loading or the image; manual (needs Chromium) |
| Worker sandbox | integration tests against the musl binary that ships: fixture corpus through the full sandbox, refusals, SIGSYS on a denied syscall, output cap, RLIMIT_AS abort, RLIMIT_CPU, deadline; seccomp allowlist derived by strace and compared |
crates/worker/tests/sandbox.rs, Dockerfile stage test (pnpm worker:test, Docker only, never native) |
every crates/worker commit, pnpm check:all; CI docker job prepared |
| Worker job loop | native unit tests on a fake backend and a fake decoder: call order claim → download → decode → upload → commit → purge, every child outcome mapped to a closed code, no deletion on a refused commit, lease left on transport failures, sweeper targets and acks, key validation, config and secret redaction | crates/worker/tests/job_loop.rs, crates/worker/src/* unit tests (pnpm rust:check) |
every Rust commit |
| Files e2e (real stack) | the worker container against the throwaway sandbox stack (since 2026-11-01; DEV before): clean upload and owner-only download (Bob, anon, quarantine, direct final-key write refused), replacement and purge after the grace (queue clock shifted through psql), late signed-upload bytes swept, hostile uploads rejected with closed codes, removal and drained queue | packages/supabase/tests/e2e/avatar-flow.e2e.ts (pnpm e2e:avatar = pnpm db:sandbox avatar-e2e, scripts/db-sandbox.sh) |
every Storage, worker or avatar change; not in check:all (needs the sandbox) |
Browser-only code (svgToPng, DOMParser in the importer) runs under
happy-dom where sufficient and under Playwright when real rendering matters
(raster hash).
2. Tests required BEFORE migration (plan Step 1)
These pin the behaviour that later steps must preserve. They are written against the current code, on real data, without changing the application.
| Area | Test to add | Why before |
|---|---|---|
| Click chain | EditorRoute.onZoneClick branches: detail trigger opens modal and is never added; no active observation → fallback then click lost; spine zone adds + opens picker and never removes; bilateral → picker, both sides present → removes both; plain toggle. Extract nothing yet: test through the route with mocked loadSvg. |
The chain moves to a service in Step 5; zero tests today (audit 13 §4). |
| Spine sync on real SVG | collectSpineVertebrae / syncSpineZones with bones parsed from the real Horse.svg (not fixtures). |
regionFromLabel is replaced by region codes; must prove equivalence. |
| SVG ↔ catalog | Parse both real SVGs, run matchZoneFindings on all 27 annotations against the 26 catalog zones: 26 score-0 matches, trigger unmatched, zero orphan catalog zones. Table from audit 06 §4 becomes the expected output. |
The importer must reproduce exactly this mapping. |
opens-detail ↔ manifest |
every data-anno-opens-detail value is a key of species.details, and every key has a trigger. |
Becomes parent_view_id/opened_by_annotation_id invariants. |
| Matcher parity | the inline matcher of pathology-editor.html (extracted by regex from the file at test time) gives the same result as zoneMatch.ts on all 26 + edge labels. |
LRN-026 recommendation; protects the editor until deletion. |
| Catalog copies parity | src/.../pathologies.json byte-equal to public/species/equine/pathologies.json. |
Guards the dual-copy quirk until removal. |
| Catalog invariants | zone.id === key, 26 zones, category ids per zone, requires_vet_referral only on leaves, the ssb-dysfunctions under sphenoid anomaly asserted as known (so a fix is a deliberate change). |
Importer normalisations are tested against these facts. |
| Full PDF roundtrip | Export a report with: a plain zone, a bilateral zone both sides, a spine zone with vertebrae, a detail zone, findings with vet flag, free text, severity, comment, conclusion, next appointment; rasterizeSvg real under happy-dom or stubbed deterministically; re-import; deep-equal. Then extract page text and assert the ZONES, CONSTATS — <ZONE>, (Gauche), (C3, C4), (!) strings. |
Current roundtrip uses observations: [] (audit 13 §4). The PDF step rewrites the pipeline. |
| Attachment fallback | traverseForJsonAttachment on a PDF whose attachment was re-saved by another tool. |
Untested path handling user data. |
| Golden fixtures | tests/fixtures/report-v2.full.json (the report above) + report-v2.minimal.json, frozen. |
Inputs for the v2ToV3 adapter tests. |
| Orphan findings | removing a zone / pruning a region leaves findings; assert current behaviour. | Migration decides to keep or clean (I4). |
| Species registry | getAllSpecies() returns only equine; AnimalForm default 'equine'. |
Replaced by DB species; test is deleted with the registry. |
Target after Step 1: roughly 230–250 cases, all current ones untouched.
3. Tests per future domain (what each step must ship)
| Domain | Unit | DB / RLS | Integration / e2e |
|---|---|---|---|
| Identity | profile form validation, role helpers in TS mirrors | is_platform_*, profile trigger, roles RPC denies self-grant |
sign-up, verify email, login, logout |
| Verification | status machine | expiry trigger, reviewer-only transitions | upload document → pending → admin verifies |
| Organizations | none | membership RPC, secretary cannot become practitioner, seat exclusivity | create cabinet, invite, switch context |
| Clients / Animals | normalisation of contacts, consent payloads | can_view_client/animal, grants expiry/revocation, candidate match reveals nothing cross-tenant |
create client+animal, share with another practitioner via consent flow |
| Anatomical Sheets | document Zod, importer special cases (audit 07 list), renderer attribute emission, GC preconditions in TS mirror | immutability trigger, numbering, can_use_sheet/can_fork_sheet, GC function with each precondition |
seed equine v1, open it in the editor read-only |
| Reports | document v3, v2ToV3, zone interaction service, autosave throttling, outbox |
numbering concurrency, completion + quota (5th ok, 6th denied, deleted counts, import counts), free_edit_until, revisions on checkpoints, restore |
create → autosave → complete → PDF → reopen; offline edit → reconnect → conflict revision |
| golden text, attachment v3, snapshot sufficiency (render with the sheet row deleted) | none | export, download, import into another account denied | |
| Editor merge | one test per legacy operation (audit 03 §2, 05 §3.5), create detail view geometry | publish creates v2, v1 reports still render | annotate, publish, use in a report |
| Community | none | follow grants use, unfollow revokes, private needs grant, fork provenance, moderation roles | discover, follow, fork |
| Booking / Travel / Calendar | expiry rule function (48/24/1h) as pure TS mirror and SQL, availability engine with provider mock | slot lock under concurrent requests, one pending per client | public booking → practitioner confirms → reminder queued |
| Billing | entitlement derivation, referral rules | get_effective_entitlements, Stripe webhook idempotency |
checkout in Stripe test mode |
| Imports | adapters on sample files, duplicate detection | atomic commit, quota preflight | CSV import preview → confirm |
| Admin / Audit | none | append-only, impersonation expiry | admin MFA required |
| Hardening | header checks, dependency audit | full RLS matrix, storage policy matrix | cross-tenant scenarios (architecture Prompt 10) |
4. Fixtures and data
- Real equine SVGs and catalog are the primary fixtures for anatomy tests; they are never edited (CLAUDE.md). Synthetic small SVGs cover error paths.
supabase/seed/test/provides two practitioners, one cabinet, two clients, three animals, the official sheet, one personal sheet, reports in each status. RLS tests reuse it.- Golden PDFs are compared by the text, images and links read back from the
file and by the attachment JSON; the writer's own bytes are pinned by the
R4 digests at a fixed export time (
goldens/r4/pdf.*.sha256), rewritten only for a declared structural change.
5. Gates before a merge, and the CI target
Today (2026-10-13) no CI runs on the forge; the gate is run on the
development machine: pnpm check:all (docs/development/RUST.md §4) before
every merge, plus supabase db reset, supabase test db and the types diff
when supabase/ changes. The workflow files under .gitea/workflows/ are
prepared for a future runner and call the same scripts.
CI target (planned): per PR the stages of pnpm check:all; nightly the
Playwright suite and cargo deny check advisories; at release the full RLS
matrix, the backup/restore rehearsal checklist, pnpm audit, a secret scan.
6. Numbers to track
Test count per step and a "legacy tests deleted" count, so that deletions are deliberate (each deleted test names the replacing test in its commit).