diff --git a/docs/development/STEP8_FILES_ANALYSIS.md b/docs/development/STEP8_FILES_ANALYSIS.md new file mode 100644 index 00000000..1801d58d --- /dev/null +++ b/docs/development/STEP8_FILES_ANALYSIS.md @@ -0,0 +1,578 @@ +# Step 8 — Files foundation and Supabase Storage: technical analysis + +Status: proposal for the owner's decision, 2026-10-12. Nothing implemented. +Applies the owner's brief of 2026-10-12 (Step 7 closed; hosting = self-hosted +Supabase first, managed later; eight private buckets; quarantine + server +validation; avatar as first consumer; service role never in the browser). + +Part I states the facts the design rests on. Part II is the proposed design. +Part III answers the ten security questions of the brief. Part IV lists the +exact scope, the tests, the risks and the decisions the owner has to take. + +--- + +## Part I — Facts + +### I.1 What exists today + +| Area | State | +|---|---| +| Storage | One bucket, `anatomical-sheet-assets` (migration `20261009100000`): private, SVG only, 20 MiB cap, content-addressed keys `.svg`, SELECT policy through the sheet (`private.can_read_sheet_asset_object`), no API write policy, operator upload script with the service key, `confirm_sheet_asset_upload` RPC (service role). pgTAP 006 (28 checks) runs on the PGlite shim and on the real stack. | +| Storage API | `storage-api v1.79.36` in the local stack (same image family as a self-hosted deployment). Signed URLs, signed upload URLs and RLS on `storage.objects` are available; image transformation needs `imgproxy`, absent here and a paid add-on on managed. | +| Profiles | `avatar_asset_id` was deliberately deferred to this step (comment in `20261008100100_profiles.sql`). | +| Shell UI | `UserBadge` (monogram) in the sidebar and the drawer; the top bar holds only the language switch; "Mon compte" = stacked sections, `PersonalInfoForm` first. The validated mockup shows the avatar with "Modifier la photo" inside the Profil section and an avatar + name in the top bar. | +| SVG sanitiser | `packages/pdf/src/svg/sanitize.ts`: allowlist of elements and attributes, `href` / `url()` kept in-document, backslash refused, caps 20 000 elements / depth 64 / 256 `` / `stdDeviation` 100, DOM Level 2 on `@xmldom/xmldom`, `SvgRejectedError`. Browser-free. | +| Edge runtime | Disabled in `config.toml`. `BACKEND_ARCHITECTURE.md` §6: self-hosted Edge Functions are hand-deployed; first steps lean on SQL. C3: PostgreSQL-backed jobs before a queue. C12: the sanitiser runs between upload and publication, never "on upload". | +| Worker | The plan creates `apps/worker` at Step 19 (notifications). Files processing needs server-side code before that. | +| Deploy | No backend in production yet. The VPS `.env` is documented as non-sensitive. CSP: `default-src 'self'`, `img-src 'self' data: blob:` (a Supabase origin would be blocked for both API calls and images). | +| Tests | pgTAP on PGlite + real stack; app vitest on happy-dom; Step 7 added a headless e2e script against the local stack (scratchpad, Mailpit). | + +### I.2 Corrections to the plan's Step 8 text + +- "RPC `issue_signed_url(asset_id)`" cannot exist in SQL: signing needs the + Storage API's secret. The Storage API signs a URL only after evaluating the + SELECT policy of `storage.objects` as the caller, so the access check and + the signature live in one place already. The design uses it directly. +- "Edge Function or SQL-only sanitizer": SQL cannot parse XML or decode + images safely; Edge Functions cannot load a native image library and are + hand-deployed on self-hosted. A containerised worker is the fit (II.1). + +--- + +## Part II — Proposed design + +### II.1 Components + +```text +Browser (anon key + user JWT) + │ begin_avatar_upload (RPC) ─► assets row pending_upload + quarantine key + │ PUT object (Storage API, INSERT policy = that key, that row, that user) + │ confirm_asset_upload (RPC) ─► status uploaded + │ poll own assets row ─► clean | rejected + │ createSignedUrl(object_key, 3600) (Storage API, SELECT policy) ─► + ▼ +Supabase (self-hosted compose, later managed) + PostgREST ─ public.assets, RPCs, private helpers Storage API ─ 8 private buckets + ▲ + │ claim_asset_jobs / complete_asset_job / reject_asset_job (RPC, service role) + │ GET quarantine object, PUT clean object, DELETE quarantine (service role) +apps/worker (container faunosteo_worker, outbound only, no port) + processors: image (sharp) · svg (@faunosteo/pdf allowlist) + sweeper: purge objects of deleted/rejected rows, abandoned uploads, stale jobs + operator commands: reconcile, migrate +``` + +Why a worker and not Edge Functions: Edge Functions on the self-hosted stack +run in a separate `edge-runtime` container, are deployed by hand, cannot +load `sharp` (native), and the config keeps them off. Why not SQL: no image +decoding, no XML parsing. The worker is one Node 24 container with the +service key in its own env file, no inbound port, the same code against the +local stack, the VPS stack and a managed project (URL + key change only). +It is the `apps/worker` the plan creates at Step 19; it arrives earlier with +one job family, Step 19 adds its own. + +Dependency added: `sharp` (0.35.x, libvips, prebuilt `linuxmusl-x64` +package, no install script, Node ≥ 20.9). It decodes JPEG / PNG / WebP, +resizes, re-encodes, and drops metadata by default. The only widely +maintained option that does all four in Node; pure-JS stacks cover PNG only +or are unmaintained. + +### II.2 Buckets + +All eight private (`public = false`). Created by one migration with +`on conflict (id) do update` so the migration converges on the local stack, +the VPS and managed alike. `file_size_limit` and `allowed_mime_types` on the +bucket are a first filter on what the Storage API accepts, never a proof of +content. + +| Bucket | Kind(s) | Owning domain and step | Step 8 state | +|---|---|---|---| +| `avatars` | profile_avatar | Identity (self) | **open**: SELECT + INSERT policies through `assets` | +| `logos` | organization_logo | Organizations, Step 10 | closed, no policy | +| `professional-documents` | professional_document | Verification, Step 9 | closed | +| `client-files` | client_file | Clients, Step 12 | closed | +| `animal-files` | animal_file | Animals, Step 12 | closed | +| `report-files` | report_file | PDF artifacts, Step 15 | closed | +| `anatomical-sheet-assets` | anatomical_sheet_asset | Sheets (existing policy kept) | unchanged | +| `imports` | import_file | Imports, Step 24 | closed | + +"Closed" = the bucket row exists, RLS on `storage.objects` has no policy +naming it, so `anon` and `authenticated` can neither list, read, upload, +update nor delete. Only the service role touches it. A pgTAP census pins +the exact policy count per bucket, so a later step cannot open a bucket by +accident without changing the test. + +Open point (D2): the brief names the sheet bucket `anatomical-sheets`; the +repository already has `anatomical-sheet-assets` with its policy, its +operator script, its tests and its seed. Recommendation: keep the existing +name and record the deviation; a rename costs a migration, a re-upload and +touches Step 13 for no behaviour. + +Proposed caps (closed buckets get the final value at their own step): + +| Bucket | Size | Declared types | +|---|---|---| +| avatars | 5 MiB | image/jpeg, image/png, image/webp | +| logos | 2 MiB | image/jpeg, image/png, image/webp | +| professional-documents | 20 MiB | application/pdf, image/jpeg, image/png | +| client-files, animal-files | 20 MiB | pdf + the three image types | +| report-files | 50 MiB | application/pdf | +| anatomical-sheet-assets | 20 MiB (existing) | image/svg+xml | +| imports | 50 MiB | application/pdf, application/json, text/csv | + +### II.3 `public.assets` — the authority on every user file + +One row per file, created before the bytes exist, kept after they are gone +(status `deleted`) until the sweeper has purged the object. + +```sql +create type public.asset_kind as enum ( + 'profile_avatar', 'organization_logo', 'professional_document', + 'client_file', 'animal_file', 'report_file', 'anatomical_sheet_asset', 'import_file'); +create type public.asset_status as enum ( + 'pending_upload', 'uploaded', 'processing', 'clean', 'rejected', 'deleted'); + +create table public.assets ( + id uuid primary key default gen_random_uuid(), + owner_id uuid not null references public.profiles (id) on delete cascade, + kind public.asset_kind not null, + bucket_id text not null, -- derived from kind, checked by trigger + resource_id uuid, -- profile id for avatars + object_key text not null unique, -- final key, server-built + quarantine_key text, -- upload key, null once purged + status public.asset_status not null default 'pending_upload', + declared_mime text not null, + declared_size integer not null check (declared_size between 1 and 52428800), + original_filename text check (char_length(original_filename) <= 120), -- display only + detected_mime text, + byte_size integer, + sha256 text check (sha256 ~ '^[0-9a-f]{64}$'), + width integer, + height integer, + rejection_reason text, -- closed codes, see II.6 + attempts smallint not null default 0, + locked_at timestamptz, + created_at timestamptz not null default now(), + updated_at timestamptz not null default now(), + processed_at timestamptz, + deleted_at timestamptz, + purged_at timestamptz -- object removed by the sweeper +); +-- one avatar in flight per account +create unique index assets_one_avatar_in_flight + on public.assets (owner_id) where kind = 'profile_avatar' and status in ('pending_upload','uploaded','processing'); + +alter table public.profiles add column avatar_asset_id uuid references public.assets (id); +``` + +Privileges: `revoke all` from API roles, `grant select` to `authenticated`, +RLS `assets_select_own` (`owner_id = auth.uid()`). No INSERT / UPDATE / +DELETE policy: every write is an RPC. `profiles.avatar_asset_id` is not in +the column grant: only the server sets it, and a trigger checks that it +points to a `clean` `profile_avatar` of the same profile. + +RPCs (security definer, `set search_path = ''`, authorisation before any +lookup, EXECUTE revoked from PUBLIC, granted per role, audited through +`private.log_event` where the state is sensitive): + +| RPC | Role | Effect | +|---|---|---| +| `begin_avatar_upload(declared_mime, declared_size, original_filename)` | authenticated | caps (type in list, size ≤ 5 MiB), rate (≤ 10 rows created in the last hour), replaces a stale `pending_upload` of the same account; inserts the row with `object_key = profiles///avatar.webp` and `quarantine_key = quarantine///upload`; returns `{asset_id, bucket_id, quarantine_key}` | +| `confirm_asset_upload(asset_id)` | authenticated | own row, `pending_upload` → checks the object exists at `quarantine_key` with `metadata->>'size'` ≤ cap → `uploaded` | +| `remove_avatar()` | authenticated | clears `profiles.avatar_asset_id`, marks the asset `deleted`; logs `asset.avatar_removed` | +| `claim_asset_jobs(limit)` | service_role | `uploaded` → `processing` with `for update skip locked`, `locked_at`, `attempts + 1`; returns the rows | +| `complete_asset_job(asset_id, result)` | service_role | `processing` → `clean` with detected fields; for an avatar: sets `profiles.avatar_asset_id`, marks the previous avatar asset `deleted`, in the same transaction; logs `asset.avatar_set` | +| `reject_asset_job(asset_id, reason)` | service_role | `processing` → `rejected`; logs `asset.rejected` | +| `reclaim_stale_jobs()` | service_role | `processing` older than 10 min → `uploaded` while `attempts < 3`, else `rejected (processing_error)` | +| `list_purgeable_assets()` / `mark_asset_purged(asset_id)` | service_role | rows whose object must go (`deleted`, `rejected`, abandoned `pending_upload` > 24 h, `clean` with a quarantine key still present) | + +Why `assets` does not absorb `anatomical_sheet_assets`: that table is +content-addressed and sheet-scoped (BDR-032/034) with its own confirmed +upload path; merging would rewrite a validated model for no gain. The SVG +processor of the worker will serve it at Step 13 through a +`sanitization_status` added then. + +### II.4 Object keys + +- Final: `///`. Avatar: + `profiles///avatar.webp`. The name is fixed per kind; + the user's filename never enters a key (it is stored, length-capped and + control-character-free, for display only). +- Quarantine: `quarantine///upload` in the same bucket. + No API role can read it: the SELECT predicate only matches the + `object_key` of `clean` rows, the INSERT predicate only the + `quarantine_key` of the caller's `pending_upload` row. +- Keys are built by the RPC from uuids; the policies compare whole strings + (no prefix match, no `storage.foldername()` so the PGlite shim and the real + stack evaluate the same expression). +- Sheet assets keep `.svg` (shared drawings, immutable versions). + Avatars are per-account and mutable: a hash key would need reference + counting for nothing. + +### II.5 Upload, processing, display (avatar) + +1. Browser pre-checks (feedback only): size ≤ 5 MiB, magic bytes of + JPEG / PNG / WebP, the file decodes (`createImageBitmap`). Preview from + `URL.createObjectURL`, shown in the circle with `object-fit: cover`, + nothing uploaded yet. +2. "Enregistrer" → `begin_avatar_upload` → row + keys. +3. `storage.from('avatars').upload(quarantine_key, file, { upsert: false })` + with the user JWT. The INSERT policy on `storage.objects` accepts only + `(bucket_id = 'avatars', name = quarantine_key)` for a `pending_upload` + row owned by `auth.uid()`. +4. `confirm_asset_upload(asset_id)` → `uploaded`. +5. Worker loop (every 2 s, backoff to 15 s when idle): `claim_asset_jobs(1)` + → download the quarantine object with the service client → processor. +6. Image processor: detect the type from bytes (JPEG `FF D8 FF`, PNG + 8-byte signature, WebP `RIFF….WEBP`); any other → `not_an_image`. + `sharp(bytes, { limitInputPixels: 24e6, failOn: 'error', animated: false })` + → header check (`metadata()`: format matches the magic, 64 ≤ sides ≤ 8192) + → `.rotate()` (applies EXIF orientation, then EXIF is dropped) + → `.resize(512, 512, { fit: 'cover', position: 'centre' })` + → `.webp({ quality: 82 })`, no `withMetadata()` → EXIF, XMP, IPTC and ICC + are gone. Output: one 512 × 512 WebP, sha256, byte size. +7. Worker uploads the clean bytes at `object_key` (service client), deletes + the quarantine object, calls `complete_asset_job` (clean + profile + pointer + previous avatar marked `deleted`, one transaction). On a + processor refusal: `reject_asset_job(reason)` and the quarantine object + is deleted. +8. Browser polls its own `assets` row (1 s, up to 30 s): `clean` → refresh + the profile and the signed URL; `rejected` → message from the closed + reason code; still `processing` after 30 s → "toujours en cours" with a + retry on next visit. +9. Display: `createSignedUrl(object_key, 3600)` with the user JWT. The + Storage API evaluates the SELECT policy, then signs. The URL is cached in + the auth store with its expiry and renewed five minutes before. Used as + `` in `UserBadge` (sidebar, drawer), the account section and, + if the owner approves D3, the top bar. + +Why no client-side resize before upload: the server re-encodes anyway and +is the boundary; a second pipeline would duplicate the crop rule and could +drift. The 5 MiB cap keeps the upload short. + +### II.6 Rejection codes (closed set, phrased by i18n) + +`not_an_image`, `unsupported_type`, `too_large`, `decode_failed`, +`dimensions` (too small or too large), `svg_rejected`, `processing_error`. +The raw library message never reaches the row nor the screen; it is logged +by the worker with the asset id only. + +### II.7 SVG processor (anatomical sheets) + +Same worker, kind `anatomical_sheet_asset`: parse with `@xmldom/xmldom` +(parse errors fatal), `sanitizeSvgRoot` from `@faunosteo/pdf` (allowlist + +caps), serialise, 20 MiB cap in and out, refusal → `svg_rejected`. The +removed element and attribute names are kept in the result (never values). +In Step 8 there is no user producer: uploads into the sheet bucket stay +operator-only until the editor step (17) and the database sheets step (13) +open a signed upload path with `sanitization_status` on +`anatomical_sheet_assets`. The processor is tested on hostile fixtures and +through the worker with the service key, so the boundary exists before the +producer does (D6). + +### II.8 Worker runtime and operations + +- `apps/worker` (`@faunosteo/worker`): TypeScript, Node 24, deps + `@supabase/supabase-js`, `sharp`, `@faunosteo/pdf` (sanitiser), + `@xmldom/xmldom`. No DB driver: every database access is an RPC through + PostgREST with the service key, so one credential and one transport, and a + managed project needs no open database port. +- Container `faunosteo_worker` (compose service `worker`): non-root, + `read_only: true` + `tmpfs /tmp`, `mem_limit: 512m`, `pids_limit`, no + published port, `restart: unless-stopped`, healthcheck = heartbeat file + refreshed each loop. Env file `.env.worker` (`SUPABASE_URL`, + `SUPABASE_SERVICE_ROLE_KEY`, `WORKER_ID`), `chmod 600`, separate from the + app `.env` which DEPLOY.md keeps non-sensitive. Egress limited to the + Supabase host (documented firewall rule). +- One job at a time, `sharp.concurrency(1)`; time is bounded by the input + caps (≤ 5 MiB, ≤ 24 MP), not by a kill switch; a crashed run is reclaimed + by `reclaim_stale_jobs` (max 3 attempts). +- Sweeper every 10 min: purge objects of `deleted` / `rejected` rows and of + abandoned uploads, remove quarantine leftovers of `clean` rows, mark + `purged_at`. Objects with no row (orphans from a crash between PUT and + RPC) are only reported; `pnpm worker reconcile --apply` deletes them after + an operator reads the dry run. +- Local dev: `pnpm worker:dev` against the local stack (service key from + `supabase status`, typed into the shell as SUPABASE.md §8 already does). +- Logging: structured lines, asset id and reason codes only; never bytes, + names or user email. + +### II.9 Orphans on replacement and deletion + +| Event | Row state | Object state | Who cleans | +|---|---|---|---| +| New avatar clean | new `clean`, previous `deleted` (same transaction) | previous object still present | sweeper (purge, idempotent) | +| `remove_avatar()` | `deleted`, `profiles.avatar_asset_id = null` | present until purge | sweeper | +| Upload never confirmed | `pending_upload` | maybe present | sweeper after 24 h | +| Worker dies after PUT clean, before `complete` | `processing` → reclaimed | clean object present | retry: `upsert: true` on the same deterministic key, then `complete` | +| Worker dies after `complete`, before quarantine DELETE | `clean`, `quarantine_key` set | quarantine present | sweeper (quarantine leftovers of clean rows) | +| Account deleted (Step 26) | rows cascade | objects orphaned | reconcile; Step 26 will mark then purge before dropping rows | + +Access is always cut by the row: a `deleted` or `rejected` row matches no +read predicate, whatever the object's fate. The object lifetime is an +eventual-consistency concern handled by the sweeper and verified by +`reconcile` (lists objects vs rows per bucket, dry run by default). + +### II.10 Backups and restore on the VPS (self-hosted) + +- Storage backend `file` on a named volume `faunosteo_supabase_storage` + (kept across renames, BDR-035). One VPS, one disk, no extra service. The + S3 backend (Scaleway Object Storage) remains available when the volume + outgrows the disk or a second host appears: copy the files, change the + backend env, rows unchanged. +- Backup order: database first (`pg_dump` of the whole instance, the + `storage` schema included), files second (restic snapshot of the volume + to Scaleway Object Storage, encrypted, 7 daily / 4 weekly / 6 monthly). + Dumping the rows first guarantees every row in the dump has its file in + the snapshot; files newer than the dump are extra and show up in + `reconcile` as orphans. +- Restore: stop the worker, restore the database, restore the volume, start + the stack, run `pnpm worker reconcile` (dry run), decide on the report. +- Runbook delivered by this step (`docs/development/STORAGE.md`, pointer in + DEPLOY.md); the owner runs it. Claude never runs a backup, restore or + transfer tool. A restore drill on the local stack is part of the manual + validation. + +### II.11 Migration to managed Supabase + +- Schema: the same migrations (`supabase db push`); buckets and policies + are SQL and apply on managed. The `storage` schema itself is never dumped + into managed: `storage.objects` rows are re-created by the uploads. +- Objects: operator command `pnpm worker migrate --from --to ` + (two service keys from env), driven by `assets` and + `anatomical_sheet_assets`: for every `clean` / confirmed row, download, + verify sha256, upload at the same key, verify; report. Then `reconcile` + on the destination. Everything is verifiable because every object has a + row with its hash. +- Worker and app: change `SUPABASE_URL` / keys. No code change. +- Not used on purpose because managed meters or lacks them: image + transformations (imgproxy), Edge Functions, Realtime. + +### II.12 UI (follows the validated mockup, no redesign) + +- Mon compte → Informations personnelles: avatar circle (photo, else the + monogram) with "Modifier la photo" under it, as drawn. Picking a file + shows the local preview in the circle with "Enregistrer la photo" / + "Annuler"; after saving, the circle shows a "Traitement en cours…" state + until the row is `clean`; "Supprimer la photo" appears when a photo + exists. Errors use `account.avatar.error.`. +- `UserBadge` (sidebar + drawer): photo when present, monogram otherwise. +- Top bar (D3): the mockup shows avatar + name with a chevron on desktop and + the avatar alone on mobile; Step 7 shipped the bar without a user element. + Proposal: add the avatar (and the name on desktop) as a link to Mon + compte, no dropdown yet (the menu's content is undefined). Decision for + the owner. +- Dashboard unchanged (the mockup shows no avatar in the content area). +- Accepted inputs: JPEG, PNG, WebP; never SVG for avatars. + +--- + +## Part III — The ten security questions + +1. **Access control**: the `assets` row is the authority. `storage.objects` + policies call `private.can_read_object(uid, bucket, name)` / + `private.can_upload_object(uid, bucket, name)` which look the key up in + `assets` (status, owner, kind) and dispatch per kind to the owning + domain's helper. Step 8 implements the `profile_avatar` branch (owner + only); every other kind returns false. Buckets without policy are + unreachable by API roles. +2. **No cross-user access**: the read predicate requires a `clean` row + whose `owner_id = auth.uid()` (avatar branch) and whose `object_key` + equals the requested name; keys contain the owner's uuid but the policy + never trusts the path, only the row. Listing is denied the same way + (SELECT policy). pgTAP: Bob reads nothing of Alice, cannot upload at her + key, cannot confirm her row, cannot sign her URL (real-stack e2e). +3. **Keys**: server-built from uuids by the RPC + (`profiles///avatar.webp`, `quarantine///upload`), + canonical file name per kind, user filename never in a key, policies + compare whole strings. +4. **Signed URLs**: issued by the Storage API after the SELECT policy, with + the user's JWT; TTL per kind in one table (`profile_avatar: 3600 s`), + cached client-side until five minutes before expiry; no `download` + flag, no transform, no public URL. A signed URL is a bearer token until + expiry: accepted for a 512 px avatar, documented; documents (Step 9) get + 60 s and `download`. +5. **Quarantine**: upload lands under `quarantine/` with the row in + `pending_upload` → `uploaded`; no API role can read it; only the worker + (service key) reads it, writes the clean bytes under the final key, + deletes the quarantine object. Status machine and every transition are + RPCs; the browser cannot set `clean`. +6. **Dangerous SVG**: the existing allowlist sanitiser (elements, + attributes, in-document `href` / `url(#)` only, no backslash, no + `