docs(docs): Step 8 files foundation analysis — storage model, worker, quarantine, policies, backups, scope
This commit is contained in:
1 parent
505b83087b
commit
54df4d851a
1 file changed
+578
@@ -0,0 +1,578 @@
|
|||||||
|
# Step 8 — Files foundation and Supabase Storage: technical analysis
|
||||||
|
|
||||||
|
Status: proposal for the owner's decision, 2026-10-12. Nothing implemented.
|
||||||
|
Applies the owner's brief of 2026-10-12 (Step 7 closed; hosting = self-hosted
|
||||||
|
Supabase first, managed later; eight private buckets; quarantine + server
|
||||||
|
validation; avatar as first consumer; service role never in the browser).
|
||||||
|
|
||||||
|
Part I states the facts the design rests on. Part II is the proposed design.
|
||||||
|
Part III answers the ten security questions of the brief. Part IV lists the
|
||||||
|
exact scope, the tests, the risks and the decisions the owner has to take.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part I — Facts
|
||||||
|
|
||||||
|
### I.1 What exists today
|
||||||
|
|
||||||
|
| Area | State |
|
||||||
|
|---|---|
|
||||||
|
| Storage | One bucket, `anatomical-sheet-assets` (migration `20261009100000`): private, SVG only, 20 MiB cap, content-addressed keys `<sha256>.svg`, SELECT policy through the sheet (`private.can_read_sheet_asset_object`), no API write policy, operator upload script with the service key, `confirm_sheet_asset_upload` RPC (service role). pgTAP 006 (28 checks) runs on the PGlite shim and on the real stack. |
|
||||||
|
| Storage API | `storage-api v1.79.36` in the local stack (same image family as a self-hosted deployment). Signed URLs, signed upload URLs and RLS on `storage.objects` are available; image transformation needs `imgproxy`, absent here and a paid add-on on managed. |
|
||||||
|
| Profiles | `avatar_asset_id` was deliberately deferred to this step (comment in `20261008100100_profiles.sql`). |
|
||||||
|
| Shell UI | `UserBadge` (monogram) in the sidebar and the drawer; the top bar holds only the language switch; "Mon compte" = stacked sections, `PersonalInfoForm` first. The validated mockup shows the avatar with "Modifier la photo" inside the Profil section and an avatar + name in the top bar. |
|
||||||
|
| SVG sanitiser | `packages/pdf/src/svg/sanitize.ts`: allowlist of elements and attributes, `href` / `url()` kept in-document, backslash refused, caps 20 000 elements / depth 64 / 256 `<use>` / `stdDeviation` 100, DOM Level 2 on `@xmldom/xmldom`, `SvgRejectedError`. Browser-free. |
|
||||||
|
| Edge runtime | Disabled in `config.toml`. `BACKEND_ARCHITECTURE.md` §6: self-hosted Edge Functions are hand-deployed; first steps lean on SQL. C3: PostgreSQL-backed jobs before a queue. C12: the sanitiser runs between upload and publication, never "on upload". |
|
||||||
|
| Worker | The plan creates `apps/worker` at Step 19 (notifications). Files processing needs server-side code before that. |
|
||||||
|
| Deploy | No backend in production yet. The VPS `.env` is documented as non-sensitive. CSP: `default-src 'self'`, `img-src 'self' data: blob:` (a Supabase origin would be blocked for both API calls and images). |
|
||||||
|
| Tests | pgTAP on PGlite + real stack; app vitest on happy-dom; Step 7 added a headless e2e script against the local stack (scratchpad, Mailpit). |
|
||||||
|
|
||||||
|
### I.2 Corrections to the plan's Step 8 text
|
||||||
|
|
||||||
|
- "RPC `issue_signed_url(asset_id)`" cannot exist in SQL: signing needs the
|
||||||
|
Storage API's secret. The Storage API signs a URL only after evaluating the
|
||||||
|
SELECT policy of `storage.objects` as the caller, so the access check and
|
||||||
|
the signature live in one place already. The design uses it directly.
|
||||||
|
- "Edge Function or SQL-only sanitizer": SQL cannot parse XML or decode
|
||||||
|
images safely; Edge Functions cannot load a native image library and are
|
||||||
|
hand-deployed on self-hosted. A containerised worker is the fit (II.1).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part II — Proposed design
|
||||||
|
|
||||||
|
### II.1 Components
|
||||||
|
|
||||||
|
```text
|
||||||
|
Browser (anon key + user JWT)
|
||||||
|
│ begin_avatar_upload (RPC) ─► assets row pending_upload + quarantine key
|
||||||
|
│ PUT object (Storage API, INSERT policy = that key, that row, that user)
|
||||||
|
│ confirm_asset_upload (RPC) ─► status uploaded
|
||||||
|
│ poll own assets row ─► clean | rejected
|
||||||
|
│ createSignedUrl(object_key, 3600) (Storage API, SELECT policy) ─► <img>
|
||||||
|
▼
|
||||||
|
Supabase (self-hosted compose, later managed)
|
||||||
|
PostgREST ─ public.assets, RPCs, private helpers Storage API ─ 8 private buckets
|
||||||
|
▲
|
||||||
|
│ claim_asset_jobs / complete_asset_job / reject_asset_job (RPC, service role)
|
||||||
|
│ GET quarantine object, PUT clean object, DELETE quarantine (service role)
|
||||||
|
apps/worker (container faunosteo_worker, outbound only, no port)
|
||||||
|
processors: image (sharp) · svg (@faunosteo/pdf allowlist)
|
||||||
|
sweeper: purge objects of deleted/rejected rows, abandoned uploads, stale jobs
|
||||||
|
operator commands: reconcile, migrate
|
||||||
|
```
|
||||||
|
|
||||||
|
Why a worker and not Edge Functions: Edge Functions on the self-hosted stack
|
||||||
|
run in a separate `edge-runtime` container, are deployed by hand, cannot
|
||||||
|
load `sharp` (native), and the config keeps them off. Why not SQL: no image
|
||||||
|
decoding, no XML parsing. The worker is one Node 24 container with the
|
||||||
|
service key in its own env file, no inbound port, the same code against the
|
||||||
|
local stack, the VPS stack and a managed project (URL + key change only).
|
||||||
|
It is the `apps/worker` the plan creates at Step 19; it arrives earlier with
|
||||||
|
one job family, Step 19 adds its own.
|
||||||
|
|
||||||
|
Dependency added: `sharp` (0.35.x, libvips, prebuilt `linuxmusl-x64`
|
||||||
|
package, no install script, Node ≥ 20.9). It decodes JPEG / PNG / WebP,
|
||||||
|
resizes, re-encodes, and drops metadata by default. The only widely
|
||||||
|
maintained option that does all four in Node; pure-JS stacks cover PNG only
|
||||||
|
or are unmaintained.
|
||||||
|
|
||||||
|
### II.2 Buckets
|
||||||
|
|
||||||
|
All eight private (`public = false`). Created by one migration with
|
||||||
|
`on conflict (id) do update` so the migration converges on the local stack,
|
||||||
|
the VPS and managed alike. `file_size_limit` and `allowed_mime_types` on the
|
||||||
|
bucket are a first filter on what the Storage API accepts, never a proof of
|
||||||
|
content.
|
||||||
|
|
||||||
|
| Bucket | Kind(s) | Owning domain and step | Step 8 state |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `avatars` | profile_avatar | Identity (self) | **open**: SELECT + INSERT policies through `assets` |
|
||||||
|
| `logos` | organization_logo | Organizations, Step 10 | closed, no policy |
|
||||||
|
| `professional-documents` | professional_document | Verification, Step 9 | closed |
|
||||||
|
| `client-files` | client_file | Clients, Step 12 | closed |
|
||||||
|
| `animal-files` | animal_file | Animals, Step 12 | closed |
|
||||||
|
| `report-files` | report_file | PDF artifacts, Step 15 | closed |
|
||||||
|
| `anatomical-sheet-assets` | anatomical_sheet_asset | Sheets (existing policy kept) | unchanged |
|
||||||
|
| `imports` | import_file | Imports, Step 24 | closed |
|
||||||
|
|
||||||
|
"Closed" = the bucket row exists, RLS on `storage.objects` has no policy
|
||||||
|
naming it, so `anon` and `authenticated` can neither list, read, upload,
|
||||||
|
update nor delete. Only the service role touches it. A pgTAP census pins
|
||||||
|
the exact policy count per bucket, so a later step cannot open a bucket by
|
||||||
|
accident without changing the test.
|
||||||
|
|
||||||
|
Open point (D2): the brief names the sheet bucket `anatomical-sheets`; the
|
||||||
|
repository already has `anatomical-sheet-assets` with its policy, its
|
||||||
|
operator script, its tests and its seed. Recommendation: keep the existing
|
||||||
|
name and record the deviation; a rename costs a migration, a re-upload and
|
||||||
|
touches Step 13 for no behaviour.
|
||||||
|
|
||||||
|
Proposed caps (closed buckets get the final value at their own step):
|
||||||
|
|
||||||
|
| Bucket | Size | Declared types |
|
||||||
|
|---|---|---|
|
||||||
|
| avatars | 5 MiB | image/jpeg, image/png, image/webp |
|
||||||
|
| logos | 2 MiB | image/jpeg, image/png, image/webp |
|
||||||
|
| professional-documents | 20 MiB | application/pdf, image/jpeg, image/png |
|
||||||
|
| client-files, animal-files | 20 MiB | pdf + the three image types |
|
||||||
|
| report-files | 50 MiB | application/pdf |
|
||||||
|
| anatomical-sheet-assets | 20 MiB (existing) | image/svg+xml |
|
||||||
|
| imports | 50 MiB | application/pdf, application/json, text/csv |
|
||||||
|
|
||||||
|
### II.3 `public.assets` — the authority on every user file
|
||||||
|
|
||||||
|
One row per file, created before the bytes exist, kept after they are gone
|
||||||
|
(status `deleted`) until the sweeper has purged the object.
|
||||||
|
|
||||||
|
```sql
|
||||||
|
create type public.asset_kind as enum (
|
||||||
|
'profile_avatar', 'organization_logo', 'professional_document',
|
||||||
|
'client_file', 'animal_file', 'report_file', 'anatomical_sheet_asset', 'import_file');
|
||||||
|
create type public.asset_status as enum (
|
||||||
|
'pending_upload', 'uploaded', 'processing', 'clean', 'rejected', 'deleted');
|
||||||
|
|
||||||
|
create table public.assets (
|
||||||
|
id uuid primary key default gen_random_uuid(),
|
||||||
|
owner_id uuid not null references public.profiles (id) on delete cascade,
|
||||||
|
kind public.asset_kind not null,
|
||||||
|
bucket_id text not null, -- derived from kind, checked by trigger
|
||||||
|
resource_id uuid, -- profile id for avatars
|
||||||
|
object_key text not null unique, -- final key, server-built
|
||||||
|
quarantine_key text, -- upload key, null once purged
|
||||||
|
status public.asset_status not null default 'pending_upload',
|
||||||
|
declared_mime text not null,
|
||||||
|
declared_size integer not null check (declared_size between 1 and 52428800),
|
||||||
|
original_filename text check (char_length(original_filename) <= 120), -- display only
|
||||||
|
detected_mime text,
|
||||||
|
byte_size integer,
|
||||||
|
sha256 text check (sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
width integer,
|
||||||
|
height integer,
|
||||||
|
rejection_reason text, -- closed codes, see II.6
|
||||||
|
attempts smallint not null default 0,
|
||||||
|
locked_at timestamptz,
|
||||||
|
created_at timestamptz not null default now(),
|
||||||
|
updated_at timestamptz not null default now(),
|
||||||
|
processed_at timestamptz,
|
||||||
|
deleted_at timestamptz,
|
||||||
|
purged_at timestamptz -- object removed by the sweeper
|
||||||
|
);
|
||||||
|
-- one avatar in flight per account
|
||||||
|
create unique index assets_one_avatar_in_flight
|
||||||
|
on public.assets (owner_id) where kind = 'profile_avatar' and status in ('pending_upload','uploaded','processing');
|
||||||
|
|
||||||
|
alter table public.profiles add column avatar_asset_id uuid references public.assets (id);
|
||||||
|
```
|
||||||
|
|
||||||
|
Privileges: `revoke all` from API roles, `grant select` to `authenticated`,
|
||||||
|
RLS `assets_select_own` (`owner_id = auth.uid()`). No INSERT / UPDATE /
|
||||||
|
DELETE policy: every write is an RPC. `profiles.avatar_asset_id` is not in
|
||||||
|
the column grant: only the server sets it, and a trigger checks that it
|
||||||
|
points to a `clean` `profile_avatar` of the same profile.
|
||||||
|
|
||||||
|
RPCs (security definer, `set search_path = ''`, authorisation before any
|
||||||
|
lookup, EXECUTE revoked from PUBLIC, granted per role, audited through
|
||||||
|
`private.log_event` where the state is sensitive):
|
||||||
|
|
||||||
|
| RPC | Role | Effect |
|
||||||
|
|---|---|---|
|
||||||
|
| `begin_avatar_upload(declared_mime, declared_size, original_filename)` | authenticated | caps (type in list, size ≤ 5 MiB), rate (≤ 10 rows created in the last hour), replaces a stale `pending_upload` of the same account; inserts the row with `object_key = profiles/<uid>/<id>/avatar.webp` and `quarantine_key = quarantine/<uid>/<id>/upload`; returns `{asset_id, bucket_id, quarantine_key}` |
|
||||||
|
| `confirm_asset_upload(asset_id)` | authenticated | own row, `pending_upload` → checks the object exists at `quarantine_key` with `metadata->>'size'` ≤ cap → `uploaded` |
|
||||||
|
| `remove_avatar()` | authenticated | clears `profiles.avatar_asset_id`, marks the asset `deleted`; logs `asset.avatar_removed` |
|
||||||
|
| `claim_asset_jobs(limit)` | service_role | `uploaded` → `processing` with `for update skip locked`, `locked_at`, `attempts + 1`; returns the rows |
|
||||||
|
| `complete_asset_job(asset_id, result)` | service_role | `processing` → `clean` with detected fields; for an avatar: sets `profiles.avatar_asset_id`, marks the previous avatar asset `deleted`, in the same transaction; logs `asset.avatar_set` |
|
||||||
|
| `reject_asset_job(asset_id, reason)` | service_role | `processing` → `rejected`; logs `asset.rejected` |
|
||||||
|
| `reclaim_stale_jobs()` | service_role | `processing` older than 10 min → `uploaded` while `attempts < 3`, else `rejected (processing_error)` |
|
||||||
|
| `list_purgeable_assets()` / `mark_asset_purged(asset_id)` | service_role | rows whose object must go (`deleted`, `rejected`, abandoned `pending_upload` > 24 h, `clean` with a quarantine key still present) |
|
||||||
|
|
||||||
|
Why `assets` does not absorb `anatomical_sheet_assets`: that table is
|
||||||
|
content-addressed and sheet-scoped (BDR-032/034) with its own confirmed
|
||||||
|
upload path; merging would rewrite a validated model for no gain. The SVG
|
||||||
|
processor of the worker will serve it at Step 13 through a
|
||||||
|
`sanitization_status` added then.
|
||||||
|
|
||||||
|
### II.4 Object keys
|
||||||
|
|
||||||
|
- Final: `<resource-plural>/<resource-id>/<asset-id>/<canonical-name>`. Avatar:
|
||||||
|
`profiles/<user_id>/<asset_id>/avatar.webp`. The name is fixed per kind;
|
||||||
|
the user's filename never enters a key (it is stored, length-capped and
|
||||||
|
control-character-free, for display only).
|
||||||
|
- Quarantine: `quarantine/<user_id>/<asset_id>/upload` in the same bucket.
|
||||||
|
No API role can read it: the SELECT predicate only matches the
|
||||||
|
`object_key` of `clean` rows, the INSERT predicate only the
|
||||||
|
`quarantine_key` of the caller's `pending_upload` row.
|
||||||
|
- Keys are built by the RPC from uuids; the policies compare whole strings
|
||||||
|
(no prefix match, no `storage.foldername()` so the PGlite shim and the real
|
||||||
|
stack evaluate the same expression).
|
||||||
|
- Sheet assets keep `<sha256>.svg` (shared drawings, immutable versions).
|
||||||
|
Avatars are per-account and mutable: a hash key would need reference
|
||||||
|
counting for nothing.
|
||||||
|
|
||||||
|
### II.5 Upload, processing, display (avatar)
|
||||||
|
|
||||||
|
1. Browser pre-checks (feedback only): size ≤ 5 MiB, magic bytes of
|
||||||
|
JPEG / PNG / WebP, the file decodes (`createImageBitmap`). Preview from
|
||||||
|
`URL.createObjectURL`, shown in the circle with `object-fit: cover`,
|
||||||
|
nothing uploaded yet.
|
||||||
|
2. "Enregistrer" → `begin_avatar_upload` → row + keys.
|
||||||
|
3. `storage.from('avatars').upload(quarantine_key, file, { upsert: false })`
|
||||||
|
with the user JWT. The INSERT policy on `storage.objects` accepts only
|
||||||
|
`(bucket_id = 'avatars', name = quarantine_key)` for a `pending_upload`
|
||||||
|
row owned by `auth.uid()`.
|
||||||
|
4. `confirm_asset_upload(asset_id)` → `uploaded`.
|
||||||
|
5. Worker loop (every 2 s, backoff to 15 s when idle): `claim_asset_jobs(1)`
|
||||||
|
→ download the quarantine object with the service client → processor.
|
||||||
|
6. Image processor: detect the type from bytes (JPEG `FF D8 FF`, PNG
|
||||||
|
8-byte signature, WebP `RIFF….WEBP`); any other → `not_an_image`.
|
||||||
|
`sharp(bytes, { limitInputPixels: 24e6, failOn: 'error', animated: false })`
|
||||||
|
→ header check (`metadata()`: format matches the magic, 64 ≤ sides ≤ 8192)
|
||||||
|
→ `.rotate()` (applies EXIF orientation, then EXIF is dropped)
|
||||||
|
→ `.resize(512, 512, { fit: 'cover', position: 'centre' })`
|
||||||
|
→ `.webp({ quality: 82 })`, no `withMetadata()` → EXIF, XMP, IPTC and ICC
|
||||||
|
are gone. Output: one 512 × 512 WebP, sha256, byte size.
|
||||||
|
7. Worker uploads the clean bytes at `object_key` (service client), deletes
|
||||||
|
the quarantine object, calls `complete_asset_job` (clean + profile
|
||||||
|
pointer + previous avatar marked `deleted`, one transaction). On a
|
||||||
|
processor refusal: `reject_asset_job(reason)` and the quarantine object
|
||||||
|
is deleted.
|
||||||
|
8. Browser polls its own `assets` row (1 s, up to 30 s): `clean` → refresh
|
||||||
|
the profile and the signed URL; `rejected` → message from the closed
|
||||||
|
reason code; still `processing` after 30 s → "toujours en cours" with a
|
||||||
|
retry on next visit.
|
||||||
|
9. Display: `createSignedUrl(object_key, 3600)` with the user JWT. The
|
||||||
|
Storage API evaluates the SELECT policy, then signs. The URL is cached in
|
||||||
|
the auth store with its expiry and renewed five minutes before. Used as
|
||||||
|
`<img src>` in `UserBadge` (sidebar, drawer), the account section and,
|
||||||
|
if the owner approves D3, the top bar.
|
||||||
|
|
||||||
|
Why no client-side resize before upload: the server re-encodes anyway and
|
||||||
|
is the boundary; a second pipeline would duplicate the crop rule and could
|
||||||
|
drift. The 5 MiB cap keeps the upload short.
|
||||||
|
|
||||||
|
### II.6 Rejection codes (closed set, phrased by i18n)
|
||||||
|
|
||||||
|
`not_an_image`, `unsupported_type`, `too_large`, `decode_failed`,
|
||||||
|
`dimensions` (too small or too large), `svg_rejected`, `processing_error`.
|
||||||
|
The raw library message never reaches the row nor the screen; it is logged
|
||||||
|
by the worker with the asset id only.
|
||||||
|
|
||||||
|
### II.7 SVG processor (anatomical sheets)
|
||||||
|
|
||||||
|
Same worker, kind `anatomical_sheet_asset`: parse with `@xmldom/xmldom`
|
||||||
|
(parse errors fatal), `sanitizeSvgRoot` from `@faunosteo/pdf` (allowlist +
|
||||||
|
caps), serialise, 20 MiB cap in and out, refusal → `svg_rejected`. The
|
||||||
|
removed element and attribute names are kept in the result (never values).
|
||||||
|
In Step 8 there is no user producer: uploads into the sheet bucket stay
|
||||||
|
operator-only until the editor step (17) and the database sheets step (13)
|
||||||
|
open a signed upload path with `sanitization_status` on
|
||||||
|
`anatomical_sheet_assets`. The processor is tested on hostile fixtures and
|
||||||
|
through the worker with the service key, so the boundary exists before the
|
||||||
|
producer does (D6).
|
||||||
|
|
||||||
|
### II.8 Worker runtime and operations
|
||||||
|
|
||||||
|
- `apps/worker` (`@faunosteo/worker`): TypeScript, Node 24, deps
|
||||||
|
`@supabase/supabase-js`, `sharp`, `@faunosteo/pdf` (sanitiser),
|
||||||
|
`@xmldom/xmldom`. No DB driver: every database access is an RPC through
|
||||||
|
PostgREST with the service key, so one credential and one transport, and a
|
||||||
|
managed project needs no open database port.
|
||||||
|
- Container `faunosteo_worker` (compose service `worker`): non-root,
|
||||||
|
`read_only: true` + `tmpfs /tmp`, `mem_limit: 512m`, `pids_limit`, no
|
||||||
|
published port, `restart: unless-stopped`, healthcheck = heartbeat file
|
||||||
|
refreshed each loop. Env file `.env.worker` (`SUPABASE_URL`,
|
||||||
|
`SUPABASE_SERVICE_ROLE_KEY`, `WORKER_ID`), `chmod 600`, separate from the
|
||||||
|
app `.env` which DEPLOY.md keeps non-sensitive. Egress limited to the
|
||||||
|
Supabase host (documented firewall rule).
|
||||||
|
- One job at a time, `sharp.concurrency(1)`; time is bounded by the input
|
||||||
|
caps (≤ 5 MiB, ≤ 24 MP), not by a kill switch; a crashed run is reclaimed
|
||||||
|
by `reclaim_stale_jobs` (max 3 attempts).
|
||||||
|
- Sweeper every 10 min: purge objects of `deleted` / `rejected` rows and of
|
||||||
|
abandoned uploads, remove quarantine leftovers of `clean` rows, mark
|
||||||
|
`purged_at`. Objects with no row (orphans from a crash between PUT and
|
||||||
|
RPC) are only reported; `pnpm worker reconcile --apply` deletes them after
|
||||||
|
an operator reads the dry run.
|
||||||
|
- Local dev: `pnpm worker:dev` against the local stack (service key from
|
||||||
|
`supabase status`, typed into the shell as SUPABASE.md §8 already does).
|
||||||
|
- Logging: structured lines, asset id and reason codes only; never bytes,
|
||||||
|
names or user email.
|
||||||
|
|
||||||
|
### II.9 Orphans on replacement and deletion
|
||||||
|
|
||||||
|
| Event | Row state | Object state | Who cleans |
|
||||||
|
|---|---|---|---|
|
||||||
|
| New avatar clean | new `clean`, previous `deleted` (same transaction) | previous object still present | sweeper (purge, idempotent) |
|
||||||
|
| `remove_avatar()` | `deleted`, `profiles.avatar_asset_id = null` | present until purge | sweeper |
|
||||||
|
| Upload never confirmed | `pending_upload` | maybe present | sweeper after 24 h |
|
||||||
|
| Worker dies after PUT clean, before `complete` | `processing` → reclaimed | clean object present | retry: `upsert: true` on the same deterministic key, then `complete` |
|
||||||
|
| Worker dies after `complete`, before quarantine DELETE | `clean`, `quarantine_key` set | quarantine present | sweeper (quarantine leftovers of clean rows) |
|
||||||
|
| Account deleted (Step 26) | rows cascade | objects orphaned | reconcile; Step 26 will mark then purge before dropping rows |
|
||||||
|
|
||||||
|
Access is always cut by the row: a `deleted` or `rejected` row matches no
|
||||||
|
read predicate, whatever the object's fate. The object lifetime is an
|
||||||
|
eventual-consistency concern handled by the sweeper and verified by
|
||||||
|
`reconcile` (lists objects vs rows per bucket, dry run by default).
|
||||||
|
|
||||||
|
### II.10 Backups and restore on the VPS (self-hosted)
|
||||||
|
|
||||||
|
- Storage backend `file` on a named volume `faunosteo_supabase_storage`
|
||||||
|
(kept across renames, BDR-035). One VPS, one disk, no extra service. The
|
||||||
|
S3 backend (Scaleway Object Storage) remains available when the volume
|
||||||
|
outgrows the disk or a second host appears: copy the files, change the
|
||||||
|
backend env, rows unchanged.
|
||||||
|
- Backup order: database first (`pg_dump` of the whole instance, the
|
||||||
|
`storage` schema included), files second (restic snapshot of the volume
|
||||||
|
to Scaleway Object Storage, encrypted, 7 daily / 4 weekly / 6 monthly).
|
||||||
|
Dumping the rows first guarantees every row in the dump has its file in
|
||||||
|
the snapshot; files newer than the dump are extra and show up in
|
||||||
|
`reconcile` as orphans.
|
||||||
|
- Restore: stop the worker, restore the database, restore the volume, start
|
||||||
|
the stack, run `pnpm worker reconcile` (dry run), decide on the report.
|
||||||
|
- Runbook delivered by this step (`docs/development/STORAGE.md`, pointer in
|
||||||
|
DEPLOY.md); the owner runs it. Claude never runs a backup, restore or
|
||||||
|
transfer tool. A restore drill on the local stack is part of the manual
|
||||||
|
validation.
|
||||||
|
|
||||||
|
### II.11 Migration to managed Supabase
|
||||||
|
|
||||||
|
- Schema: the same migrations (`supabase db push`); buckets and policies
|
||||||
|
are SQL and apply on managed. The `storage` schema itself is never dumped
|
||||||
|
into managed: `storage.objects` rows are re-created by the uploads.
|
||||||
|
- Objects: operator command `pnpm worker migrate --from <url> --to <url>`
|
||||||
|
(two service keys from env), driven by `assets` and
|
||||||
|
`anatomical_sheet_assets`: for every `clean` / confirmed row, download,
|
||||||
|
verify sha256, upload at the same key, verify; report. Then `reconcile`
|
||||||
|
on the destination. Everything is verifiable because every object has a
|
||||||
|
row with its hash.
|
||||||
|
- Worker and app: change `SUPABASE_URL` / keys. No code change.
|
||||||
|
- Not used on purpose because managed meters or lacks them: image
|
||||||
|
transformations (imgproxy), Edge Functions, Realtime.
|
||||||
|
|
||||||
|
### II.12 UI (follows the validated mockup, no redesign)
|
||||||
|
|
||||||
|
- Mon compte → Informations personnelles: avatar circle (photo, else the
|
||||||
|
monogram) with "Modifier la photo" under it, as drawn. Picking a file
|
||||||
|
shows the local preview in the circle with "Enregistrer la photo" /
|
||||||
|
"Annuler"; after saving, the circle shows a "Traitement en cours…" state
|
||||||
|
until the row is `clean`; "Supprimer la photo" appears when a photo
|
||||||
|
exists. Errors use `account.avatar.error.<code>`.
|
||||||
|
- `UserBadge` (sidebar + drawer): photo when present, monogram otherwise.
|
||||||
|
- Top bar (D3): the mockup shows avatar + name with a chevron on desktop and
|
||||||
|
the avatar alone on mobile; Step 7 shipped the bar without a user element.
|
||||||
|
Proposal: add the avatar (and the name on desktop) as a link to Mon
|
||||||
|
compte, no dropdown yet (the menu's content is undefined). Decision for
|
||||||
|
the owner.
|
||||||
|
- Dashboard unchanged (the mockup shows no avatar in the content area).
|
||||||
|
- Accepted inputs: JPEG, PNG, WebP; never SVG for avatars.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part III — The ten security questions
|
||||||
|
|
||||||
|
1. **Access control**: the `assets` row is the authority. `storage.objects`
|
||||||
|
policies call `private.can_read_object(uid, bucket, name)` /
|
||||||
|
`private.can_upload_object(uid, bucket, name)` which look the key up in
|
||||||
|
`assets` (status, owner, kind) and dispatch per kind to the owning
|
||||||
|
domain's helper. Step 8 implements the `profile_avatar` branch (owner
|
||||||
|
only); every other kind returns false. Buckets without policy are
|
||||||
|
unreachable by API roles.
|
||||||
|
2. **No cross-user access**: the read predicate requires a `clean` row
|
||||||
|
whose `owner_id = auth.uid()` (avatar branch) and whose `object_key`
|
||||||
|
equals the requested name; keys contain the owner's uuid but the policy
|
||||||
|
never trusts the path, only the row. Listing is denied the same way
|
||||||
|
(SELECT policy). pgTAP: Bob reads nothing of Alice, cannot upload at her
|
||||||
|
key, cannot confirm her row, cannot sign her URL (real-stack e2e).
|
||||||
|
3. **Keys**: server-built from uuids by the RPC
|
||||||
|
(`profiles/<uid>/<asset>/avatar.webp`, `quarantine/<uid>/<asset>/upload`),
|
||||||
|
canonical file name per kind, user filename never in a key, policies
|
||||||
|
compare whole strings.
|
||||||
|
4. **Signed URLs**: issued by the Storage API after the SELECT policy, with
|
||||||
|
the user's JWT; TTL per kind in one table (`profile_avatar: 3600 s`),
|
||||||
|
cached client-side until five minutes before expiry; no `download`
|
||||||
|
flag, no transform, no public URL. A signed URL is a bearer token until
|
||||||
|
expiry: accepted for a 512 px avatar, documented; documents (Step 9) get
|
||||||
|
60 s and `download`.
|
||||||
|
5. **Quarantine**: upload lands under `quarantine/` with the row in
|
||||||
|
`pending_upload` → `uploaded`; no API role can read it; only the worker
|
||||||
|
(service key) reads it, writes the clean bytes under the final key,
|
||||||
|
deletes the quarantine object. Status machine and every transition are
|
||||||
|
RPCs; the browser cannot set `clean`.
|
||||||
|
6. **Dangerous SVG**: the existing allowlist sanitiser (elements,
|
||||||
|
attributes, in-document `href` / `url(#)` only, no backslash, no
|
||||||
|
`<script>` / `<foreignObject>` / `<image>` / event attributes by
|
||||||
|
construction, caps on elements, depth, `<use>`, filters, 20 MiB) runs in
|
||||||
|
the worker; refusal = `svg_rejected`; hostile fixtures (script,
|
||||||
|
foreignObject, `on*`, `javascript:`, external `use`, CSS `url()`, entity
|
||||||
|
expansion, deep nesting, oversized) are unit tests. Avatars never accept
|
||||||
|
SVG.
|
||||||
|
7. **Images without metadata**: the worker never stores the uploaded
|
||||||
|
bytes. It decodes (type from magic bytes, header limits, 24 MP pixel cap),
|
||||||
|
applies the EXIF orientation, resizes and re-encodes to WebP without
|
||||||
|
`withMetadata()`: EXIF, XMP, IPTC and ICC are not copied. A test asserts
|
||||||
|
the output of an EXIF-laden JPEG carries no `EXIF` / `XMP ` chunk and
|
||||||
|
that a rotated portrait comes out upright.
|
||||||
|
8. **Orphans**: replacement and deletion flip the row first (one
|
||||||
|
transaction), the sweeper removes the object later and marks
|
||||||
|
`purged_at`; abandoned uploads expire; `reconcile` lists objects vs rows
|
||||||
|
and only deletes with `--apply` after a dry run (II.9).
|
||||||
|
9. **Backups and restore**: file backend on a named volume, database dump
|
||||||
|
first then restic snapshot of the volume to Scaleway Object Storage,
|
||||||
|
restore in the same order followed by `reconcile`; runbook in
|
||||||
|
`STORAGE.md`, drill on the local stack (II.10).
|
||||||
|
10. **Managed migration**: same migrations, objects copied by an operator
|
||||||
|
command driven by the rows with hash verification, URLs and keys
|
||||||
|
switched by env (II.11).
|
||||||
|
|
||||||
|
Service role: only the worker's env file and the operator commands hold
|
||||||
|
it; `env.js` is generated from `APP_*` variables only (unchanged).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part IV — Scope, tests, risks, decisions
|
||||||
|
|
||||||
|
### IV.1 Exact scope of the implementation
|
||||||
|
|
||||||
|
In:
|
||||||
|
|
||||||
|
- Migration `20261012100000_files_foundation.sql`: enums, `assets`, trigger
|
||||||
|
(`bucket_id` from kind, avatar pointer check), seven bucket rows (the
|
||||||
|
existing one untouched), `private.can_read_object` /
|
||||||
|
`private.can_upload_object` (avatar branch, closed default),
|
||||||
|
`storage.objects` SELECT + INSERT policies for `avatars`, the eight RPCs
|
||||||
|
of II.3, `profiles.avatar_asset_id`, explicit EXECUTE revokes. pgTAP
|
||||||
|
`010_files_foundation.sql` (allowed and denied cases, policy census per
|
||||||
|
bucket), shim extension if a column is missing, `database.types.ts`
|
||||||
|
regenerated.
|
||||||
|
- `apps/worker`: loop, claim / complete / reject, image processor, SVG
|
||||||
|
processor, sweeper, `reconcile` and `migrate` commands, Dockerfile
|
||||||
|
(`node:24-alpine`, non-root), compose service, `.env.worker.example`,
|
||||||
|
vitest on fixtures, an integration test against the local stack run by
|
||||||
|
the CI `supabase` job.
|
||||||
|
- `packages/supabase/src/assets.ts`: `beginAvatarUpload`, `uploadToQuarantine`,
|
||||||
|
`confirmAssetUpload`, `removeAvatar`, `fetchOwnAsset`, `signedUrlFor`
|
||||||
|
with the TTL table; closed `AssetFailure` kinds.
|
||||||
|
- `apps/app`: avatar block in `PersonalInfoForm` (picker, preview, save,
|
||||||
|
processing, delete), `UserBadge` photo, top bar avatar if D3 = yes, auth
|
||||||
|
store avatar URL cache, i18n keys FR/EN, component and store tests.
|
||||||
|
- Docs: `docs/development/STORAGE.md` (model, keys, statuses, worker
|
||||||
|
operations, backup / restore runbook, managed migration), DEPLOY.md
|
||||||
|
(worker service, env file, CSP additions for the Supabase origin),
|
||||||
|
SUPABASE.md (worker in dev), CLAUDE.md, CHANGELOG; plan note on the
|
||||||
|
early worker.
|
||||||
|
- CI: worker typecheck / lint / test / build, `docker build` of the worker
|
||||||
|
image, worker integration test in the `supabase` job.
|
||||||
|
|
||||||
|
Out (named so nothing is implied): user SVG upload path and
|
||||||
|
`sanitization_status` on sheet assets (Steps 13 / 17), logos, documents,
|
||||||
|
client / animal / report / import flows (their steps), image
|
||||||
|
transformations, public profile photo, the production deployment of the
|
||||||
|
self-hosted stack itself (runbook only), account erasure (Step 26),
|
||||||
|
Realtime, a user menu dropdown in the top bar.
|
||||||
|
|
||||||
|
### IV.2 Test scenarios
|
||||||
|
|
||||||
|
pgTAP (PGlite + real stack), every case with its denial twin:
|
||||||
|
|
||||||
|
- buckets: eight rows, all private, caps and types as the table; policy
|
||||||
|
census = 2 on `avatars`, 1 on `anatomical-sheet-assets`, 0 elsewhere;
|
||||||
|
`authenticated` cannot insert nor select in a closed bucket.
|
||||||
|
- `begin_avatar_upload`: happy path returns keys of the right shape; wrong
|
||||||
|
type, oversize, rate exceeded, anon → denied; a second begin replaces the
|
||||||
|
stale pending row (unique index holds).
|
||||||
|
- upload policy: Alice inserts at her quarantine key; Bob at her key →
|
||||||
|
42501; Alice at a final key, at a key of a `clean` row, at an unknown key
|
||||||
|
→ denied; no UPDATE / DELETE policy exists.
|
||||||
|
- confirm: own pending row with the object present → `uploaded`; without
|
||||||
|
the object → error; Bob confirming Alice's row → denied; confirming twice
|
||||||
|
→ error.
|
||||||
|
- read policy: Alice reads her `clean` object only; nothing while
|
||||||
|
`pending_upload` / `uploaded` / `processing` / `rejected` / `deleted`;
|
||||||
|
the quarantine object is never readable; Bob reads nothing; anon nothing.
|
||||||
|
- worker RPCs: `authenticated` cannot execute any of them; claim locks and
|
||||||
|
increments attempts; complete sets the profile pointer and marks the
|
||||||
|
previous avatar `deleted` atomically; reject stores a closed code;
|
||||||
|
reclaim after 10 min, rejection after three attempts; purge list contents.
|
||||||
|
- profile pointer trigger: pointing at a `rejected` row, a foreign row, a
|
||||||
|
non-avatar kind → refused; `authenticated` cannot update the column.
|
||||||
|
- audit: `asset.avatar_set`, `asset.avatar_removed`, `asset.rejected` rows.
|
||||||
|
|
||||||
|
Worker (vitest, fixtures under `apps/worker/tests/fixtures`):
|
||||||
|
|
||||||
|
- image: JPEG / PNG / WebP accepted; PNG bytes with `.jpg` name accepted as
|
||||||
|
PNG; GIF, BMP, SVG, PDF, HTML, polyglot (PNG + trailing script), zip bomb
|
||||||
|
PNG (declares 20 000 × 20 000) → rejected with the right code; truncated
|
||||||
|
file → `decode_failed`; 32 px image → `dimensions`; EXIF-laden JPEG → no
|
||||||
|
EXIF / XMP / ICC in the output, orientation applied; output is 512 × 512
|
||||||
|
WebP, deterministic size bounds.
|
||||||
|
- svg: each hostile fixture rejected or rewritten as the sanitiser rules
|
||||||
|
say; caps enforced; clean draft `horse.svg` passes unchanged.
|
||||||
|
- loop: claim → process → complete call sequence; processor throw →
|
||||||
|
reject; Storage error → row left for reclaim; sweeper purges only listed
|
||||||
|
rows and reports orphans without deleting.
|
||||||
|
|
||||||
|
Integration (local stack, CI `supabase` job): Alice's full flow with the
|
||||||
|
service key for the worker side → row `clean`, object at the final key, no
|
||||||
|
quarantine object, profile pointer set; a second upload replaces and the
|
||||||
|
first object disappears after the sweep; `remove_avatar` then sweep →
|
||||||
|
bucket empty.
|
||||||
|
|
||||||
|
Real-stack e2e (scratchpad script, like Step 7): Alice signs her URL
|
||||||
|
(200), Bob signing Alice's key (400 / 403), anon (400), Alice signing her
|
||||||
|
quarantine key (denied), signed URL fetch works then expires.
|
||||||
|
|
||||||
|
App (vitest + happy-dom): picker rejects a 6 MiB file and a `.jpg` that is
|
||||||
|
a PNG by magic bytes with the phrased error; preview appears before any
|
||||||
|
request; save calls begin → upload → confirm in order; processing state
|
||||||
|
then photo on `clean`; rejection message per code; delete flow; `UserBadge`
|
||||||
|
shows the photo when the store has a URL, the monogram otherwise; i18n
|
||||||
|
parity.
|
||||||
|
|
||||||
|
Manual validation (owner): upload, replace, delete from Mon compte on
|
||||||
|
desktop and mobile; photo visible in the sidebar, the drawer and the top
|
||||||
|
bar; copy the signed URL, open it in a private window (works until expiry,
|
||||||
|
then 400); try the object URL without token (400); upload a renamed PNG and
|
||||||
|
an HTML file (refused with a clear message); restore drill on the local
|
||||||
|
stack.
|
||||||
|
|
||||||
|
### IV.3 Risks
|
||||||
|
|
||||||
|
- **Decoder vulnerabilities** (libvips / libwebp history): the worker is
|
||||||
|
the only process that decodes untrusted images, runs non-root, read-only,
|
||||||
|
memory-capped, with no inbound port and egress limited to the Supabase
|
||||||
|
host; `sharp` updates become a routine chore. Residual risk accepted and
|
||||||
|
documented.
|
||||||
|
- **Worker down**: uploads stay `uploaded`; the UI says "toujours en
|
||||||
|
cours"; nothing is lost; the healthcheck and `restart: unless-stopped`
|
||||||
|
cover the common case; a monitoring hook is Step 27.
|
||||||
|
- **Signed URL as bearer**: a leaked avatar URL exposes a 512 px picture
|
||||||
|
for at most one hour. Documents will use short TTLs.
|
||||||
|
- **CSP in production**: `connect-src` and `img-src` must include the
|
||||||
|
Supabase origin, or neither the API nor the avatars load. Documented in
|
||||||
|
DEPLOY.md; the owner applies it at deployment.
|
||||||
|
- **PGlite shim vs real `storage` schema**: policies use only `bucket_id`
|
||||||
|
and `name`, no storage helper function, so both engines evaluate the same
|
||||||
|
expression; the real stack's `protect_delete` trigger forbids direct
|
||||||
|
deletes in tests (LRN-041), writes are asserted by census.
|
||||||
|
- **Storage API version drift** between the CLI image, the self-hosted
|
||||||
|
compose and managed: signing and RLS semantics are stable since 2023; the
|
||||||
|
integration test on the CI stack catches a regression.
|
||||||
|
- **Service key handling on the VPS**: a second env file with `chmod 600`;
|
||||||
|
the app `.env` stays key-free; `env.js` never carries it (test).
|
||||||
|
- **Orientation and colour**: EXIF orientation is applied before the
|
||||||
|
rotation data is dropped; wide-gamut ICC profiles are dropped, a slight
|
||||||
|
colour shift on rare photos is accepted for an avatar.
|
||||||
|
- **Plan deviation**: the worker arrives at Step 8 instead of Step 19;
|
||||||
|
Step 19 extends it rather than creating it. Recorded in the plan.
|
||||||
|
|
||||||
|
### IV.4 Decisions for the owner
|
||||||
|
|
||||||
|
| Id | Question | Recommendation |
|
||||||
|
|---|---|---|
|
||||||
|
| D1 | Server-side processing runtime | Containerised worker `faunosteo_worker` (II.1), no Edge Functions |
|
||||||
|
| D2 | Sheet bucket name | Keep `anatomical-sheet-assets`, record the deviation from the announced `anatomical-sheets` |
|
||||||
|
| D3 | Top bar avatar | Add avatar (+ name on desktop) as a link to Mon compte, no dropdown; or sidebar / drawer / account only |
|
||||||
|
| D4 | Avatar output | One 512 × 512 WebP, centre square crop, no metadata; inputs JPEG / PNG / WebP ≤ 5 MiB |
|
||||||
|
| D5 | Self-hosted Storage backend and backups | File backend on a named volume + restic to Scaleway Object Storage, database dumped first |
|
||||||
|
| D6 | SVG processor in Step 8 | Yes, in the worker with hostile fixtures, producer at Steps 13 / 17 |
|
||||||
|
| D7 | Signed URL TTL for avatars | 3600 s, cached client-side |
|
||||||
|
| D8 | Polling vs Realtime for the processing state | Polling the own row (1 s, 30 s cap); Realtime stays off |
|
||||||
Reference in new issue
Block a user