628 lines
40 KiB
Markdown
628 lines
40 KiB
Markdown
# Step 8 — Files foundation and Supabase Storage: technical analysis
|
||
|
||
Status: proposal for the owner's decision, 2026-10-12, revision 2 after three
|
||
independent plan challenges (correctness, robustness, simplicity: 47
|
||
findings, folded in; the changes are listed in I.3). Nothing implemented.
|
||
Applies the owner's brief of 2026-10-12 (Step 7 closed; hosting =
|
||
self-hosted Supabase first, managed later; eight private buckets closed by
|
||
default; quarantine + server validation; avatar as first consumer; service
|
||
role never in the browser).
|
||
|
||
Part I states the facts the design rests on. Part II is the proposed design.
|
||
Part III answers the ten security questions of the brief. Part IV lists the
|
||
exact scope, the tests, the risks and the decisions the owner has to take.
|
||
|
||
---
|
||
|
||
## Part I — Facts
|
||
|
||
### I.1 What exists today
|
||
|
||
| Area | State |
|
||
|---|---|
|
||
| Storage | One bucket, `anatomical-sheet-assets` (migration `20261009100000`): private, SVG only, 20 MiB cap, content-addressed keys `<sha256>.svg`, SELECT policy through the sheet (`private.can_read_sheet_asset_object`), no API write policy, operator upload script with the service key, `confirm_sheet_asset_upload` RPC (service role). pgTAP 006 (28 checks) runs on the PGlite shim and on the real stack; pgTAP 001 pins the exact list of `private` functions `authenticated` may execute. |
|
||
| Storage API | `storage-api v1.79.36` locally (same image family as self-hosted). Verified on the client library (storage-js 2.117): an upload with the user JWT and `upsert: false` needs only an INSERT policy on `storage.objects`, evaluated on `bucket_id` and `name`; the bucket MIME check runs before RLS on the multipart part type (the browser's `File.type`, not sniffed); `createSignedUrl` evaluates the SELECT policy as the caller and answers 400 when denied; a signed upload URL is checked at signing time only and stays valid two hours. Image transformation needs `imgproxy`, absent here and a paid add-on on managed. |
|
||
| Profiles | `avatar_asset_id` was deferred to this step (comment in `20261008100100_profiles.sql`). |
|
||
| Shell UI | `UserBadge` (monogram) in the sidebar and the drawer; the top bar holds only the language switch; "Mon compte" = stacked sections, `PersonalInfoForm` first. The validated mockup shows the avatar with "Modifier la photo" inside the Profil section and an avatar + name in the top bar. Production CSP: `default-src 'self'`, `img-src 'self' data: blob:`. |
|
||
| SVG sanitiser | `packages/pdf/src/svg/{parse,sanitize,view}.ts`: `parseSvg` refuses DOCTYPE / ENTITY, `sanitizeSvgRoot` = allowlist of elements and attributes, in-document `href` / `url(#)` only, backslash refused, caps 20 000 elements / depth 64 / 256 `<use>` / `stdDeviation` 100; `cleanSvgRoot` chains both. Hostile fixtures already covered by `packages/pdf/tests/security.test.ts` (script, `on*`, `javascript:`, external image, CSS escapes, ENTITY, 257 `<use>`, `horse.svg` unchanged). Not exported from the package index; serialisation drops the XML declaration (byte change). |
|
||
| Edge runtime | Disabled in `config.toml`. `BACKEND_ARCHITECTURE.md` §6; C3: PostgreSQL-backed jobs before a queue; C12: the sanitiser runs between upload and publication. |
|
||
| Worker | The plan creates `apps/worker` at Step 19 (notifications). Files processing needs server-side code before that. |
|
||
| Deploy | No backend in production yet. The VPS `.env` is documented as non-sensitive. The VPS is shared by several SaaS (BDR-035). |
|
||
| sharp 0.35.5 | No install script, Node ≥ 20.9, prebuilt `linuxmusl-x64`. `rotate()` with no angle auto-orients from EXIF; output strips all metadata by default and converts through the embedded ICC profile to sRGB; `failOn` defaults to `warning` (stricter than `error`); `limitInputPixels` is enforced before `metadata()` returns. |
|
||
| Tests | pgTAP on PGlite + real stack; app vitest on happy-dom; Step 7 added a headless e2e script against the local stack. PostgREST `max_rows = 1000` caps set-returning RPCs. |
|
||
|
||
### I.2 Corrections to the plan's Step 8 text
|
||
|
||
- "RPC `issue_signed_url(asset_id)`" cannot exist in SQL: signing needs the
|
||
Storage API's secret. The Storage API signs only after the SELECT policy
|
||
passes, so the check and the signature already live in one place. For an
|
||
owner-only avatar, Step 8 does not sign at all (II.6).
|
||
- "Edge Function or SQL-only sanitizer": SQL cannot parse XML or decode
|
||
images; a containerised worker is the fit (II.1).
|
||
- "Upload a hostile SVG as a sheet asset and verify it is rewritten": no user
|
||
can upload a sheet asset before Step 13, so this manual check moves there
|
||
with the processor (D6).
|
||
|
||
### I.3 Changes after the challenges (revision 2)
|
||
|
||
Decoder isolated from the service key and bounded by a hard deadline;
|
||
write order PUT clean → commit → delete quarantine; `uploaded` rows
|
||
superseded by a new upload instead of a blocking unique index; purge queue
|
||
with a grace period longer than a backup run (the backup invariant was
|
||
false otherwise); every quarantine leftover auto-purged (signed upload URLs
|
||
can land bytes after the row moved on); client-side downscale as transport
|
||
(current phones exceed a 24 MP cap); owner-only avatar read by `download`
|
||
under RLS, no signed URL, no bearer token; schema narrowed to the one kind
|
||
that has rules (no multi-kind dispatcher, no derived-bucket trigger, no
|
||
unread columns); avatar pointer derived by a unique index instead of a
|
||
profile column + trigger; SVG processor, `migrate` and `reconcile`
|
||
commands, backup tooling and restore drill moved out of Step 8 (runbook
|
||
text stays); policy census pins one bucket literal per policy.
|
||
|
||
---
|
||
|
||
## Part II — Proposed design
|
||
|
||
### II.1 Components
|
||
|
||
```text
|
||
Browser (anon key + user JWT)
|
||
│ pick file → decode → canvas downscale (transport) → JPEG blob
|
||
│ begin_avatar_upload (RPC) ─► assets row pending_upload + quarantine key
|
||
│ PUT object (Storage API, INSERT policy = that key, that row, that user)
|
||
│ confirm_asset_upload (RPC) ─► status uploaded
|
||
│ poll own assets row ─► clean | rejected
|
||
│ download(object_key) (Storage API, SELECT policy) ─► blob URL ─► <img>
|
||
▼
|
||
Supabase (self-hosted compose, later managed)
|
||
PostgREST ─ public.assets, RPCs, private helpers Storage API ─ 8 private buckets
|
||
▲
|
||
│ claim_avatar_jobs / complete_avatar_job / reject_avatar_job (service role)
|
||
│ claim_purges / ack_purge (service role)
|
||
│ GET quarantine, PUT clean, DELETE objects (service role)
|
||
apps/worker — container faunosteo_worker, outbound only, no port
|
||
keyed I/O loop ──stdin/stdout──► unkeyed decode child (sharp), 20 s deadline, SIGKILL
|
||
sweeper: purge queue (row-driven), quarantine leftovers (key-driven)
|
||
```
|
||
|
||
Why a worker: image validation needs a native decoder on the server side,
|
||
behind the service key, with a hard resource bound. Postgres cannot read
|
||
object bytes; a client-side re-encode is not validation (a polyglot
|
||
passes); pg_net or a webhook would give the worker an inbound port. Edge
|
||
Functions would be one more container on self-hosted too, hand-deployed,
|
||
and on managed they decode in WASM under a CPU and memory budget not sized
|
||
for photos (`sharp` is native and cannot run there). The worker is one Node
|
||
24 container, the same code against the local stack, the VPS and a managed
|
||
project (URL + key only). It is the `apps/worker` of plan Step 19, arriving
|
||
early with one job family; Step 19 extends it.
|
||
|
||
Dependency added: `sharp` 0.35.x (libvips). The only maintained Node
|
||
library that decodes JPEG / PNG / WebP, resizes, re-encodes and strips
|
||
metadata; pure-JS alternatives cover PNG only or are unmaintained. The
|
||
worker imports no workspace package, so Node 24 runs its TypeScript
|
||
directly (type stripping), no bundler, no `tsx`.
|
||
|
||
### II.2 Buckets
|
||
|
||
All eight private. One migration, `on conflict (id) do update`, so it
|
||
converges on the local stack, the VPS and managed alike.
|
||
|
||
| Bucket | Owning domain and step | Step 8 state |
|
||
|---|---|---|
|
||
| `avatars` | Identity (self) | **open**: SELECT + INSERT policies through `assets`, 5 MiB, image/jpeg · png · webp |
|
||
| `logos` | Organizations, Step 10 | closed, no policy, no cap yet |
|
||
| `professional-documents` | Verification, Step 9 | closed |
|
||
| `client-files` | Clients, Step 12 | closed |
|
||
| `animal-files` | Animals, Step 12 | closed |
|
||
| `report-files` | PDF artifacts, Step 15 | closed |
|
||
| `anatomical-sheet-assets` | Sheets (existing policy, kept) | unchanged |
|
||
| `imports` | Imports, Step 24 | closed |
|
||
|
||
"Closed" = the bucket row exists with `public = false`, no policy on
|
||
`storage.objects` names it, so `anon` and `authenticated` can neither list,
|
||
read, upload, update nor delete; only the service role can. Caps and MIME
|
||
lists of closed buckets are set by their own step. A pgTAP census asserts
|
||
that every `storage.objects` policy pins exactly one bucket literal
|
||
(`bucket_id = '…'`), counts them per bucket (2 on `avatars`, 1 on
|
||
`anatomical-sheet-assets`, 0 elsewhere) and pins the total, so a generic
|
||
policy naming no bucket cannot open the others.
|
||
|
||
Open point (D2): the brief names the sheet bucket `anatomical-sheets`; the
|
||
repository already has `anatomical-sheet-assets` with policy, operator
|
||
script, tests and seed. Recommendation: keep the name, record the deviation.
|
||
|
||
### II.3 `public.assets` and the RPCs
|
||
|
||
One row per file, created before the bytes exist, kept after they are gone
|
||
(status `deleted`) until the purge queue has been served.
|
||
|
||
```sql
|
||
create type public.asset_status as enum
|
||
('pending_upload', 'uploaded', 'clean', 'rejected', 'deleted');
|
||
|
||
create table public.assets (
|
||
id uuid primary key default gen_random_uuid(),
|
||
owner_id uuid not null references public.profiles (id) on delete cascade,
|
||
kind text not null check (kind in ('profile_avatar')), -- extended per step
|
||
bucket_id text not null check (bucket_id = 'avatars'), -- extended with kind
|
||
object_key text not null unique, -- final key, server-built
|
||
quarantine_key text, -- upload key, null once purged
|
||
status public.asset_status not null default 'pending_upload',
|
||
declared_mime text not null,
|
||
declared_size integer not null check (declared_size between 1 and 5242880),
|
||
byte_size integer, -- clean output
|
||
sha256 text check (sha256 ~ '^[0-9a-f]{64}$'),
|
||
rejection_reason text, -- closed codes, II.7
|
||
attempts smallint not null default 0,
|
||
locked_at timestamptz, -- lease, II.8
|
||
created_at timestamptz not null default now(),
|
||
updated_at timestamptz not null default now()
|
||
);
|
||
-- "the" avatar of an account = its single clean row (no profile column, no trigger)
|
||
create unique index assets_one_clean_avatar
|
||
on public.assets (owner_id) where kind = 'profile_avatar' and status = 'clean';
|
||
create index assets_claimable on public.assets (created_at) where status = 'uploaded';
|
||
```
|
||
|
||
Privileges: `revoke all` from API roles, `grant select` to `authenticated`,
|
||
RLS `assets_select_own` (`owner_id = auth.uid()`). No INSERT / UPDATE /
|
||
DELETE policy: every write is an RPC (security definer, `search_path = ''`,
|
||
authorisation before lookup, EXECUTE revoked from PUBLIC, granted per role,
|
||
`private.log_event` on sensitive transitions). pgTAP 001's census of
|
||
`private` functions executable by `authenticated` is updated for the two
|
||
new predicates.
|
||
|
||
| RPC | Role | Effect |
|
||
|---|---|---|
|
||
| `begin_avatar_upload(declared_mime, declared_size)` | authenticated | locks the caller's `profiles` row (`for update`, serialises a double click); rate: ≤ 10 rows of the caller created in the last hour, all statuses counted → `rate_limited`; global ceiling: sum of `declared_size` of non-deleted rows < `private.storage_ceiling()` → `storage_full`; caps (type in the list, size ≤ 5 MiB); supersedes any `pending_upload` / `uploaded` row of the caller (→ `deleted`, enqueued for purge); inserts the row with `object_key = profiles/<uid>/<id>/avatar.webp`, `quarantine_key = quarantine/<uid>/<id>/upload`; returns `{asset_id, bucket_id, quarantine_key}` |
|
||
| `confirm_asset_upload(asset_id)` | authenticated | own row, `pending_upload`, a `storage.objects` row exists at `(bucket_id, quarantine_key)` with `metadata->>'size'` ≤ cap → `uploaded`. Kept as an explicit client signal: the worker never has to guess from Storage API internals whether a PUT finished |
|
||
| `remove_avatar()` | authenticated | every `profile_avatar` row of the caller not already `deleted` / `rejected` (clean and in flight) → `deleted`; logs `asset.avatar_removed` |
|
||
| `claim_avatar_jobs(max_jobs)` | service_role | `uploaded` rows with `locked_at is null or locked_at < now() - 10 min` and `attempts < 3`, `for update skip locked`, oldest first → sets `locked_at`, `attempts + 1`, returns the rows; a row at its 3rd expired lease becomes `rejected (processing_error)` instead. VOLATILE (a STABLE function runs read-only through PostgREST) |
|
||
| `complete_avatar_job(asset_id, byte_size, sha256)` | service_role | guard `status = 'uploaded'` (a superseded row is not completed: the worker then deletes what it wrote) → marks the caller's previous `clean` avatar `deleted` (enqueued), sets the row `clean`; one transaction; logs `asset.avatar_set` |
|
||
| `reject_avatar_job(asset_id, reason)` | service_role | guard `uploaded` → `rejected` with a closed code; logs `asset.rejected` |
|
||
| `mark_quarantine_purged(asset_id)` | service_role | nulls `quarantine_key` after the worker deleted the quarantine object of a `clean` row |
|
||
| `claim_purges(max_items)` / `ack_purge(purge_id)` | service_role | the purge queue, II.9 |
|
||
|
||
Why `assets` does not absorb `anatomical_sheet_assets`: that table is
|
||
content-addressed and sheet-scoped (BDR-032/034) with its own confirmed
|
||
upload path; the kinds that join `assets` later (logos, documents, client
|
||
and animal files, report files, imports) extend the `kind` CHECK and add a
|
||
predicate branch at their step. The profile pointer is derived by the
|
||
unique index rather than the pre-announced `profiles.avatar_asset_id`
|
||
column: no cross-table write, no trigger, three fewer tests; a column can be
|
||
added additively when a public profile needs a join (D5).
|
||
|
||
### II.4 Object keys
|
||
|
||
- Final: `profiles/<user_id>/<asset_id>/avatar.webp`. The name is fixed per
|
||
kind; the user's filename never reaches the server.
|
||
- Quarantine: `quarantine/<user_id>/<asset_id>/upload` in the same bucket.
|
||
No API role can read it: the SELECT predicate only matches the
|
||
`object_key` of a `clean` row, the INSERT predicate only the
|
||
`quarantine_key` of the caller's `pending_upload` row.
|
||
- Keys are built by the RPC from uuids; the policies compare whole strings,
|
||
no prefix match, no `storage.foldername()`, so the PGlite shim and the real
|
||
stack evaluate the same expression. The asset id in the key lets the
|
||
sweeper classify a leftover without a row.
|
||
- Sheet assets keep `<sha256>.svg` (shared drawings, immutable versions).
|
||
|
||
### II.5 Upload and processing (avatar)
|
||
|
||
1. Browser: `File.size` ≤ 20 MiB (sanity), `createImageBitmap` decodes it
|
||
(else `unsupported_type`), draw on a canvas with the longest side capped
|
||
at 1024 px, `toBlob('image/jpeg', 0.9)`. This is transport only (phones
|
||
produce 24 MP / 8 MiB photos; a 1024 px JPEG weighs ~150 KB); the server
|
||
re-decodes and re-encodes regardless. The preview shows that blob in the
|
||
circle with `object-fit: cover`. Nothing uploaded yet.
|
||
2. "Enregistrer la photo" → `begin_avatar_upload('image/jpeg', blob.size)`.
|
||
3. `storage.from('avatars').upload(quarantine_key, blob, { upsert: false, contentType: 'image/jpeg' })`
|
||
with the user JWT. The INSERT policy accepts only
|
||
`(bucket_id = 'avatars', name = quarantine_key)` of a `pending_upload`
|
||
row owned by `auth.uid()`; `upsert` would need UPDATE and SELECT rights
|
||
the plan withholds on purpose.
|
||
4. `confirm_asset_upload(asset_id)` → `uploaded`.
|
||
5. Worker loop, fixed 2 s interval: `claim_avatar_jobs(1)` → GET the
|
||
quarantine object (service client, 30 s timeout on every network call).
|
||
6. Decode child: the I/O process spawns `decode.ts` with an empty
|
||
environment (no key, `PATH` only), feeds the bytes on stdin, reads the
|
||
result on stdout, kills it with SIGKILL after 20 s. The child: type from
|
||
magic bytes (JPEG `FF D8 FF`, PNG 8-byte signature, WebP `RIFF….WEBP`),
|
||
`sharp(bytes, { limitInputPixels: 16_777_216 })` with default `failOn`,
|
||
`sharp.cache(false)`, `sharp.concurrency(1)`, `metadata()` (format matches
|
||
the magic; 64 ≤ sides ≤ 4096; the pixel-limit error maps to
|
||
`dimensions`), `.rotate()`, `.resize(512, 512, { fit: 'cover',
|
||
position: 'centre' })`, `.webp({ quality: 82 })`, no `withMetadata()`:
|
||
EXIF, XMP, IPTC and the ICC profile are not copied (colours converted to
|
||
sRGB through the profile first). Output: one 512 × 512 WebP, sha256,
|
||
byte size. Data after a PNG `IEND` is ignored by the decoder and absent
|
||
from the output.
|
||
7. I/O process: PUT the clean bytes at `object_key` (`upsert: true`,
|
||
`contentType: 'image/webp'`; idempotent on retry) → `complete_avatar_job`
|
||
(commit) → DELETE the quarantine object → `mark_quarantine_purged`. A
|
||
refusal → `reject_avatar_job(reason)`; the quarantine object is
|
||
enqueued for purge by the status trigger. A `complete` refused by the
|
||
guard (row superseded) → the worker deletes the clean object it wrote.
|
||
8. Browser polls its own row (1 s, up to 30 s): `clean` → download the
|
||
photo; `rejected` → message from the closed code; still `uploaded` after
|
||
30 s → "Traitement en cours, revenez dans un instant"; on the next visit
|
||
the account page loads any in-flight row and shows the same state. The
|
||
save button is disabled while a row of the caller is in flight; a new
|
||
pick supersedes it.
|
||
|
||
Crash matrix (the retry is idempotent at every point):
|
||
|
||
| Worker dies | Row | Objects | Recovery |
|
||
|---|---|---|---|
|
||
| before PUT clean | `uploaded`, lease set | quarantine | lease expires (10 min) → re-claim |
|
||
| after PUT clean, before complete | same | quarantine + clean | re-claim, PUT again (`upsert`), complete |
|
||
| after complete, before DELETE quarantine | `clean`, `quarantine_key` set | both | sweeper: quarantine leftovers of `clean` rows, then `mark_quarantine_purged` |
|
||
| 3rd lease expired | `rejected (processing_error)` | enqueued | purge queue |
|
||
|
||
### II.6 Read path (owner-only, no signed URL)
|
||
|
||
`storage.from('avatars').download(object_key)` with the user JWT: the
|
||
Storage API evaluates the SELECT policy, the bytes come back, the app keeps
|
||
a `URL.createObjectURL(blob)` in the auth store (revoked on change or sign
|
||
out) and uses it as `<img src>` in `UserBadge`, the account section and, if
|
||
D3 is accepted, the top bar. One download of ~30 KB per session; the
|
||
production CSP already allows `blob:` images, `connect-src` must name the
|
||
Supabase origin (already required by Step 7's API calls).
|
||
|
||
Why not a signed URL: in Step 8 nobody but the owner reads the avatar, and a
|
||
signed URL is a bearer token that keeps serving after `remove_avatar` until
|
||
it expires. Signed URLs arrive with the first kind shared with other users
|
||
(documents, Step 9): issued by the Storage API after the SELECT policy, TTL
|
||
per kind, `download` flag for documents, query strings dropped from the
|
||
proxy access logs.
|
||
|
||
### II.7 Rejection codes (closed set, phrased by i18n)
|
||
|
||
Row codes: `unsupported_type`, `decode_failed`, `dimensions`,
|
||
`processing_error`. Client-side failure kinds (never rows): `too_large`,
|
||
`unsupported_type`, `rate_limited`, `storage_full`, `network`, `unknown`.
|
||
The raw library or supabase-js error never reaches a row or the screen; the
|
||
worker logs an allowlist of fields (asset id, code, duration) with control
|
||
characters escaped (LRN-044).
|
||
|
||
### II.8 Worker runtime and operations
|
||
|
||
- `apps/worker` (`@faunosteo/worker`): TypeScript run by Node 24, deps
|
||
`@supabase/supabase-js` and `sharp` only. No DB driver: every database
|
||
access is an RPC through PostgREST with the service key (one credential,
|
||
one transport; a managed project needs no open database port).
|
||
- Two processes in one container: the keyed I/O loop and the unkeyed decode
|
||
child per job (II.5 step 6). A decoder exploit lands in a process with no
|
||
key, no file access beyond stdin/stdout and a 20 s life; it can reach
|
||
Kong with anonymous rights only, which is what any browser has.
|
||
- Container `faunosteo_worker` (compose service `worker`): non-root,
|
||
`read_only: true` + `tmpfs /tmp`, `mem_limit: 512m`, `cpus: 1`,
|
||
`pids_limit: 64`, no published port, attached to the Supabase compose
|
||
network only (`internal`, Kong reachable, nothing else; a host firewall
|
||
rule would not filter container egress), `restart: unless-stopped`. A
|
||
watchdog exits the process when one loop iteration exceeds 60 s so the
|
||
restart policy fires (Docker never restarts an `unhealthy` container on
|
||
its own). Env file `.env.worker` (`SUPABASE_URL`,
|
||
`SUPABASE_SERVICE_ROLE_KEY`, `WORKER_ID`), `chmod 600`, separate from the
|
||
app `.env`, excluded from any backup copy of env files, readable through
|
||
`docker inspect` by the `docker` group (documented).
|
||
- Fairness: one in-flight avatar per account (begin supersedes), oldest
|
||
first → no account can queue more than one job.
|
||
- Sweeper every 10 min, two sources: (a) the purge queue (row-driven, II.9);
|
||
(b) key-driven: list `quarantine/` of the bucket, parse the asset id,
|
||
delete any object older than 3 h whose row is not `pending_upload` /
|
||
`uploaded` (covers bytes landed through a signed upload URL or a slow PUT
|
||
after the row moved on; the 3 h grace exceeds the 2 h validity of such a
|
||
URL). No operator step, no `--apply`.
|
||
- `sharp` updates: a Renovate / dependabot rule on the worker package,
|
||
not a manual chore.
|
||
- Local dev: `pnpm worker:dev` against the local stack (service key typed
|
||
into the shell as SUPABASE.md §8 already does).
|
||
|
||
### II.9 Purge queue, orphans, replacement, deletion
|
||
|
||
`private.object_purge_queue(id, bucket_id, object_name, not_before,
|
||
enqueued_at)` fed by triggers on `assets`: a transition to `deleted` or
|
||
`rejected` enqueues `object_key` and `quarantine_key`; a row DELETE (account
|
||
cascade) enqueues them too. `not_before` = now + 48 h for the final key of a
|
||
`deleted` row (longer than any backup run, II.10), now + 1 h for quarantine
|
||
keys and rejected uploads. The sweeper calls `claim_purges(max_items)`,
|
||
deletes each object (a missing object is a success), `ack_purge(id)`. The
|
||
only automated deletions are those two sources; a `clean` row's `object_key`
|
||
is never deleted (test).
|
||
|
||
| Event | Row | Object | Cleanup |
|
||
|---|---|---|---|
|
||
| New avatar clean | new `clean`, previous `deleted` (same transaction) | previous object kept 48 h | queue |
|
||
| `remove_avatar()` | clean and in-flight rows `deleted` | kept 48 h | queue |
|
||
| New pick while in flight | superseded row `deleted` | its bytes, if any | queue; a `complete` for it is refused |
|
||
| Upload never confirmed | `pending_upload` superseded or expired by the next begin / after 24 h | maybe present | queue |
|
||
| Bytes landed after the row moved on | none or not pending | present | key-driven sweep after 3 h |
|
||
| Account deleted (Step 26) | rows cascade | enqueued by the DELETE trigger | queue |
|
||
|
||
Access is cut by the row: a `deleted` or `rejected` row matches no read
|
||
predicate, and no URL outlives the policy check in Step 8 (download, not
|
||
signing). Object lifetime is eventual, bounded by the grace periods.
|
||
|
||
Reconciliation (operator, documented in STORAGE.md, not a tool): one SQL
|
||
anti-join between `storage.objects` and `assets` (same database) lists
|
||
objects without a row and rows without an object per bucket; sheet assets
|
||
are checked against `anatomical_sheet_assets`. Reading only; deletions stay
|
||
row-driven.
|
||
|
||
### II.10 Backups and restore on the VPS (self-hosted)
|
||
|
||
- Storage backend `file` on a named volume `faunosteo_supabase_storage`
|
||
(kept across renames, BDR-035), on a filesystem separate from the
|
||
PostgreSQL volume with a quota, and a disk-usage alert: the VPS hosts
|
||
other products, a full disk would take PostgreSQL down with it. The S3
|
||
backend (Scaleway Object Storage) remains the alternative when the volume
|
||
outgrows the disk: copy the files, change the backend env, rows unchanged.
|
||
- Global ceiling: `private.storage_ceiling()` (constant, 10 GiB to start)
|
||
checked by `begin_avatar_upload` on top of the per-account rate.
|
||
- Backup order: database dump first (whole instance, `storage` schema
|
||
included), files second. With the 48 h purge grace, every `clean` row of
|
||
the dump still has its object in the snapshot, provided the snapshot runs
|
||
within 48 h of the dump (the invariant the restore checks: every `clean`
|
||
row has an object).
|
||
- Restore: stop the worker, start the database, restore the dump, restore
|
||
the volume, start the rest, run the reconciliation query, start the
|
||
worker. Files newer than the dump have no row and are invisible to the
|
||
Storage API (it lists from `storage.objects`); the directory listing of
|
||
the volume against `storage.objects` finds them.
|
||
- Tooling (restic to Scaleway Object Storage, encrypted, dump in the same
|
||
run, write-only key with retention pruned from another host, the
|
||
Supabase secrets `JWT_SECRET` / `POSTGRES_PASSWORD` in the encrypted set,
|
||
`.env.worker` never in a plaintext copy) and the restore drill belong to
|
||
the self-hosted deployment runbook (plan Step 27 "backup/restore
|
||
rehearsal"). Step 8 writes the ordering, the invariant and the backend
|
||
choice in `STORAGE.md`; the owner runs every backup or restore command.
|
||
|
||
### II.11 Migration to managed Supabase (runbook, no tool in Step 8)
|
||
|
||
Order: freeze writes (stop the worker, maintenance notice); migrate the
|
||
schema (`supabase db push`, buckets and policies are SQL; the `storage`
|
||
schema itself is never dumped into managed); migrate the data rows
|
||
(`auth.users` through the Supabase auth migration path, then `public`:
|
||
profiles, roles, sheets, `assets`); copy the objects driven by the rows
|
||
(`assets` and `anatomical_sheet_assets`: for every `clean` / confirmed row,
|
||
download, verify sha256, upload at the same key, verify; paginate by keyset,
|
||
PostgREST caps a set at 1000 rows); run the reconciliation query on the
|
||
destination; switch `SUPABASE_URL` and the keys of the app and the worker;
|
||
resume. The service key of a self-hosted stack is a long-lived JWT: revoking
|
||
it means rotating `JWT_SECRET` (every key and session); documented, and a
|
||
reason to prefer managed secret keys once there.
|
||
|
||
### II.12 UI (follows the validated mockup, no redesign)
|
||
|
||
- Mon compte → Informations personnelles: avatar circle (photo, else the
|
||
monogram from `UserBadge`) with "Modifier la photo" under it, as drawn.
|
||
Picking a file shows the local preview in the circle with "Enregistrer la
|
||
photo" / "Annuler"; after saving, the circle shows "Traitement en cours…"
|
||
until the row is `clean`; "Supprimer la photo" appears when a photo
|
||
exists (one confirmation). Errors use `account.avatar.error.<code>`.
|
||
- `UserBadge` (sidebar + drawer): photo when present, monogram otherwise.
|
||
- Top bar (D3): the mockup shows avatar + name with a chevron on desktop and
|
||
the avatar alone on mobile; Step 7 shipped the bar without a user element.
|
||
Proposal: add the avatar (and the name on desktop) as a link to Mon
|
||
compte, no dropdown (its content is undefined). Decision for the owner.
|
||
- Dashboard unchanged (the mockup shows no avatar in the content area).
|
||
- Accepted inputs at the picker: JPEG, PNG, WebP; never SVG for avatars.
|
||
What the browser cannot decode is refused there with `unsupported_type`;
|
||
what it decodes is sent as JPEG and judged again by the worker.
|
||
|
||
---
|
||
|
||
## Part III — The ten security questions
|
||
|
||
1. **Access control**: the `assets` row is the authority. The two policies
|
||
on `storage.objects` are bucket-scoped
|
||
(`bucket_id = 'avatars' and private.can_read_avatar_object(uid, name)`,
|
||
same for upload) and look the key up in `assets` (status, owner, kind).
|
||
A bucket without policy is unreachable by API roles; the census pins one
|
||
bucket literal per policy and the counts.
|
||
2. **No cross-user access**: the read predicate requires a `clean` row with
|
||
`owner_id = auth.uid()` whose `object_key` equals the requested name; the
|
||
path is never trusted, only the row. Listing is denied the same way.
|
||
pgTAP: Bob reads nothing of Alice, cannot upload at her key, cannot
|
||
confirm her row; real-stack e2e: Bob's `download` of Alice's key → 400.
|
||
3. **Keys**: server-built from uuids by the RPC, canonical file name per
|
||
kind, no user filename anywhere, whole-string comparison in the policies.
|
||
4. **Signed URLs**: not used in Step 8. An owner-only avatar is read by
|
||
`download` under RLS and shown through a blob URL, so no bearer token
|
||
exists and a removed avatar stops being readable at the row flip. When a
|
||
shared kind arrives (documents, Step 9): Storage API signing after the
|
||
SELECT policy, TTL per kind (60 s and `download` for documents), query
|
||
strings dropped from access logs, the row cuts signing and the purge cuts
|
||
serving.
|
||
5. **Quarantine**: upload lands under `quarantine/` with the row
|
||
`pending_upload` → `uploaded`; no API role can read it; only the worker
|
||
reads it, writes the clean bytes under the final key, commits the row,
|
||
deletes the quarantine object. Every transition is an RPC; the browser
|
||
cannot set `clean`. Bytes landed late (signed upload URL, slow PUT) are
|
||
swept by key after 3 h without an operator.
|
||
6. **Dangerous SVG**: avatars never accept SVG (the browser decodes raster
|
||
only, the worker checks magic bytes, the bucket lists three raster types).
|
||
For sheet drawings the allowlist sanitiser exists and is tested on
|
||
hostile fixtures (`packages/pdf`); it becomes the worker's SVG processor
|
||
at Step 13 with its producer, after a pre-parse byte and tag-count cap
|
||
(the DOM is built before the element cap counts) and a bounded removed
|
||
list (D6).
|
||
7. **Images without metadata**: the worker never stores the uploaded bytes.
|
||
The browser already re-encoded through a canvas (no EXIF survives), and
|
||
the worker decodes again (type from magic bytes, 16.7 MP pixel cap, sides
|
||
≤ 4096), applies the EXIF orientation if any, resizes and re-encodes to
|
||
WebP without `withMetadata()`: no EXIF, XMP, IPTC or ICC chunk in the
|
||
output. Tests: an EXIF-laden JPEG fed directly to the worker yields an
|
||
output without `EXIF` / `XMP ` / `ICCP` chunks, upright.
|
||
8. **Orphans**: replacement, removal, supersede and rejection flip the row
|
||
in one transaction and enqueue the keys; the sweeper serves the queue
|
||
after the grace period and sweeps quarantine leftovers by key; a `clean`
|
||
row's object is never deleted; the reconciliation query reports the rest.
|
||
9. **Backups and restore**: file backend on a named, quota-limited volume;
|
||
dump first, files second, 48 h purge grace so every `clean` row of a dump
|
||
has its object; restore in the documented order followed by the
|
||
reconciliation query; tooling and drill in the deployment runbook.
|
||
10. **Managed migration**: same migrations; rows then objects driven by the
|
||
rows with hash verification; keys and URLs switched by env; write freeze
|
||
during the copy.
|
||
|
||
Service role: only `.env.worker` and operator commands hold it; the decode
|
||
child never sees it; `env.js` is generated from `APP_*` variables only.
|
||
|
||
---
|
||
|
||
## Part IV — Scope, tests, risks, decisions
|
||
|
||
### IV.1 Exact scope of the implementation
|
||
|
||
In:
|
||
|
||
- Migration `20261012100000_files_foundation.sql`: `asset_status`, `assets`
|
||
(CHECKs, indexes), seven bucket rows (the existing one untouched),
|
||
`private.can_read_avatar_object` / `can_upload_avatar_object`,
|
||
`storage.objects` SELECT + INSERT policies for `avatars`,
|
||
`private.object_purge_queue` + triggers, `private.storage_ceiling()`, the
|
||
nine RPCs of II.3, explicit EXECUTE revokes. pgTAP `010_files_foundation.sql`
|
||
(allowed and denied cases, policy census), pgTAP 001 census update, shim
|
||
extension if a column is missing, `database.types.ts` regenerated.
|
||
- `apps/worker`: I/O loop, decode child, sweeper (queue + key-driven),
|
||
watchdog, Dockerfile (`node:24-alpine`, non-root), compose service,
|
||
`.env.worker.example`, vitest on fixtures (with wall-time and RSS
|
||
assertions on the pathological ones), an integration test against the
|
||
local stack run by the CI `supabase` job.
|
||
- `packages/supabase/src/assets.ts`: `beginAvatarUpload`,
|
||
`uploadToQuarantine`, `confirmAssetUpload`, `removeAvatar`,
|
||
`fetchOwnAvatar` (clean or in flight), `downloadAvatar`; closed
|
||
`AssetFailure` kinds.
|
||
- `apps/app`: avatar block in `PersonalInfoForm` (picker, canvas transport,
|
||
preview, save, processing, delete), `UserBadge` photo, top bar avatar if
|
||
D3 = yes, auth store blob URL, i18n keys FR/EN, component and store tests.
|
||
- Docs: `docs/development/STORAGE.md` (model, keys, statuses, worker
|
||
operations, purge and grace rules, reconciliation query, backup ordering
|
||
and invariant, managed migration runbook), DEPLOY.md (worker service, env
|
||
file, network, CSP `connect-src`), SUPABASE.md (worker in dev), CLAUDE.md,
|
||
CHANGELOG; plan note (worker early, SVG processor at Step 13).
|
||
- CI: worker typecheck / lint / test / build, `docker build` of the worker
|
||
image, worker integration test in the `supabase` job.
|
||
|
||
Out (named so nothing is implied): SVG processor and user SVG upload path
|
||
(Steps 13 / 17), logos, documents, client / animal / report / import flows
|
||
(their steps), signed URLs, image transformations, public profile photo,
|
||
`profiles.avatar_asset_id`, the production deployment of the self-hosted
|
||
stack, backup tooling and restore drill, migration and reconcile tools,
|
||
account erasure (Step 26), Realtime, a user menu dropdown.
|
||
|
||
### IV.2 Test scenarios
|
||
|
||
pgTAP (PGlite + real stack), every case with its denial twin:
|
||
|
||
- buckets: eight rows, all private; `avatars` cap and types; census: one
|
||
bucket literal per policy, 2 / 1 / 0 per bucket, total pinned;
|
||
`authenticated` cannot insert nor select in a closed bucket.
|
||
- `begin_avatar_upload`: keys of the right shape; wrong type, oversize →
|
||
denied; 11th begin within the hour fails even after supersedes; ceiling
|
||
reached → `storage_full`; a begin while a row is `pending_upload` or
|
||
`uploaded` supersedes it (`deleted`, queue row written); anon denied.
|
||
- upload policy: Alice inserts at her quarantine key; Bob at her key →
|
||
42501; Alice at a final key, at the key of a `clean` row, at a superseded
|
||
key, at an unknown key → denied; no UPDATE / DELETE policy exists.
|
||
- confirm: own pending row with the object present → `uploaded`; without
|
||
the object → error; Bob confirming Alice's row → denied; twice → error.
|
||
- read policy: Alice reads her `clean` object only; nothing while
|
||
`pending_upload` / `uploaded` / `rejected` / `deleted`; the quarantine
|
||
object never; Bob nothing; anon nothing.
|
||
- worker RPCs: `authenticated` cannot execute any; claim leases oldest
|
||
first and skips a live lease; an expired lease is re-claimed; the third
|
||
expiry rejects with `processing_error`; complete marks the previous clean
|
||
avatar `deleted` and enqueues it atomically; complete on a superseded row
|
||
is refused; reject stores a closed code; the queue holds the right keys
|
||
with the right `not_before`; a `clean` row's `object_key` never enters
|
||
the queue.
|
||
- `remove_avatar`: clean and in-flight rows → `deleted`; a later complete
|
||
of the in-flight row is refused (the photo does not reappear).
|
||
- cascade: deleting a profile enqueues its keys (trigger).
|
||
- audit: `asset.avatar_set`, `asset.avatar_removed`, `asset.rejected` rows.
|
||
|
||
Worker (vitest, fixtures under `apps/worker/tests/fixtures`):
|
||
|
||
- decode child: JPEG / PNG / WebP accepted; PNG bytes under a `.jpg` name
|
||
accepted as PNG (the name is never seen); GIF, BMP, SVG, PDF, HTML →
|
||
`unsupported_type`; truncated → `decode_failed`; 32 px → `dimensions`;
|
||
header-declared 20 000 × 20 000 and a 5000 × 5000 → `dimensions` through
|
||
the pixel limit; EXIF-laden rotated JPEG → upright output without EXIF /
|
||
XMP / ICCP chunks; PNG with a trailer after IEND → accepted, trailer
|
||
absent from the output; output always 512 × 512 WebP under a size bound;
|
||
pathological-but-within-cap fixtures (progressive JPEG with many scans,
|
||
large ICC) finish under the deadline with RSS asserted; a child that
|
||
sleeps is killed at 20 s → `processing_error`.
|
||
- I/O loop: claim → decode → PUT → complete → DELETE sequence; complete
|
||
refused → the clean object is deleted; Storage error → row left for the
|
||
lease; the child never receives the environment; sweeper serves the queue
|
||
and the key-driven list only, never a `clean` `object_key`.
|
||
|
||
Integration (local stack, CI `supabase` job): Alice's full flow with the
|
||
service key for the worker side → row `clean`, object at the final key, no
|
||
quarantine object; a second upload replaces, the first object disappears
|
||
once its `not_before` is reached (clock-shifted in the test); `remove_avatar`
|
||
then sweep → bucket empty; a signed upload URL obtained during
|
||
`pending_upload` and used after supersede → object swept by key.
|
||
|
||
Real-stack e2e (scratchpad script, like Step 7): Alice `download` 200, Bob
|
||
`download` of Alice's key 400, anon 400, Alice's quarantine key 400, a
|
||
direct GET of the object path without JWT 400.
|
||
|
||
App (vitest + happy-dom): picker refuses a file the browser cannot decode
|
||
with the phrased error; preview appears before any request; save calls
|
||
begin → upload → confirm in order with a JPEG blob; processing state then
|
||
photo on `clean`; rejection message per code; delete flow; in-flight row
|
||
shown on mount; save disabled while in flight; `UserBadge` shows the photo
|
||
when the store has a blob URL, the monogram otherwise; i18n parity.
|
||
|
||
Manual validation (owner): upload, replace, delete from Mon compte on
|
||
desktop and mobile with a phone photo; photo visible in the sidebar, the
|
||
drawer and the top bar; open the object path in a private window (400);
|
||
upload a renamed PNG (accepted) and an HTML file (refused with a clear
|
||
message); sign in as another seeded user and check nothing of the first
|
||
user shows.
|
||
|
||
### IV.3 Risks
|
||
|
||
- **Decoder vulnerabilities** (libvips / libwebp history): confined to a
|
||
keyless child with a 20 s life; the keyed process never decodes; `sharp`
|
||
updates automated. Residual: a child exploit reaches Kong with anonymous
|
||
rights, which any browser has.
|
||
- **Worker down or stuck**: uploads stay `uploaded`; the UI says so; the
|
||
watchdog exits on a hung loop and the restart policy restarts the
|
||
container; a stuck decode is killed by the deadline; an account can
|
||
always supersede. Monitoring and alerting are Step 27.
|
||
- **Disk**: separate quota-limited filesystem, global ceiling, per-account
|
||
rate, 48 h grace. Residual: a burst of accounts within the ceiling fills
|
||
the quota, not the host.
|
||
- **CSP in production**: `connect-src` must include the Supabase origin
|
||
(already needed by Step 7); documented in DEPLOY.md.
|
||
- **PGlite shim vs real `storage` schema**: policies use only `bucket_id`
|
||
and `name`; pgTAP never deletes storage objects (LRN-041).
|
||
- **Storage API behaviour relied on**: INSERT policy on direct upload,
|
||
SELECT policy on `download`, signed upload URL semantics; the integration
|
||
test on the CI stack pins them on the pinned image.
|
||
- **Service key handling**: a second env file, `chmod 600`, excluded from
|
||
env backups; rotation = `JWT_SECRET` rotation on self-hosted (documented).
|
||
- **Colour**: ICC profiles are converted to sRGB then dropped; only
|
||
out-of-gamut colours clip. Accepted for an avatar.
|
||
- **Plan deviations**: worker at Step 8 instead of 19; SVG processor at
|
||
Step 13; avatar pointer derived instead of a profile column. Recorded.
|
||
|
||
### IV.4 Decisions for the owner
|
||
|
||
| Id | Question | Recommendation |
|
||
|---|---|---|
|
||
| D1 | Server-side processing runtime | Containerised worker `faunosteo_worker`, keyless decode child; no Edge Functions (no native decoder, WASM budget on managed, one more container on self-hosted anyway) |
|
||
| D2 | Sheet bucket name | Keep `anatomical-sheet-assets`, record the deviation from the announced `anatomical-sheets` |
|
||
| D3 | Top bar avatar | Add avatar (+ name on desktop) as a link to Mon compte, no dropdown; or sidebar / drawer / account only |
|
||
| D4 | Avatar pipeline | Browser: decode, canvas ≤ 1024 px, JPEG transport. Server: magic check, ≤ 16.7 MP, one 512 × 512 WebP, centre crop, no metadata. Inputs JPEG / PNG / WebP ≤ 5 MiB at upload |
|
||
| D5 | Avatar pointer | Derived from the single `clean` row (unique index), no `profiles.avatar_asset_id` yet |
|
||
| D6 | SVG processor | Deferred to Step 13 with its producer; the sanitiser and its hostile fixtures already exist in `packages/pdf` |
|
||
| D7 | Read path | `download` under RLS + blob URL (no signed URL, no bearer) for the owner-only avatar |
|
||
| D8 | Processing feedback | Browser polling of the own row (1 s, 30 s cap), worker at a fixed 2 s; Realtime stays off |
|
||
| D9 | Self-hosted Storage backend | File backend on a named, quota-limited volume; dump-first backups with 48 h purge grace; tooling at the deployment step |
|