Third audit (BLOCK 1): counting terminal rows while queued let begin spam pin the ceiling with declared sizes. The ceiling now sums storage.objects sizes of the bucket (whatever the row's state) plus the declared size of pending_upload rows whose quarantine object has not landed: a claim dies with its row, a present object counts whatever its row says, no full scan of the assets history. confirm_asset_upload treats an object without a size as upload_missing. pgTAP (plan 138): rejected rows without bytes weigh nothing, pending claims fill, superseded claims vanish, present bytes fill, unsized object refused. Worker: a single-label cleartext host must start with an ASCII letter (0X… hex forms refused, test added); STORAGE.md names the native-run search-domain residual.
628 lines
40 KiB
Markdown
628 lines
40 KiB
Markdown
# Step 8 — Files foundation and Supabase Storage: technical analysis
|
||
|
||
Status: proposal for the owner's decision, 2026-10-12, revision 2 after three
|
||
independent plan challenges (correctness, robustness, simplicity: 47
|
||
findings, folded in; the changes are listed in I.3). Nothing implemented.
|
||
Applies the owner's brief of 2026-10-12 (Step 7 closed; hosting =
|
||
self-hosted Supabase first, managed later; eight private buckets closed by
|
||
default; quarantine + server validation; avatar as first consumer; service
|
||
role never in the browser).
|
||
|
||
Part I states the facts the design rests on. Part II is the proposed design.
|
||
Part III answers the ten security questions of the brief. Part IV lists the
|
||
exact scope, the tests, the risks and the decisions the owner has to take.
|
||
|
||
---
|
||
|
||
## Part I — Facts
|
||
|
||
### I.1 What exists today
|
||
|
||
| Area | State |
|
||
|---|---|
|
||
| Storage | One bucket, `anatomical-sheet-assets` (migration `20261009100000`): private, SVG only, 20 MiB cap, content-addressed keys `<sha256>.svg`, SELECT policy through the sheet (`private.can_read_sheet_asset_object`), no API write policy, operator upload script with the service key, `confirm_sheet_asset_upload` RPC (service role). pgTAP 006 (28 checks) runs on the PGlite shim and on the real stack; pgTAP 001 pins the exact list of `private` functions `authenticated` may execute. |
|
||
| Storage API | `storage-api v1.79.36` locally (same image family as self-hosted). Verified on the client library (storage-js 2.117): an upload with the user JWT and `upsert: false` needs only an INSERT policy on `storage.objects`, evaluated on `bucket_id` and `name`; the bucket MIME check runs before RLS on the multipart part type (the browser's `File.type`, not sniffed); `createSignedUrl` evaluates the SELECT policy as the caller and answers 400 when denied; a signed upload URL is checked at signing time only and stays valid two hours. Image transformation needs `imgproxy`, absent here and a paid add-on on managed. |
|
||
| Profiles | `avatar_asset_id` was deferred to this step (comment in `20261008100100_profiles.sql`). |
|
||
| Shell UI | `UserBadge` (monogram) in the sidebar and the drawer; the top bar holds only the language switch; "Mon compte" = stacked sections, `PersonalInfoForm` first. The validated mockup shows the avatar with "Modifier la photo" inside the Profil section and an avatar + name in the top bar. Production CSP: `default-src 'self'`, `img-src 'self' data: blob:`. |
|
||
| SVG sanitiser | `packages/pdf/src/svg/{parse,sanitize,view}.ts`: `parseSvg` refuses DOCTYPE / ENTITY, `sanitizeSvgRoot` = allowlist of elements and attributes, in-document `href` / `url(#)` only, backslash refused, caps 20 000 elements / depth 64 / 256 `<use>` / `stdDeviation` 100; `cleanSvgRoot` chains both. Hostile fixtures already covered by `packages/pdf/tests/security.test.ts` (script, `on*`, `javascript:`, external image, CSS escapes, ENTITY, 257 `<use>`, `horse.svg` unchanged). Not exported from the package index; serialisation drops the XML declaration (byte change). |
|
||
| Edge runtime | Disabled in `config.toml`. `BACKEND_ARCHITECTURE.md` §6; C3: PostgreSQL-backed jobs before a queue; C12: the sanitiser runs between upload and publication. |
|
||
| Worker | The plan creates `apps/worker` at Step 19 (notifications). Files processing needs server-side code before that. |
|
||
| Deploy | No backend in production yet. The VPS `.env` is documented as non-sensitive. The VPS is shared by several SaaS (BDR-035). |
|
||
| sharp 0.35.5 | No install script, Node ≥ 20.9, prebuilt `linuxmusl-x64`. `rotate()` with no angle auto-orients from EXIF; output strips all metadata by default and converts through the embedded ICC profile to sRGB; `failOn` defaults to `warning` (stricter than `error`); `limitInputPixels` is enforced before `metadata()` returns. |
|
||
| Tests | pgTAP on PGlite + real stack; app vitest on happy-dom; Step 7 added a headless e2e script against the local stack. PostgREST `max_rows = 1000` caps set-returning RPCs. |
|
||
|
||
### I.2 Corrections to the plan's Step 8 text
|
||
|
||
- "RPC `issue_signed_url(asset_id)`" cannot exist in SQL: signing needs the
|
||
Storage API's secret. The Storage API signs only after the SELECT policy
|
||
passes, so the check and the signature already live in one place. For an
|
||
owner-only avatar, Step 8 does not sign at all (II.6).
|
||
- "Edge Function or SQL-only sanitizer": SQL cannot parse XML or decode
|
||
images; a containerised worker is the fit (II.1).
|
||
- "Upload a hostile SVG as a sheet asset and verify it is rewritten": no user
|
||
can upload a sheet asset before Step 13, so this manual check moves there
|
||
with the processor (D6).
|
||
|
||
### I.3 Changes after the challenges (revision 2)
|
||
|
||
Decoder isolated from the service key and bounded by a hard deadline;
|
||
write order PUT clean → commit → delete quarantine; `uploaded` rows
|
||
superseded by a new upload instead of a blocking unique index; purge queue
|
||
with a grace period longer than a backup run (the backup invariant was
|
||
false otherwise); every quarantine leftover auto-purged (signed upload URLs
|
||
can land bytes after the row moved on); client-side downscale as transport
|
||
(current phones exceed a 24 MP cap); owner-only avatar read by `download`
|
||
under RLS, no signed URL, no bearer token; schema narrowed to the one kind
|
||
that has rules (no multi-kind dispatcher, no derived-bucket trigger, no
|
||
unread columns); avatar pointer derived by a unique index instead of a
|
||
profile column + trigger; SVG processor, `migrate` and `reconcile`
|
||
commands, backup tooling and restore drill moved out of Step 8 (runbook
|
||
text stays); policy census pins one bucket literal per policy.
|
||
|
||
---
|
||
|
||
## Part II — Proposed design
|
||
|
||
### II.1 Components
|
||
|
||
```text
|
||
Browser (anon key + user JWT)
|
||
│ pick file → decode → canvas downscale (transport) → JPEG blob
|
||
│ begin_avatar_upload (RPC) ─► assets row pending_upload + quarantine key
|
||
│ PUT object (Storage API, INSERT policy = that key, that row, that user)
|
||
│ confirm_asset_upload (RPC) ─► status uploaded
|
||
│ poll own assets row ─► clean | rejected
|
||
│ download(object_key) (Storage API, SELECT policy) ─► blob URL ─► <img>
|
||
▼
|
||
Supabase (self-hosted compose, later managed)
|
||
PostgREST ─ public.assets, RPCs, private helpers Storage API ─ 8 private buckets
|
||
▲
|
||
│ claim_avatar_jobs / complete_avatar_job / reject_avatar_job (service role)
|
||
│ claim_purges / ack_purge (service role)
|
||
│ GET quarantine, PUT clean, DELETE objects (service role)
|
||
apps/worker — container faunosteo_worker, outbound only, no port
|
||
keyed I/O loop ──stdin/stdout──► unkeyed decode child (sharp), 20 s deadline, SIGKILL
|
||
sweeper: purge queue (row-driven), quarantine leftovers (key-driven)
|
||
```
|
||
|
||
Why a worker: image validation needs a native decoder on the server side,
|
||
behind the service key, with a hard resource bound. Postgres cannot read
|
||
object bytes; a client-side re-encode is not validation (a polyglot
|
||
passes); pg_net or a webhook would give the worker an inbound port. Edge
|
||
Functions would be one more container on self-hosted too, hand-deployed,
|
||
and on managed they decode in WASM under a CPU and memory budget not sized
|
||
for photos (`sharp` is native and cannot run there). The worker is one Node
|
||
24 container, the same code against the local stack, the VPS and a managed
|
||
project (URL + key only). It is the `apps/worker` of plan Step 19, arriving
|
||
early with one job family; Step 19 extends it.
|
||
|
||
Dependency added: `sharp` 0.35.x (libvips). The only maintained Node
|
||
library that decodes JPEG / PNG / WebP, resizes, re-encodes and strips
|
||
metadata; pure-JS alternatives cover PNG only or are unmaintained. The
|
||
worker imports no workspace package, so Node 24 runs its TypeScript
|
||
directly (type stripping), no bundler, no `tsx`.
|
||
|
||
### II.2 Buckets
|
||
|
||
All eight private. One migration, `on conflict (id) do update`, so it
|
||
converges on the local stack, the VPS and managed alike.
|
||
|
||
| Bucket | Owning domain and step | Step 8 state |
|
||
|---|---|---|
|
||
| `avatars` | Identity (self) | **open**: SELECT + INSERT policies through `assets`, 5 MiB, image/jpeg · png · webp |
|
||
| `logos` | Organizations, Step 10 | closed, no policy, no cap yet |
|
||
| `professional-documents` | Verification, Step 9 | closed |
|
||
| `client-files` | Clients, Step 12 | closed |
|
||
| `animal-files` | Animals, Step 12 | closed |
|
||
| `report-files` | PDF artifacts, Step 15 | closed |
|
||
| `anatomical-sheet-assets` | Sheets (existing policy, kept) | unchanged |
|
||
| `imports` | Imports, Step 24 | closed |
|
||
|
||
"Closed" = the bucket row exists with `public = false`, no policy on
|
||
`storage.objects` names it, so `anon` and `authenticated` can neither list,
|
||
read, upload, update nor delete; only the service role can. Caps and MIME
|
||
lists of closed buckets are set by their own step. A pgTAP census asserts
|
||
that every `storage.objects` policy pins exactly one bucket literal
|
||
(`bucket_id = '…'`), counts them per bucket (2 on `avatars`, 1 on
|
||
`anatomical-sheet-assets`, 0 elsewhere) and pins the total, so a generic
|
||
policy naming no bucket cannot open the others.
|
||
|
||
Open point (D2): the brief names the sheet bucket `anatomical-sheets`; the
|
||
repository already has `anatomical-sheet-assets` with policy, operator
|
||
script, tests and seed. Recommendation: keep the name, record the deviation.
|
||
|
||
### II.3 `public.assets` and the RPCs
|
||
|
||
One row per file, created before the bytes exist, kept after they are gone
|
||
(status `deleted`) until the purge queue has been served.
|
||
|
||
```sql
|
||
create type public.asset_status as enum
|
||
('pending_upload', 'uploaded', 'clean', 'rejected', 'deleted');
|
||
|
||
create table public.assets (
|
||
id uuid primary key default gen_random_uuid(),
|
||
owner_id uuid not null references public.profiles (id) on delete cascade,
|
||
kind text not null check (kind in ('profile_avatar')), -- extended per step
|
||
bucket_id text not null check (bucket_id = 'avatars'), -- extended with kind
|
||
object_key text not null unique, -- final key, server-built
|
||
quarantine_key text, -- upload key, null once purged
|
||
status public.asset_status not null default 'pending_upload',
|
||
declared_mime text not null,
|
||
declared_size integer not null check (declared_size between 1 and 5242880),
|
||
byte_size integer, -- clean output
|
||
sha256 text check (sha256 ~ '^[0-9a-f]{64}$'),
|
||
rejection_reason text, -- closed codes, II.7
|
||
attempts smallint not null default 0,
|
||
locked_at timestamptz, -- lease, II.8
|
||
created_at timestamptz not null default now(),
|
||
updated_at timestamptz not null default now()
|
||
);
|
||
-- "the" avatar of an account = its single clean row (no profile column, no trigger)
|
||
create unique index assets_one_clean_avatar
|
||
on public.assets (owner_id) where kind = 'profile_avatar' and status = 'clean';
|
||
create index assets_claimable on public.assets (created_at) where status = 'uploaded';
|
||
```
|
||
|
||
Privileges: `revoke all` from API roles, `grant select` to `authenticated`,
|
||
RLS `assets_select_own` (`owner_id = auth.uid()`). No INSERT / UPDATE /
|
||
DELETE policy: every write is an RPC (security definer, `search_path = ''`,
|
||
authorisation before lookup, EXECUTE revoked from PUBLIC, granted per role,
|
||
`private.log_event` on sensitive transitions). pgTAP 001's census of
|
||
`private` functions executable by `authenticated` is updated for the two
|
||
new predicates.
|
||
|
||
| RPC | Role | Effect |
|
||
|---|---|---|
|
||
| `begin_avatar_upload(declared_mime, declared_size)` | authenticated | locks the caller's `profiles` row (`for update`, serialises a double click); rate: ≤ 10 rows of the caller created in the last hour, all statuses counted → `rate_limited`; global ceiling: bytes present in the bucket plus declared sizes of pending rows without bytes (as built, STORAGE.md §2.1) < `private.storage_ceiling()` → `storage_full`; caps (type in the list, size ≤ 5 MiB); supersedes any `pending_upload` / `uploaded` row of the caller (→ `deleted`, enqueued for purge); inserts the row with `object_key = profiles/<uid>/<id>/avatar.webp`, `quarantine_key = quarantine/<uid>/<id>/upload`; returns `{asset_id, bucket_id, quarantine_key}` |
|
||
| `confirm_asset_upload(asset_id)` | authenticated | own row, `pending_upload`, a `storage.objects` row exists at `(bucket_id, quarantine_key)` with `metadata->>'size'` ≤ cap → `uploaded`. Kept as an explicit client signal: the worker never has to guess from Storage API internals whether a PUT finished |
|
||
| `remove_avatar()` | authenticated | every `profile_avatar` row of the caller not already `deleted` / `rejected` (clean and in flight) → `deleted`; logs `asset.avatar_removed` |
|
||
| `claim_avatar_jobs(max_jobs)` | service_role | `uploaded` rows with `locked_at is null or locked_at < now() - 10 min` and `attempts < 3`, `for update skip locked`, oldest first → sets `locked_at`, `attempts + 1`, returns the rows; a row at its 3rd expired lease becomes `rejected (processing_error)` instead. VOLATILE (a STABLE function runs read-only through PostgREST) |
|
||
| `complete_avatar_job(asset_id, byte_size, sha256)` | service_role | guard `status = 'uploaded'` (a superseded row is not completed: the worker then deletes what it wrote) → marks the caller's previous `clean` avatar `deleted` (enqueued), sets the row `clean`; one transaction; logs `asset.avatar_set` |
|
||
| `reject_avatar_job(asset_id, reason)` | service_role | guard `uploaded` → `rejected` with a closed code; logs `asset.rejected` |
|
||
| `mark_quarantine_purged(asset_id)` | service_role | nulls `quarantine_key` after the worker deleted the quarantine object of a `clean` row |
|
||
| `claim_purges(max_items)` / `ack_purge(purge_id)` | service_role | the purge queue, II.9 |
|
||
|
||
Why `assets` does not absorb `anatomical_sheet_assets`: that table is
|
||
content-addressed and sheet-scoped (BDR-032/034) with its own confirmed
|
||
upload path; the kinds that join `assets` later (logos, documents, client
|
||
and animal files, report files, imports) extend the `kind` CHECK and add a
|
||
predicate branch at their step. The profile pointer is derived by the
|
||
unique index rather than the pre-announced `profiles.avatar_asset_id`
|
||
column: no cross-table write, no trigger, three fewer tests; a column can be
|
||
added additively when a public profile needs a join (D5).
|
||
|
||
### II.4 Object keys
|
||
|
||
- Final: `profiles/<user_id>/<asset_id>/avatar.webp`. The name is fixed per
|
||
kind; the user's filename never reaches the server.
|
||
- Quarantine: `quarantine/<user_id>/<asset_id>/upload` in the same bucket.
|
||
No API role can read it: the SELECT predicate only matches the
|
||
`object_key` of a `clean` row, the INSERT predicate only the
|
||
`quarantine_key` of the caller's `pending_upload` row.
|
||
- Keys are built by the RPC from uuids; the policies compare whole strings,
|
||
no prefix match, no `storage.foldername()`, so the PGlite shim and the real
|
||
stack evaluate the same expression. The asset id in the key lets the
|
||
sweeper classify a leftover without a row.
|
||
- Sheet assets keep `<sha256>.svg` (shared drawings, immutable versions).
|
||
|
||
### II.5 Upload and processing (avatar)
|
||
|
||
1. Browser: `File.size` ≤ 20 MiB (sanity), `createImageBitmap` decodes it
|
||
(else `unsupported_type`), draw on a canvas with the longest side capped
|
||
at 1024 px, `toBlob('image/jpeg', 0.9)`. This is transport only (phones
|
||
produce 24 MP / 8 MiB photos; a 1024 px JPEG weighs ~150 KB); the server
|
||
re-decodes and re-encodes regardless. The preview shows that blob in the
|
||
circle with `object-fit: cover`. Nothing uploaded yet.
|
||
2. "Enregistrer la photo" → `begin_avatar_upload('image/jpeg', blob.size)`.
|
||
3. `storage.from('avatars').upload(quarantine_key, blob, { upsert: false, contentType: 'image/jpeg' })`
|
||
with the user JWT. The INSERT policy accepts only
|
||
`(bucket_id = 'avatars', name = quarantine_key)` of a `pending_upload`
|
||
row owned by `auth.uid()`; `upsert` would need UPDATE and SELECT rights
|
||
the plan withholds on purpose.
|
||
4. `confirm_asset_upload(asset_id)` → `uploaded`.
|
||
5. Worker loop, fixed 2 s interval: `claim_avatar_jobs(1)` → GET the
|
||
quarantine object (service client, 30 s timeout on every network call).
|
||
6. Decode child: the I/O process spawns `decode.ts` with an empty
|
||
environment (no key, `PATH` only), feeds the bytes on stdin, reads the
|
||
result on stdout, kills it with SIGKILL after 20 s. The child: type from
|
||
magic bytes (JPEG `FF D8 FF`, PNG 8-byte signature, WebP `RIFF….WEBP`),
|
||
`sharp(bytes, { limitInputPixels: 16_777_216 })` with default `failOn`,
|
||
`sharp.cache(false)`, `sharp.concurrency(1)`, `metadata()` (format matches
|
||
the magic; 64 ≤ sides ≤ 4096; the pixel-limit error maps to
|
||
`dimensions`), `.rotate()`, `.resize(512, 512, { fit: 'cover',
|
||
position: 'centre' })`, `.webp({ quality: 82 })`, no `withMetadata()`:
|
||
EXIF, XMP, IPTC and the ICC profile are not copied (colours converted to
|
||
sRGB through the profile first). Output: one 512 × 512 WebP, sha256,
|
||
byte size. Data after a PNG `IEND` is ignored by the decoder and absent
|
||
from the output.
|
||
7. I/O process: PUT the clean bytes at `object_key` (`upsert: true`,
|
||
`contentType: 'image/webp'`; idempotent on retry) → `complete_avatar_job`
|
||
(commit) → DELETE the quarantine object → `mark_quarantine_purged`. A
|
||
refusal → `reject_avatar_job(reason)`; the quarantine object is
|
||
enqueued for purge by the status trigger. A `complete` refused by the
|
||
guard (row superseded) → the worker deletes the clean object it wrote.
|
||
8. Browser polls its own row (1 s, up to 30 s): `clean` → download the
|
||
photo; `rejected` → message from the closed code; still `uploaded` after
|
||
30 s → "Traitement en cours, revenez dans un instant"; on the next visit
|
||
the account page loads any in-flight row and shows the same state. The
|
||
save button is disabled while a row of the caller is in flight; a new
|
||
pick supersedes it.
|
||
|
||
Crash matrix (the retry is idempotent at every point):
|
||
|
||
| Worker dies | Row | Objects | Recovery |
|
||
|---|---|---|---|
|
||
| before PUT clean | `uploaded`, lease set | quarantine | lease expires (10 min) → re-claim |
|
||
| after PUT clean, before complete | same | quarantine + clean | re-claim, PUT again (`upsert`), complete |
|
||
| after complete, before DELETE quarantine | `clean`, `quarantine_key` set | both | sweeper: quarantine leftovers of `clean` rows, then `mark_quarantine_purged` |
|
||
| 3rd lease expired | `rejected (processing_error)` | enqueued | purge queue |
|
||
|
||
### II.6 Read path (owner-only, no signed URL)
|
||
|
||
`storage.from('avatars').download(object_key)` with the user JWT: the
|
||
Storage API evaluates the SELECT policy, the bytes come back, the app keeps
|
||
a `URL.createObjectURL(blob)` in the auth store (revoked on change or sign
|
||
out) and uses it as `<img src>` in `UserBadge`, the account section and, if
|
||
D3 is accepted, the top bar. One download of ~30 KB per session; the
|
||
production CSP already allows `blob:` images, `connect-src` must name the
|
||
Supabase origin (already required by Step 7's API calls).
|
||
|
||
Why not a signed URL: in Step 8 nobody but the owner reads the avatar, and a
|
||
signed URL is a bearer token that keeps serving after `remove_avatar` until
|
||
it expires. Signed URLs arrive with the first kind shared with other users
|
||
(documents, Step 9): issued by the Storage API after the SELECT policy, TTL
|
||
per kind, `download` flag for documents, query strings dropped from the
|
||
proxy access logs.
|
||
|
||
### II.7 Rejection codes (closed set, phrased by i18n)
|
||
|
||
Row codes: `unsupported_type`, `decode_failed`, `dimensions`,
|
||
`processing_error`. Client-side failure kinds (never rows): `too_large`,
|
||
`unsupported_type`, `rate_limited`, `storage_full`, `network`, `unknown`.
|
||
The raw library or supabase-js error never reaches a row or the screen; the
|
||
worker logs an allowlist of fields (asset id, code, duration) with control
|
||
characters escaped (LRN-044).
|
||
|
||
### II.8 Worker runtime and operations
|
||
|
||
- `apps/worker` (`@faunosteo/worker`): TypeScript run by Node 24, deps
|
||
`@supabase/supabase-js` and `sharp` only. No DB driver: every database
|
||
access is an RPC through PostgREST with the service key (one credential,
|
||
one transport; a managed project needs no open database port).
|
||
- Two processes in one container: the keyed I/O loop and the unkeyed decode
|
||
child per job (II.5 step 6). A decoder exploit lands in a process with no
|
||
key, no file access beyond stdin/stdout and a 20 s life; it can reach
|
||
Kong with anonymous rights only, which is what any browser has.
|
||
- Container `faunosteo_worker` (compose service `worker`): non-root,
|
||
`read_only: true` + `tmpfs /tmp`, `mem_limit: 512m`, `cpus: 1`,
|
||
`pids_limit: 64`, no published port, attached to the Supabase compose
|
||
network only (`internal`, Kong reachable, nothing else; a host firewall
|
||
rule would not filter container egress), `restart: unless-stopped`. A
|
||
watchdog exits the process when one loop iteration exceeds 60 s so the
|
||
restart policy fires (Docker never restarts an `unhealthy` container on
|
||
its own). Env file `.env.worker` (`SUPABASE_URL`,
|
||
`SUPABASE_SERVICE_ROLE_KEY`, `WORKER_ID`), `chmod 600`, separate from the
|
||
app `.env`, excluded from any backup copy of env files, readable through
|
||
`docker inspect` by the `docker` group (documented).
|
||
- Fairness: one in-flight avatar per account (begin supersedes), oldest
|
||
first → no account can queue more than one job.
|
||
- Sweeper every 10 min, two sources: (a) the purge queue (row-driven, II.9);
|
||
(b) key-driven: list `quarantine/` of the bucket, parse the asset id,
|
||
delete any object older than 3 h whose row is not `pending_upload` /
|
||
`uploaded` (covers bytes landed through a signed upload URL or a slow PUT
|
||
after the row moved on; the 3 h grace exceeds the 2 h validity of such a
|
||
URL). No operator step, no `--apply`.
|
||
- `sharp` updates: a Renovate / dependabot rule on the worker package,
|
||
not a manual chore.
|
||
- Local dev: `pnpm worker:dev` against the local stack (service key typed
|
||
into the shell as SUPABASE.md §8 already does).
|
||
|
||
### II.9 Purge queue, orphans, replacement, deletion
|
||
|
||
`private.object_purge_queue(id, bucket_id, object_name, not_before,
|
||
enqueued_at)` fed by triggers on `assets`: a transition to `deleted` or
|
||
`rejected` enqueues `object_key` and `quarantine_key`; a row DELETE (account
|
||
cascade) enqueues them too. `not_before` = now + 48 h for the final key of a
|
||
`deleted` row (longer than any backup run, II.10), now + 1 h for quarantine
|
||
keys and rejected uploads. The sweeper calls `claim_purges(max_items)`,
|
||
deletes each object (a missing object is a success), `ack_purge(id)`. The
|
||
only automated deletions are those two sources; a `clean` row's `object_key`
|
||
is never deleted (test).
|
||
|
||
| Event | Row | Object | Cleanup |
|
||
|---|---|---|---|
|
||
| New avatar clean | new `clean`, previous `deleted` (same transaction) | previous object kept 48 h | queue |
|
||
| `remove_avatar()` | clean and in-flight rows `deleted` | kept 48 h | queue |
|
||
| New pick while in flight | superseded row `deleted` | its bytes, if any | queue; a `complete` for it is refused |
|
||
| Upload never confirmed | `pending_upload` superseded or expired by the next begin / after 24 h | maybe present | queue |
|
||
| Bytes landed after the row moved on | none or not pending | present | key-driven sweep after 3 h |
|
||
| Account deleted (Step 26) | rows cascade | enqueued by the DELETE trigger | queue |
|
||
|
||
Access is cut by the row: a `deleted` or `rejected` row matches no read
|
||
predicate, and no URL outlives the policy check in Step 8 (download, not
|
||
signing). Object lifetime is eventual, bounded by the grace periods.
|
||
|
||
Reconciliation (operator, documented in STORAGE.md, not a tool): one SQL
|
||
anti-join between `storage.objects` and `assets` (same database) lists
|
||
objects without a row and rows without an object per bucket; sheet assets
|
||
are checked against `anatomical_sheet_assets`. Reading only; deletions stay
|
||
row-driven.
|
||
|
||
### II.10 Backups and restore on the VPS (self-hosted)
|
||
|
||
- Storage backend `file` on a named volume `faunosteo_supabase_storage`
|
||
(kept across renames, BDR-035), on a filesystem separate from the
|
||
PostgreSQL volume with a quota, and a disk-usage alert: the VPS hosts
|
||
other products, a full disk would take PostgreSQL down with it. The S3
|
||
backend (Scaleway Object Storage) remains the alternative when the volume
|
||
outgrows the disk: copy the files, change the backend env, rows unchanged.
|
||
- Global ceiling: `private.storage_ceiling()` (constant, 10 GiB to start)
|
||
checked by `begin_avatar_upload` on top of the per-account rate.
|
||
- Backup order: database dump first (whole instance, `storage` schema
|
||
included), files second. With the 48 h purge grace, every `clean` row of
|
||
the dump still has its object in the snapshot, provided the snapshot runs
|
||
within 48 h of the dump (the invariant the restore checks: every `clean`
|
||
row has an object).
|
||
- Restore: stop the worker, start the database, restore the dump, restore
|
||
the volume, start the rest, run the reconciliation query, start the
|
||
worker. Files newer than the dump have no row and are invisible to the
|
||
Storage API (it lists from `storage.objects`); the directory listing of
|
||
the volume against `storage.objects` finds them.
|
||
- Tooling (restic to Scaleway Object Storage, encrypted, dump in the same
|
||
run, write-only key with retention pruned from another host, the
|
||
Supabase secrets `JWT_SECRET` / `POSTGRES_PASSWORD` in the encrypted set,
|
||
`.env.worker` never in a plaintext copy) and the restore drill belong to
|
||
the self-hosted deployment runbook (plan Step 27 "backup/restore
|
||
rehearsal"). Step 8 writes the ordering, the invariant and the backend
|
||
choice in `STORAGE.md`; the owner runs every backup or restore command.
|
||
|
||
### II.11 Migration to managed Supabase (runbook, no tool in Step 8)
|
||
|
||
Order: freeze writes (stop the worker, maintenance notice); migrate the
|
||
schema (`supabase db push`, buckets and policies are SQL; the `storage`
|
||
schema itself is never dumped into managed); migrate the data rows
|
||
(`auth.users` through the Supabase auth migration path, then `public`:
|
||
profiles, roles, sheets, `assets`); copy the objects driven by the rows
|
||
(`assets` and `anatomical_sheet_assets`: for every `clean` / confirmed row,
|
||
download, verify sha256, upload at the same key, verify; paginate by keyset,
|
||
PostgREST caps a set at 1000 rows); run the reconciliation query on the
|
||
destination; switch `SUPABASE_URL` and the keys of the app and the worker;
|
||
resume. The service key of a self-hosted stack is a long-lived JWT: revoking
|
||
it means rotating `JWT_SECRET` (every key and session); documented, and a
|
||
reason to prefer managed secret keys once there.
|
||
|
||
### II.12 UI (follows the validated mockup, no redesign)
|
||
|
||
- Mon compte → Informations personnelles: avatar circle (photo, else the
|
||
monogram from `UserBadge`) with "Modifier la photo" under it, as drawn.
|
||
Picking a file shows the local preview in the circle with "Enregistrer la
|
||
photo" / "Annuler"; after saving, the circle shows "Traitement en cours…"
|
||
until the row is `clean`; "Supprimer la photo" appears when a photo
|
||
exists (one confirmation). Errors use `account.avatar.error.<code>`.
|
||
- `UserBadge` (sidebar + drawer): photo when present, monogram otherwise.
|
||
- Top bar (D3): the mockup shows avatar + name with a chevron on desktop and
|
||
the avatar alone on mobile; Step 7 shipped the bar without a user element.
|
||
Proposal: add the avatar (and the name on desktop) as a link to Mon
|
||
compte, no dropdown (its content is undefined). Decision for the owner.
|
||
- Dashboard unchanged (the mockup shows no avatar in the content area).
|
||
- Accepted inputs at the picker: JPEG, PNG, WebP; never SVG for avatars.
|
||
What the browser cannot decode is refused there with `unsupported_type`;
|
||
what it decodes is sent as JPEG and judged again by the worker.
|
||
|
||
---
|
||
|
||
## Part III — The ten security questions
|
||
|
||
1. **Access control**: the `assets` row is the authority. The two policies
|
||
on `storage.objects` are bucket-scoped
|
||
(`bucket_id = 'avatars' and private.can_read_avatar_object(uid, name)`,
|
||
same for upload) and look the key up in `assets` (status, owner, kind).
|
||
A bucket without policy is unreachable by API roles; the census pins one
|
||
bucket literal per policy and the counts.
|
||
2. **No cross-user access**: the read predicate requires a `clean` row with
|
||
`owner_id = auth.uid()` whose `object_key` equals the requested name; the
|
||
path is never trusted, only the row. Listing is denied the same way.
|
||
pgTAP: Bob reads nothing of Alice, cannot upload at her key, cannot
|
||
confirm her row; real-stack e2e: Bob's `download` of Alice's key → 400.
|
||
3. **Keys**: server-built from uuids by the RPC, canonical file name per
|
||
kind, no user filename anywhere, whole-string comparison in the policies.
|
||
4. **Signed URLs**: not used in Step 8. An owner-only avatar is read by
|
||
`download` under RLS and shown through a blob URL, so no bearer token
|
||
exists and a removed avatar stops being readable at the row flip. When a
|
||
shared kind arrives (documents, Step 9): Storage API signing after the
|
||
SELECT policy, TTL per kind (60 s and `download` for documents), query
|
||
strings dropped from access logs, the row cuts signing and the purge cuts
|
||
serving.
|
||
5. **Quarantine**: upload lands under `quarantine/` with the row
|
||
`pending_upload` → `uploaded`; no API role can read it; only the worker
|
||
reads it, writes the clean bytes under the final key, commits the row,
|
||
deletes the quarantine object. Every transition is an RPC; the browser
|
||
cannot set `clean`. Bytes landed late (signed upload URL, slow PUT) are
|
||
swept by key after 3 h without an operator.
|
||
6. **Dangerous SVG**: avatars never accept SVG (the browser decodes raster
|
||
only, the worker checks magic bytes, the bucket lists three raster types).
|
||
For sheet drawings the allowlist sanitiser exists and is tested on
|
||
hostile fixtures (`packages/pdf`); it becomes the worker's SVG processor
|
||
at Step 13 with its producer, after a pre-parse byte and tag-count cap
|
||
(the DOM is built before the element cap counts) and a bounded removed
|
||
list (D6).
|
||
7. **Images without metadata**: the worker never stores the uploaded bytes.
|
||
The browser already re-encoded through a canvas (no EXIF survives), and
|
||
the worker decodes again (type from magic bytes, 16.7 MP pixel cap, sides
|
||
≤ 4096), applies the EXIF orientation if any, resizes and re-encodes to
|
||
WebP without `withMetadata()`: no EXIF, XMP, IPTC or ICC chunk in the
|
||
output. Tests: an EXIF-laden JPEG fed directly to the worker yields an
|
||
output without `EXIF` / `XMP ` / `ICCP` chunks, upright.
|
||
8. **Orphans**: replacement, removal, supersede and rejection flip the row
|
||
in one transaction and enqueue the keys; the sweeper serves the queue
|
||
after the grace period and sweeps quarantine leftovers by key; a `clean`
|
||
row's object is never deleted; the reconciliation query reports the rest.
|
||
9. **Backups and restore**: file backend on a named, quota-limited volume;
|
||
dump first, files second, 48 h purge grace so every `clean` row of a dump
|
||
has its object; restore in the documented order followed by the
|
||
reconciliation query; tooling and drill in the deployment runbook.
|
||
10. **Managed migration**: same migrations; rows then objects driven by the
|
||
rows with hash verification; keys and URLs switched by env; write freeze
|
||
during the copy.
|
||
|
||
Service role: only `.env.worker` and operator commands hold it; the decode
|
||
child never sees it; `env.js` is generated from `APP_*` variables only.
|
||
|
||
---
|
||
|
||
## Part IV — Scope, tests, risks, decisions
|
||
|
||
### IV.1 Exact scope of the implementation
|
||
|
||
In:
|
||
|
||
- Migration `20261012100000_files_foundation.sql`: `asset_status`, `assets`
|
||
(CHECKs, indexes), seven bucket rows (the existing one untouched),
|
||
`private.can_read_avatar_object` / `can_upload_avatar_object`,
|
||
`storage.objects` SELECT + INSERT policies for `avatars`,
|
||
`private.object_purge_queue` + triggers, `private.storage_ceiling()`, the
|
||
nine RPCs of II.3, explicit EXECUTE revokes. pgTAP `010_files_foundation.sql`
|
||
(allowed and denied cases, policy census), pgTAP 001 census update, shim
|
||
extension if a column is missing, `database.types.ts` regenerated.
|
||
- `apps/worker`: I/O loop, decode child, sweeper (queue + key-driven),
|
||
watchdog, Dockerfile (`node:24-alpine`, non-root), compose service,
|
||
`.env.worker.example`, vitest on fixtures (with wall-time and RSS
|
||
assertions on the pathological ones), an integration test against the
|
||
local stack run by the CI `supabase` job.
|
||
- `packages/supabase/src/assets.ts`: `beginAvatarUpload`,
|
||
`uploadToQuarantine`, `confirmAssetUpload`, `removeAvatar`,
|
||
`fetchOwnAvatar` (clean or in flight), `downloadAvatar`; closed
|
||
`AssetFailure` kinds.
|
||
- `apps/app`: avatar block in `PersonalInfoForm` (picker, canvas transport,
|
||
preview, save, processing, delete), `UserBadge` photo, top bar avatar if
|
||
D3 = yes, auth store blob URL, i18n keys FR/EN, component and store tests.
|
||
- Docs: `docs/development/STORAGE.md` (model, keys, statuses, worker
|
||
operations, purge and grace rules, reconciliation query, backup ordering
|
||
and invariant, managed migration runbook), DEPLOY.md (worker service, env
|
||
file, network, CSP `connect-src`), SUPABASE.md (worker in dev), CLAUDE.md,
|
||
CHANGELOG; plan note (worker early, SVG processor at Step 13).
|
||
- CI: worker typecheck / lint / test / build, `docker build` of the worker
|
||
image, worker integration test in the `supabase` job.
|
||
|
||
Out (named so nothing is implied): SVG processor and user SVG upload path
|
||
(Steps 13 / 17), logos, documents, client / animal / report / import flows
|
||
(their steps), signed URLs, image transformations, public profile photo,
|
||
`profiles.avatar_asset_id`, the production deployment of the self-hosted
|
||
stack, backup tooling and restore drill, migration and reconcile tools,
|
||
account erasure (Step 26), Realtime, a user menu dropdown.
|
||
|
||
### IV.2 Test scenarios
|
||
|
||
pgTAP (PGlite + real stack), every case with its denial twin:
|
||
|
||
- buckets: eight rows, all private; `avatars` cap and types; census: one
|
||
bucket literal per policy, 2 / 1 / 0 per bucket, total pinned;
|
||
`authenticated` cannot insert nor select in a closed bucket.
|
||
- `begin_avatar_upload`: keys of the right shape; wrong type, oversize →
|
||
denied; 11th begin within the hour fails even after supersedes; ceiling
|
||
reached → `storage_full`; a begin while a row is `pending_upload` or
|
||
`uploaded` supersedes it (`deleted`, queue row written); anon denied.
|
||
- upload policy: Alice inserts at her quarantine key; Bob at her key →
|
||
42501; Alice at a final key, at the key of a `clean` row, at a superseded
|
||
key, at an unknown key → denied; no UPDATE / DELETE policy exists.
|
||
- confirm: own pending row with the object present → `uploaded`; without
|
||
the object → error; Bob confirming Alice's row → denied; twice → error.
|
||
- read policy: Alice reads her `clean` object only; nothing while
|
||
`pending_upload` / `uploaded` / `rejected` / `deleted`; the quarantine
|
||
object never; Bob nothing; anon nothing.
|
||
- worker RPCs: `authenticated` cannot execute any; claim leases oldest
|
||
first and skips a live lease; an expired lease is re-claimed; the third
|
||
expiry rejects with `processing_error`; complete marks the previous clean
|
||
avatar `deleted` and enqueues it atomically; complete on a superseded row
|
||
is refused; reject stores a closed code; the queue holds the right keys
|
||
with the right `not_before`; a `clean` row's `object_key` never enters
|
||
the queue.
|
||
- `remove_avatar`: clean and in-flight rows → `deleted`; a later complete
|
||
of the in-flight row is refused (the photo does not reappear).
|
||
- cascade: deleting a profile enqueues its keys (trigger).
|
||
- audit: `asset.avatar_set`, `asset.avatar_removed`, `asset.rejected` rows.
|
||
|
||
Worker (vitest, fixtures under `apps/worker/tests/fixtures`):
|
||
|
||
- decode child: JPEG / PNG / WebP accepted; PNG bytes under a `.jpg` name
|
||
accepted as PNG (the name is never seen); GIF, BMP, SVG, PDF, HTML →
|
||
`unsupported_type`; truncated → `decode_failed`; 32 px → `dimensions`;
|
||
header-declared 20 000 × 20 000 and a 5000 × 5000 → `dimensions` through
|
||
the pixel limit; EXIF-laden rotated JPEG → upright output without EXIF /
|
||
XMP / ICCP chunks; PNG with a trailer after IEND → accepted, trailer
|
||
absent from the output; output always 512 × 512 WebP under a size bound;
|
||
pathological-but-within-cap fixtures (progressive JPEG with many scans,
|
||
large ICC) finish under the deadline with RSS asserted; a child that
|
||
sleeps is killed at 20 s → `processing_error`.
|
||
- I/O loop: claim → decode → PUT → complete → DELETE sequence; complete
|
||
refused → the clean object is deleted; Storage error → row left for the
|
||
lease; the child never receives the environment; sweeper serves the queue
|
||
and the key-driven list only, never a `clean` `object_key`.
|
||
|
||
Integration (local stack, CI `supabase` job): Alice's full flow with the
|
||
service key for the worker side → row `clean`, object at the final key, no
|
||
quarantine object; a second upload replaces, the first object disappears
|
||
once its `not_before` is reached (clock-shifted in the test); `remove_avatar`
|
||
then sweep → bucket empty; a signed upload URL obtained during
|
||
`pending_upload` and used after supersede → object swept by key.
|
||
|
||
Real-stack e2e (scratchpad script, like Step 7): Alice `download` 200, Bob
|
||
`download` of Alice's key 400, anon 400, Alice's quarantine key 400, a
|
||
direct GET of the object path without JWT 400.
|
||
|
||
App (vitest + happy-dom): picker refuses a file the browser cannot decode
|
||
with the phrased error; preview appears before any request; save calls
|
||
begin → upload → confirm in order with a JPEG blob; processing state then
|
||
photo on `clean`; rejection message per code; delete flow; in-flight row
|
||
shown on mount; save disabled while in flight; `UserBadge` shows the photo
|
||
when the store has a blob URL, the monogram otherwise; i18n parity.
|
||
|
||
Manual validation (owner): upload, replace, delete from Mon compte on
|
||
desktop and mobile with a phone photo; photo visible in the sidebar, the
|
||
drawer and the top bar; open the object path in a private window (400);
|
||
upload a renamed PNG (accepted) and an HTML file (refused with a clear
|
||
message); sign in as another seeded user and check nothing of the first
|
||
user shows.
|
||
|
||
### IV.3 Risks
|
||
|
||
- **Decoder vulnerabilities** (libvips / libwebp history): confined to a
|
||
keyless child with a 20 s life; the keyed process never decodes; `sharp`
|
||
updates automated. Residual: a child exploit reaches Kong with anonymous
|
||
rights, which any browser has.
|
||
- **Worker down or stuck**: uploads stay `uploaded`; the UI says so; the
|
||
watchdog exits on a hung loop and the restart policy restarts the
|
||
container; a stuck decode is killed by the deadline; an account can
|
||
always supersede. Monitoring and alerting are Step 27.
|
||
- **Disk**: separate quota-limited filesystem, global ceiling, per-account
|
||
rate, 48 h grace. Residual: a burst of accounts within the ceiling fills
|
||
the quota, not the host.
|
||
- **CSP in production**: `connect-src` must include the Supabase origin
|
||
(already needed by Step 7); documented in DEPLOY.md.
|
||
- **PGlite shim vs real `storage` schema**: policies use only `bucket_id`
|
||
and `name`; pgTAP never deletes storage objects (LRN-041).
|
||
- **Storage API behaviour relied on**: INSERT policy on direct upload,
|
||
SELECT policy on `download`, signed upload URL semantics; the integration
|
||
test on the CI stack pins them on the pinned image.
|
||
- **Service key handling**: a second env file, `chmod 600`, excluded from
|
||
env backups; rotation = `JWT_SECRET` rotation on self-hosted (documented).
|
||
- **Colour**: ICC profiles are converted to sRGB then dropped; only
|
||
out-of-gamut colours clip. Accepted for an avatar.
|
||
- **Plan deviations**: worker at Step 8 instead of 19; SVG processor at
|
||
Step 13; avatar pointer derived instead of a profile column. Recorded.
|
||
|
||
### IV.4 Decisions for the owner
|
||
|
||
| Id | Question | Recommendation |
|
||
|---|---|---|
|
||
| D1 | Server-side processing runtime | Containerised worker `faunosteo_worker`, keyless decode child; no Edge Functions (no native decoder, WASM budget on managed, one more container on self-hosted anyway) |
|
||
| D2 | Sheet bucket name | Keep `anatomical-sheet-assets`, record the deviation from the announced `anatomical-sheets` |
|
||
| D3 | Top bar avatar | Add avatar (+ name on desktop) as a link to Mon compte, no dropdown; or sidebar / drawer / account only |
|
||
| D4 | Avatar pipeline | Browser: decode, canvas ≤ 1024 px, JPEG transport. Server: magic check, ≤ 16.7 MP, one 512 × 512 WebP, centre crop, no metadata. Inputs JPEG / PNG / WebP ≤ 5 MiB at upload |
|
||
| D5 | Avatar pointer | Derived from the single `clean` row (unique index), no `profiles.avatar_asset_id` yet |
|
||
| D6 | SVG processor | Deferred to Step 13 with its producer; the sanitiser and its hostile fixtures already exist in `packages/pdf` |
|
||
| D7 | Read path | `download` under RLS + blob URL (no signed URL, no bearer) for the owner-only avatar |
|
||
| D8 | Processing feedback | Browser polling of the own row (1 s, 30 s cap), worker at a fixed 2 s; Realtime stays off |
|
||
| D9 | Self-hosted Storage backend | File backend on a named, quota-limited volume; dump-first backups with 48 h purge grace; tooling at the deployment step |
|