Files
faunosteo/docs/development/STORAGE.md
T
bchanot cd2dbb3b79 fix(supabase): security pass on the files foundation — ceiling counts live bytes only, confirm checks the account, null sha256 refused, browser RPCs one role; worker: cleartext only on loopback or a service name, no env proxy, key ownership cross-check, no library text in logs
Security audit (PASS, 2 MEDIUM + LOW): the global ceiling summed every
non-deleted row, so rejected uploads ratcheted it toward storage_full with no
disk used → in-flight rows by declared size, clean rows by byte_size, nothing
else (pgTAP: rejected rows do not count, in-flight rows do); confirm refuses a
deactivated account like begin; complete_avatar_job refuses a null sha256;
begin / confirm / remove granted to authenticated only. Worker: SUPABASE_URL
over http:// accepted only for loopback or a single-label Docker service
name; ureq proxy(None); the job's keys must embed its owner and asset ids;
backend_failure / claim_failed / sweep_failure lines carry the closed kind and
the HTTP status, never the library's error text. e2e: uuid shape checked
before the psql query. Contract revision 2 (oracle fixes).
2026-10-13 19:49:58 +02:00

331 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Files foundation: Supabase Storage, the `assets` table and the Rust worker
Status: Step 8 / Rust R1, 2026-10-13, branch `feature/step-08-files-foundation`.
Design origin: `STEP8_FILES_ANALYSIS.md` (D1 to D9 validated by the owner, D1
amended to the Rust worker of `RUST_ARCHITECTURE_REVIEW.md` Part IV). This
file is the implementation contract: table, keys, RPCs, policies, worker
behaviour and client flow are what the code and the tests pin.
First consumer: the profile photo (avatar). Every other bucket exists, closed.
---
## 1. Model
### 1.1 Statuses
```text
pending_upload ─(confirm_asset_upload)─► uploaded ─(complete_avatar_job)─► clean
│ │
│ begin supersedes / 24 h expiry ├─(reject_avatar_job, 3rd expired lease)─► rejected
▼ ▼
deleted ◄──────(remove_avatar, replacement by a newer clean row, begin supersedes)
```
`deleted` and `rejected` are terminal: the row stays (history, purge
bookkeeping) and matches no read or upload predicate.
### 1.2 `public.assets`
| Column | Type | Rule |
|---|---|---|
| `id` | uuid pk | |
| `owner_id` | uuid → `profiles.id` on delete cascade | |
| `kind` | text | `'profile_avatar'` only in Step 8 (CHECK) |
| `bucket_id` | text | `'avatars'` for that kind (CHECK) |
| `object_key` | text unique | final key, server-built: `profiles/<owner_id>/<id>/avatar.webp` |
| `quarantine_key` | text | `quarantine/<owner_id>/<id>/upload`; null once the worker confirmed the quarantine object is gone |
| `status` | `public.asset_status` | enum above |
| `declared_mime` | text | `image/jpeg`, `image/png`, `image/webp` (CHECK) |
| `declared_size` | integer | 1 … 5 242 880 (CHECK), the transport size announced by the browser |
| `byte_size`, `sha256` | integer, text | the clean output, set by `complete_avatar_job` |
| `rejection_reason` | text | closed set `unsupported_type`, `decode_failed`, `dimensions`, `processing_error`; non-null iff `rejected` (CHECK) |
| `attempts` | smallint | leases taken; the 3rd expired lease rejects with `processing_error` |
| `locked_at`, `lock_token` | timestamptz, uuid | the lease: set by `claim_avatar_jobs`, required by `complete` / `reject`, 10 min validity |
| `created_at`, `updated_at` | timestamptz | trigger `set_updated_at` |
Indexes: unique partial `assets_one_clean_avatar (owner_id) where kind =
'profile_avatar' and status = 'clean'` ("the" avatar of an account is its
single clean row: no profile column, D5); `assets_claimable (created_at)
where status = 'uploaded'`; `assets_owner_idx (owner_id, created_at desc)`.
Privileges: `revoke all` from `public`, `anon`, `authenticated`; `grant
select (id, owner_id, kind, bucket_id, object_key, status, declared_mime,
declared_size, byte_size, sha256, rejection_reason, created_at, updated_at)`
to `authenticated` (the lease columns and `quarantine_key` stay server-side;
a client must list its columns, `select=*` is refused). RLS
`assets_select_own` (`owner_id = auth.uid()`). No INSERT / UPDATE / DELETE
policy: every write is an RPC.
### 1.3 Buckets (all private)
| Bucket | Step 8 state |
|---|---|
| `avatars` | open: SELECT + INSERT policies through `assets`, 5 MiB, `image/jpeg`, `image/png`, `image/webp` |
| `logos`, `professional-documents`, `client-files`, `animal-files`, `report-files`, `imports` | closed: row exists, `public = false`, no policy names it, no cap or MIME list yet (their step sets them) |
| `anatomical-sheet-assets` | untouched (migration `20261009100000`, read through the sheet) |
The migration upserts the rows (`on conflict (id) do update set public =
false`) so it converges on the local stack, the VPS and a managed project.
### 1.4 Keys
- Built by `begin_avatar_upload` from uuids; the user's filename never
reaches the server; one canonical name per kind (`avatar.webp`).
- Policies compare whole strings (`name = quarantine_key`, `name =
object_key`): no prefix match, no `storage.foldername()`, so PGlite and the
real stack evaluate the same expression.
- The worker validates every key it receives from the database before
putting it in a URL: exactly four segments, `profiles|quarantine`, two
uuids, `avatar.webp|upload`.
---
## 2. RPCs
Every function is `security definer`, `set search_path = ''`, authorisation
before lookup, EXECUTE revoked from `public` / `anon` / `authenticated` and
granted to the one role that calls it. Business refusals raise SQLSTATE
`P0001` with the closed code as the message (PostgREST answers 400 with
`{"code":"P0001","message":"<code>"}`); authorisation refusals raise
`42501` (403).
### 2.1 Browser (`authenticated`)
| RPC | Effect | Refusals (message) |
|---|---|---|
| `begin_avatar_upload(declared_mime text, declared_size integer) returns jsonb` | locks the caller's `profiles` row (`for update`); checks the type and the size; rate: at most 10 rows created by the caller in the last hour, every status counted; global ceiling: the bytes that can exist (`declared_size` of in-flight rows, `byte_size` of clean rows; rejected and deleted rows never count) below `private.storage_ceiling()` (10 GiB); supersedes the caller's `pending_upload` / `uploaded` rows (→ `deleted`); inserts the row; returns `{"asset_id", "bucket_id", "quarantine_key"}` | `unsupported_type`, `too_large`, `rate_limited`, `storage_full` |
| `confirm_asset_upload(asset_id uuid) returns void` | active account; own `pending_upload` row with a `storage.objects` row at `(bucket_id, quarantine_key)` whose `metadata->>'size'` is within the cap → `uploaded` | `not_found` (unknown or not own), `invalid_state` (not pending), `upload_missing` (no object yet), `too_large` (object over the cap, a guard the bucket cap already enforces: the row stays pending and expires after 24 h) |
| `remove_avatar() returns integer` | every `profile_avatar` row of the caller in `pending_upload` / `uploaded` / `clean` → `deleted`; returns the count; logs `asset.avatar_removed` | none (0 rows is fine) |
`auth.uid() is null` → `42501` on all three.
### 2.2 Worker (`service_role`)
| RPC | Effect |
|---|---|
| `claim_avatar_jobs(max_jobs integer) returns table (asset_id, owner_id, bucket_id, object_key, quarantine_key, lock_token, attempts)` | first rejects (`processing_error`) every `uploaded` row whose lease expired with `attempts >= 3`; then claims `uploaded` rows with `locked_at is null or locked_at < now() - 10 min` and `attempts < 3`, oldest first, `for update skip locked`, at most `max_jobs` (capped at 10): `locked_at = now()`, `attempts + 1`, fresh `lock_token`. VOLATILE (a STABLE function would run read-only through PostgREST) |
| `complete_avatar_job(asset_id uuid, lock_token uuid, byte_size integer, sha256 text) returns void` | guard: `status = 'uploaded'` and the token matches, else `invalid_state`. In one transaction: the owner's previous `clean` avatar → `deleted` (trigger enqueues its object 48 h later), the row → `clean` with `byte_size` / `sha256`, lease cleared; logs `asset.avatar_set` (actor null = worker, target = owner) |
| `reject_avatar_job(asset_id uuid, lock_token uuid, reason text) returns void` | same guard; `reason` in the closed set else `invalid_reason`; → `rejected`; logs `asset.rejected` with the reason |
| `mark_quarantine_purged(asset_id uuid) returns void` | `clean` row → `quarantine_key = null` (after the worker deleted the quarantine object); other statuses: no-op |
| `claim_purges(max_items integer) returns table (purge_id, bucket_id, object_name)` | queue rows with `not_before <= now()` and no live lease (`locked_at` null or older than 10 min), `for update skip locked`, at most `max_items` (capped at 100); sets `locked_at` |
| `ack_purge(purge_id uuid) returns void` | deletes the queue row (a missing object counted as success by the worker) |
| `claim_quarantine_leftovers(older_than_seconds integer, max_items integer) returns table (bucket_id, object_name, asset_id)` | `storage.objects` under `quarantine/` in `avatars`, created before `now() - older_than`, whose row is not `pending_upload` / `uploaded` (anti-join on `quarantine_key`); `asset_id` is the matching row when one exists (the worker calls `mark_quarantine_purged` for a `clean` one) |
| `expire_pending_uploads(max_age_seconds integer) returns integer` | `pending_upload` rows older than the age → `deleted` (trigger enqueues their keys); returns the count |
The three stale-task guards together: a superseded or removed row is
`deleted` (never `uploaded`, so no `complete`); a `complete` from an expired
lease carries a stale `lock_token` and is refused even while the row is
still `uploaded`; the purge of an old key never touches a newer row's
object (keys embed the asset id).
### 2.3 `storage.objects` policies (bucket `avatars`)
| Policy | Command | Expression |
|---|---|---|
| `avatars_read` | SELECT, `authenticated` | `bucket_id = 'avatars' and private.can_read_avatar_object((select auth.uid()), name)` = a `clean` row of the caller with `object_key = name` |
| `avatars_upload` | INSERT, `authenticated` | `bucket_id = 'avatars' and private.can_upload_avatar_object((select auth.uid()), name)` = a `pending_upload` row of the caller with `quarantine_key = name` |
No UPDATE / DELETE policy exists for any API role. The pgTAP census asserts
that every policy on `storage.objects` pins exactly one bucket literal and
counts them: 2 on `avatars`, 1 on `anatomical-sheet-assets`, 0 elsewhere,
total 3.
### 2.4 Purge queue (`private.object_purge_queue`)
`(id, bucket_id, object_name, not_before, enqueued_at, locked_at, attempts)`,
unique on `(bucket_id, object_name)` (a second enqueue keeps the earliest
`not_before`). Fed by triggers on `assets`:
| Transition | `object_key` | `quarantine_key` |
|---|---|---|
| → `deleted` | `now() + 48 h` | `now() + 1 h` |
| → `rejected` | `now() + 1 h` (a clean object may exist after a PUT before a 3rd expired lease) | `now() + 1 h` |
| row DELETE (account cascade) | `now() + 48 h` | `now() + 1 h` |
A `clean` row's `object_key` is never enqueued (test). The 48 h grace is the
backup invariant of §5.
---
## 3. The worker (`crates/worker`, container `faunosteo_worker`)
### 3.1 Process model
One static musl binary, two roles in one container:
- `faunosteo-worker run`: the keyed parent. Holds `SUPABASE_SERVICE_ROLE_KEY`,
talks to PostgREST and the Storage API over HTTP, never decodes a pixel.
- `faunosteo-worker decode`: the child spawned per job by the parent with an
empty environment and piped stdin/stdout (R0 sandbox unchanged: `Threads:
1`, no core dump, `RLIMIT_AS` 512 MiB, `RLIMIT_CPU` 10 s / 12 s, seccomp
allowlist, 5 MiB input, 512 KiB output, 20 s deadline). It sees no key,
no file, no network.
The parent checks the child's output at container level only
(`webp_check`: RIFF structure, chunk allowlist, 512 × 512 canvas, no
metadata chunk), hashes it and uploads it.
### 3.2 Configuration (environment of the parent only)
| Variable | Required | Default | Meaning |
|---|---|---|---|
| `SUPABASE_URL` | yes | | origin of the stack (`http://supabase_kong_faunosteo:8000` from the Supabase compose network, `http://127.0.0.1:54321` natively). Cleartext `http://` is accepted only for loopback or a single-label host (a Docker service name); any other host must be `https://`, the service key travels in the headers |
| `SUPABASE_SERVICE_ROLE_KEY` | yes | | the service key; never printed; the JWT `exp` claim, when the key is a JWT, is logged at start and warned about 30 days ahead |
| `WORKER_ID` | no | `worker-<pid>` | tag of every log line |
| `WORKER_POLL_SECONDS` | no | `2` | idle interval between claims |
| `WORKER_SWEEP_SECONDS` | no | `600` | interval of the sweeper |
| `WORKER_QUARANTINE_GRACE_SECONDS` | no | `10800` | age before a quarantine leftover is deleted (above the 2 h validity of a signed upload URL) |
| `WORKER_PENDING_MAX_AGE_SECONDS` | no | `86400` | age before a never-confirmed `pending_upload` row is expired |
| `WORKER_ONCE` | no | unset | `1` = one claim pass and one sweep, then exit (tests) |
`faunosteo-worker run --check` prints the resolved configuration without
any secret value. Exit 0 when the worker could start (libwebp ≥ 1.6.0, both
required variables set, URL parseable) or when no worker variable is set at
all (an unconfigured image is a valid artifact check, `scripts/check-all.sh`);
a partial or invalid configuration exits 2 naming the variable, a too-old
libwebp exits 18.
### 3.3 HTTP endpoints used (service key: `apikey` and `Authorization: Bearer`)
| Call | Request |
|---|---|
| RPC | `POST <url>/rest/v1/rpc/<name>`, JSON body, `Content-Type: application/json`, `Prefer: return=representation` |
| download quarantine | `GET <url>/storage/v1/object/<bucket>/<key>`, body read with a 5 MiB + 1 cap; a body over the cap (impossible through the bucket cap and `confirm_asset_upload`, kept as a guard) is a content refusal: the row is rejected `decode_failed` and the worker logs `download_overflow` |
| upload clean | `POST <url>/storage/v1/object/<bucket>/<key>`, `Content-Type: image/webp`, `x-upsert: true` (idempotent on retry) |
| delete object | `DELETE <url>/storage/v1/object/<bucket>/<key>` (a missing object counts as done: HTTP 404, or the HTTP 400 with `statusCode` 404 this Storage API version answers) |
Every call: 30 s global timeout, 5 s connect timeout, no redirect follow, no
proxy (the environment's `HTTP_PROXY` is ignored on purpose),
`http_status_as_error(false)` so each status is classified by the caller.
Log lines carry the closed error kind and the HTTP status, never the library's
error text.
Keys are validated (§1.4) before any URL is built.
### 3.4 One job
```text
claim_avatar_jobs(1) stage claim deadline 35 s
GET quarantine stage download 35 s 404 → reject decode_failed (bytes never landed; the row said uploaded)
spawn decode child, feed bytes stage decode 25 s
Produced(webp) → POST clean (upsert) → complete_avatar_job(asset, token, size, sha256)
stage upload 35 s, stage commit 35 s
→ DELETE quarantine → mark_quarantine_purged (best effort: the sweeper finishes it)
Rejected{code} → reject_avatar_job(asset, token, code) code ∈ unsupported_type | decode_failed | dimensions | resource_limit→decode_failed
Signalled / Timeout / OutputOverflow / Failed
→ reject_avatar_job(asset, token, 'processing_error') (logged with signal number / detail)
Broken{code} → log level=error event=child_broken, leave the lease (the row is retried, 3rd lease rejects), stop the loop with exit code 20
complete refused (invalid_state: superseded, removed, stale lease)
→ log, delete nothing: the purge queue owns every object of a terminal row
```
Every network failure in a stage leaves the row to its lease (re-claimed
after 10 min, rejected `processing_error` on the 3rd expiry). A stage that
exceeds its deadline makes the watchdog thread exit the process (code 21);
Docker's restart policy brings it back.
### 3.5 Sweeper (every `WORKER_SWEEP_SECONDS`, and once at start)
1. `expire_pending_uploads(WORKER_PENDING_MAX_AGE_SECONDS)`.
2. `claim_purges(100)` → DELETE each object (404 = done) → `ack_purge`.
3. `claim_quarantine_leftovers(WORKER_QUARANTINE_GRACE_SECONDS, 100)` →
DELETE each object → `mark_quarantine_purged(asset_id)` when present.
Those are the only deletions the worker ever makes: a key it did not get
from the queue or the leftover query is never deleted.
### 3.6 Logs
One JSON object per line on stdout, fields from an allowlist only: `ts`,
`level`, `event`, `worker_id`, `asset_id`, `stage`, `code`, `signal`,
`http_status`, `duration_ms`, `count`, `detail` (worker-authored text only).
The child's stderr is child-controlled: it is logged under `child_stderr`
JSON-escaped and truncated to 200 bytes, never raw. No key, no URL with a
token, no user content.
---
## 4. Client flow (browser, anon key + user JWT)
1. Pick: `<input type="file" accept="image/jpeg,image/png,image/webp">`.
The first bytes are sniffed (`FF D8 FF`, PNG signature, `RIFF….WEBP`);
anything else, SVG included, is refused with `unsupported_type` before
any decode. `File.size` over 20 MiB → `too_large`.
2. Transport: `createImageBitmap` → canvas with the longest side at 1024 px
→ `toBlob('image/jpeg', 0.9)`; over 5 MiB after that → `too_large`.
The preview shows that blob. Nothing uploaded yet.
3. "Enregistrer la photo": `begin_avatar_upload('image/jpeg', blob.size)` →
`storage.from('avatars').upload(quarantine_key, blob, { upsert: false,
contentType: 'image/jpeg' })` → `confirm_asset_upload(asset_id)`.
4. Poll the own row every 1 s, 30 s at most: `clean` → download
`object_key` (`storage.from('avatars').download`), keep a blob URL in the
auth store; `rejected` → message per code; still `uploaded` after 30 s →
"processing, come back later" (the next mount polls again).
5. "Supprimer la photo": confirmation dialog → `remove_avatar()` → monogram.
Closed failure kinds the UI phrases (`AssetFailure.kind`): `too_large`,
`unsupported_type`, `rate_limited`, `storage_full`, `upload_missing`,
`invalid_state`, `not_found`, `network`, `unknown`; rejection codes of a row:
`unsupported_type`, `decode_failed`, `dimensions`, `processing_error`.
No signed URL, no bearer token: the download runs under the SELECT policy
and stops at the row flip (D7). The production CSP already allows `blob:`
images; `connect-src` must name the Supabase origin (DEPLOY.md).
---
## 5. Operations
### 5.1 Reconciliation (read-only, operator)
```sql
-- objects without a row (any status) and rows whose object is missing
select o.bucket_id, o.name from storage.objects o
where o.bucket_id = 'avatars'
and not exists (select 1 from public.assets a where a.object_key = o.name or a.quarantine_key = o.name);
select a.id, a.object_key from public.assets a
where a.status = 'clean'
and not exists (select 1 from storage.objects o where o.bucket_id = a.bucket_id and o.name = a.object_key);
```
Deletions stay row-driven (queue and leftover sweep); the query only reports.
### 5.2 Backups (self-hosted, file backend)
Order: database dump first (whole instance, `storage` schema included),
files second, within 48 h of each other. With the 48 h purge grace every
`clean` row of the dump still has its object in the file snapshot (the
invariant the restore checks with the second query above). Restore: stop
the worker, restore the database, restore the volume, run the
reconciliation, start the worker. Tooling (restic, retention, drill) is the
deployment runbook's (plan Step 27).
### 5.3 Managed migration
Same migrations (`supabase db push`); rows (`auth.users`, `public`), then the
objects driven by the rows (`clean` rows and confirmed sheet assets:
download, verify sha256, upload at the same key, verify), write freeze while
copying (worker stopped), reconciliation on the destination, switch
`SUPABASE_URL` and the keys of the app and the worker.
---
## 6. Development
```bash
pnpm db:start # local stack (migrations + seed)
pnpm worker:image # faunosteo_worker:dev
docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges:true \
--network supabase_network_faunosteo \
-e SUPABASE_URL=http://supabase_kong_faunosteo:8000 \
-e SUPABASE_SERVICE_ROLE_KEY="$(pnpm -s supabase status -o env | sed -n 's/^SERVICE_ROLE_KEY="\(.*\)"/\1/p')" \
faunosteo_worker:dev run
pnpm --filter @faunosteo/supabase test:e2e # real-stack end-to-end (needs the worker above)
pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace
```
`docker compose --profile worker up worker` runs the same container from
`docker-compose.yml` with `.env.worker` (`SUPABASE_URL`,
`SUPABASE_SERVICE_ROLE_KEY`, `WORKER_ID`; `chmod 600`, never in the app
`.env`, never in a backup of env files).