# Files foundation: Supabase Storage, the `assets` table and the Rust worker Status: Step 8 / Rust R1, 2026-10-13, branch `feature/step-08-files-foundation`. Design origin: `STEP8_FILES_ANALYSIS.md` (D1 to D9 validated by the owner, D1 amended to the Rust worker of `RUST_ARCHITECTURE_REVIEW.md` Part IV). This file is the implementation contract: table, keys, RPCs, policies, worker behaviour and client flow are what the code and the tests pin. First consumer: the profile photo (avatar). Every other bucket exists, closed. --- ## 1. Model ### 1.1 Statuses ```text pending_upload ─(confirm_asset_upload)─► uploaded ─(complete_avatar_job)─► clean │ │ │ begin supersedes / 24 h expiry ├─(reject_avatar_job, 3rd expired lease)─► rejected ▼ ▼ deleted ◄──────(remove_avatar, replacement by a newer clean row, begin supersedes) ``` `deleted` and `rejected` are terminal: the row stays (history, purge bookkeeping) and matches no read or upload predicate. ### 1.2 `public.assets` | Column | Type | Rule | |---|---|---| | `id` | uuid pk | | | `owner_id` | uuid → `profiles.id` on delete cascade | | | `kind` | text | `'profile_avatar'` only in Step 8 (CHECK) | | `bucket_id` | text | `'avatars'` for that kind (CHECK) | | `object_key` | text unique | final key, server-built: `profiles///avatar.webp` | | `quarantine_key` | text | `quarantine///upload`; null once the worker confirmed the quarantine object is gone | | `status` | `public.asset_status` | enum above | | `declared_mime` | text | `image/jpeg`, `image/png`, `image/webp` (CHECK) | | `declared_size` | integer | 1 … 5 242 880 (CHECK), the transport size announced by the browser | | `byte_size`, `sha256` | integer, text | the clean output, set by `complete_avatar_job` | | `rejection_reason` | text | closed set `unsupported_type`, `decode_failed`, `dimensions`, `processing_error`; non-null iff `rejected` (CHECK) | | `attempts` | smallint | leases taken; the 3rd expired lease rejects with `processing_error` | | `locked_at`, `lock_token` | timestamptz, uuid | the lease: set by `claim_avatar_jobs`, required by `complete` / `reject`, 10 min validity | | `created_at`, `updated_at` | timestamptz | trigger `set_updated_at` | Indexes: unique partial `assets_one_clean_avatar (owner_id) where kind = 'profile_avatar' and status = 'clean'` ("the" avatar of an account is its single clean row: no profile column, D5); `assets_claimable (created_at) where status = 'uploaded'`; `assets_owner_idx (owner_id, created_at desc)`. Privileges: `revoke all` from `public`, `anon`, `authenticated`; `grant select (id, owner_id, kind, bucket_id, object_key, status, declared_mime, declared_size, byte_size, sha256, rejection_reason, created_at, updated_at)` to `authenticated` (the lease columns and `quarantine_key` stay server-side; a client must list its columns, `select=*` is refused). RLS `assets_select_own` (`owner_id = auth.uid()`). No INSERT / UPDATE / DELETE policy: every write is an RPC. ### 1.3 Buckets (all private) | Bucket | Step 8 state | |---|---| | `avatars` | open: SELECT + INSERT policies through `assets`, 5 MiB, `image/jpeg`, `image/png`, `image/webp` | | `logos`, `professional-documents`, `client-files`, `animal-files`, `report-files`, `imports` | closed: row exists, `public = false`, no policy names it, no cap or MIME list yet (their step sets them) | | `anatomical-sheet-assets` | untouched (migration `20261009100000`, read through the sheet) | The migration upserts the rows (`on conflict (id) do update set public = false`) so it converges on the local stack, the VPS and a managed project. ### 1.4 Keys - Built by `begin_avatar_upload` from uuids; the user's filename never reaches the server; one canonical name per kind (`avatar.webp`). - Policies compare whole strings (`name = quarantine_key`, `name = object_key`): no prefix match, no `storage.foldername()`, so PGlite and the real stack evaluate the same expression. - The worker validates every key it receives from the database before putting it in a URL: exactly four segments, `profiles|quarantine`, two uuids, `avatar.webp|upload`. --- ## 2. RPCs Every function is `security definer`, `set search_path = ''`, authorisation before lookup, EXECUTE revoked from `public` / `anon` / `authenticated` and granted to the one role that calls it. Business refusals raise SQLSTATE `P0001` with the closed code as the message (PostgREST answers 400 with `{"code":"P0001","message":""}`); authorisation refusals raise `42501` (403). ### 2.1 Browser (`authenticated`) | RPC | Effect | Refusals (message) | |---|---|---| | `begin_avatar_upload(declared_mime text, declared_size integer) returns jsonb` | locks the caller's `profiles` row (`for update`); checks the type and the size; rate: at most 10 rows created by the caller in the last hour, every status counted; global ceiling: `sum(declared_size)` of non-deleted rows below `private.storage_ceiling()` (10 GiB); supersedes the caller's `pending_upload` / `uploaded` rows (→ `deleted`); inserts the row; returns `{"asset_id", "bucket_id", "quarantine_key"}` | `unsupported_type`, `too_large`, `rate_limited`, `storage_full` | | `confirm_asset_upload(asset_id uuid) returns void` | own `pending_upload` row with a `storage.objects` row at `(bucket_id, quarantine_key)` whose `metadata->>'size'` is within the cap → `uploaded` | `not_found` (unknown or not own), `invalid_state` (not pending), `upload_missing` (no object yet), `too_large` (object over the cap: the row is deleted) | | `remove_avatar() returns integer` | every `profile_avatar` row of the caller in `pending_upload` / `uploaded` / `clean` → `deleted`; returns the count; logs `asset.avatar_removed` | none (0 rows is fine) | `auth.uid() is null` → `42501` on all three. ### 2.2 Worker (`service_role`) | RPC | Effect | |---|---| | `claim_avatar_jobs(max_jobs integer) returns table (asset_id, owner_id, bucket_id, object_key, quarantine_key, lock_token, attempts)` | first rejects (`processing_error`) every `uploaded` row whose lease expired with `attempts >= 3`; then claims `uploaded` rows with `locked_at is null or locked_at < now() - 10 min` and `attempts < 3`, oldest first, `for update skip locked`, at most `max_jobs` (capped at 10): `locked_at = now()`, `attempts + 1`, fresh `lock_token`. VOLATILE (a STABLE function would run read-only through PostgREST) | | `complete_avatar_job(asset_id uuid, lock_token uuid, byte_size integer, sha256 text) returns void` | guard: `status = 'uploaded'` and the token matches, else `invalid_state`. In one transaction: the owner's previous `clean` avatar → `deleted` (trigger enqueues its object 48 h later), the row → `clean` with `byte_size` / `sha256`, lease cleared; logs `asset.avatar_set` (actor null = worker, target = owner) | | `reject_avatar_job(asset_id uuid, lock_token uuid, reason text) returns void` | same guard; `reason` in the closed set else `invalid_reason`; → `rejected`; logs `asset.rejected` with the reason | | `mark_quarantine_purged(asset_id uuid) returns void` | `clean` row → `quarantine_key = null` (after the worker deleted the quarantine object); other statuses: no-op | | `claim_purges(max_items integer) returns table (purge_id, bucket_id, object_name)` | queue rows with `not_before <= now()` and no live lease (`locked_at` null or older than 10 min), `for update skip locked`, at most `max_items` (capped at 100); sets `locked_at` | | `ack_purge(purge_id uuid) returns void` | deletes the queue row (a missing object counted as success by the worker) | | `claim_quarantine_leftovers(older_than_seconds integer, max_items integer) returns table (bucket_id, object_name, asset_id)` | `storage.objects` under `quarantine/` in `avatars`, created before `now() - older_than`, whose row is not `pending_upload` / `uploaded` (anti-join on `quarantine_key`); `asset_id` is the matching row when one exists (the worker calls `mark_quarantine_purged` for a `clean` one) | | `expire_pending_uploads(max_age_seconds integer) returns integer` | `pending_upload` rows older than the age → `deleted` (trigger enqueues their keys); returns the count | The three stale-task guards together: a superseded or removed row is `deleted` (never `uploaded`, so no `complete`); a `complete` from an expired lease carries a stale `lock_token` and is refused even while the row is still `uploaded`; the purge of an old key never touches a newer row's object (keys embed the asset id). ### 2.3 `storage.objects` policies (bucket `avatars`) | Policy | Command | Expression | |---|---|---| | `avatars_read` | SELECT, `authenticated` | `bucket_id = 'avatars' and private.can_read_avatar_object((select auth.uid()), name)` = a `clean` row of the caller with `object_key = name` | | `avatars_upload` | INSERT, `authenticated` | `bucket_id = 'avatars' and private.can_upload_avatar_object((select auth.uid()), name)` = a `pending_upload` row of the caller with `quarantine_key = name` | No UPDATE / DELETE policy exists for any API role. The pgTAP census asserts that every policy on `storage.objects` pins exactly one bucket literal and counts them: 2 on `avatars`, 1 on `anatomical-sheet-assets`, 0 elsewhere, total 3. ### 2.4 Purge queue (`private.object_purge_queue`) `(id, bucket_id, object_name, not_before, enqueued_at, locked_at, attempts)`, unique on `(bucket_id, object_name)` (a second enqueue keeps the earliest `not_before`). Fed by triggers on `assets`: | Transition | `object_key` | `quarantine_key` | |---|---|---| | → `deleted` | `now() + 48 h` | `now() + 1 h` | | → `rejected` | `now() + 1 h` (a clean object may exist after a PUT before a 3rd expired lease) | `now() + 1 h` | | row DELETE (account cascade) | `now() + 48 h` | `now() + 1 h` | A `clean` row's `object_key` is never enqueued (test). The 48 h grace is the backup invariant of §5. --- ## 3. The worker (`crates/worker`, container `faunosteo_worker`) ### 3.1 Process model One static musl binary, two roles in one container: - `faunosteo-worker run`: the keyed parent. Holds `SUPABASE_SERVICE_ROLE_KEY`, talks to PostgREST and the Storage API over HTTP, never decodes a pixel. - `faunosteo-worker decode`: the child spawned per job by the parent with an empty environment and piped stdin/stdout (R0 sandbox unchanged: `Threads: 1`, no core dump, `RLIMIT_AS` 512 MiB, `RLIMIT_CPU` 10 s / 12 s, seccomp allowlist, 5 MiB input, 512 KiB output, 20 s deadline). It sees no key, no file, no network. The parent checks the child's output at container level only (`webp_check`: RIFF structure, chunk allowlist, 512 × 512 canvas, no metadata chunk), hashes it and uploads it. ### 3.2 Configuration (environment of the parent only) | Variable | Required | Default | Meaning | |---|---|---|---| | `SUPABASE_URL` | yes | | origin of the stack (`http://supabase_kong_faunosteo:8000` from the Supabase compose network, `http://127.0.0.1:54321` natively) | | `SUPABASE_SERVICE_ROLE_KEY` | yes | | the service key; never printed; the JWT `exp` claim, when the key is a JWT, is logged at start and warned about 30 days ahead | | `WORKER_ID` | no | `worker-` | tag of every log line | | `WORKER_POLL_SECONDS` | no | `2` | idle interval between claims | | `WORKER_SWEEP_SECONDS` | no | `600` | interval of the sweeper | | `WORKER_QUARANTINE_GRACE_SECONDS` | no | `10800` | age before a quarantine leftover is deleted (above the 2 h validity of a signed upload URL) | | `WORKER_PENDING_MAX_AGE_SECONDS` | no | `86400` | age before a never-confirmed `pending_upload` row is expired | | `WORKER_ONCE` | no | unset | `1` = one claim pass and one sweep, then exit (tests) | `faunosteo-worker run --check` prints the resolved configuration without any secret value. Exit 0 when the worker could start (libwebp ≥ 1.6.0, both required variables set, URL parseable) or when no worker variable is set at all (an unconfigured image is a valid artifact check, `scripts/check-all.sh`); a partial or invalid configuration exits 2 naming the variable, a too-old libwebp exits 18. ### 3.3 HTTP endpoints used (service key: `apikey` and `Authorization: Bearer`) | Call | Request | |---|---| | RPC | `POST /rest/v1/rpc/`, JSON body, `Content-Type: application/json`, `Prefer: return=representation` | | download quarantine | `GET /storage/v1/object//`, body read with a 5 MiB + 1 cap; a body over the cap (impossible through the bucket cap and `confirm_asset_upload`, kept as a guard) is a content refusal: the row is rejected `decode_failed` and the worker logs `download_overflow` | | upload clean | `POST /storage/v1/object//`, `Content-Type: image/webp`, `x-upsert: true` (idempotent on retry) | | delete object | `DELETE /storage/v1/object//` (a missing object counts as done: HTTP 404, or the HTTP 400 with `statusCode` 404 this Storage API version answers) | Every call: 30 s global timeout, 5 s connect timeout, no redirect follow, `http_status_as_error(false)` so each status is classified by the caller. Keys are validated (§1.4) before any URL is built. ### 3.4 One job ```text claim_avatar_jobs(1) stage claim deadline 35 s GET quarantine stage download 35 s 404 → reject decode_failed (bytes never landed; the row said uploaded) spawn decode child, feed bytes stage decode 25 s Produced(webp) → POST clean (upsert) → complete_avatar_job(asset, token, size, sha256) stage upload 35 s, stage commit 35 s → DELETE quarantine → mark_quarantine_purged (best effort: the sweeper finishes it) Rejected{code} → reject_avatar_job(asset, token, code) code ∈ unsupported_type | decode_failed | dimensions | resource_limit→decode_failed Signalled / Timeout / OutputOverflow / Failed → reject_avatar_job(asset, token, 'processing_error') (logged with signal number / detail) Broken{code} → log level=error event=child_broken, leave the lease (the row is retried, 3rd lease rejects), stop the loop with exit code 20 complete refused (invalid_state: superseded, removed, stale lease) → log, delete nothing: the purge queue owns every object of a terminal row ``` Every network failure in a stage leaves the row to its lease (re-claimed after 10 min, rejected `processing_error` on the 3rd expiry). A stage that exceeds its deadline makes the watchdog thread exit the process (code 21); Docker's restart policy brings it back. ### 3.5 Sweeper (every `WORKER_SWEEP_SECONDS`, and once at start) 1. `expire_pending_uploads(WORKER_PENDING_MAX_AGE_SECONDS)`. 2. `claim_purges(100)` → DELETE each object (404 = done) → `ack_purge`. 3. `claim_quarantine_leftovers(WORKER_QUARANTINE_GRACE_SECONDS, 100)` → DELETE each object → `mark_quarantine_purged(asset_id)` when present. Those are the only deletions the worker ever makes: a key it did not get from the queue or the leftover query is never deleted. ### 3.6 Logs One JSON object per line on stdout, fields from an allowlist only: `ts`, `level`, `event`, `worker_id`, `asset_id`, `stage`, `code`, `signal`, `http_status`, `duration_ms`, `count`, `detail` (worker-authored text only). The child's stderr is child-controlled: it is logged under `child_stderr` JSON-escaped and truncated to 200 bytes, never raw. No key, no URL with a token, no user content. --- ## 4. Client flow (browser, anon key + user JWT) 1. Pick: ``. The first bytes are sniffed (`FF D8 FF`, PNG signature, `RIFF….WEBP`); anything else, SVG included, is refused with `unsupported_type` before any decode. `File.size` over 20 MiB → `too_large`. 2. Transport: `createImageBitmap` → canvas with the longest side at 1024 px → `toBlob('image/jpeg', 0.9)`; over 5 MiB after that → `too_large`. The preview shows that blob. Nothing uploaded yet. 3. "Enregistrer la photo": `begin_avatar_upload('image/jpeg', blob.size)` → `storage.from('avatars').upload(quarantine_key, blob, { upsert: false, contentType: 'image/jpeg' })` → `confirm_asset_upload(asset_id)`. 4. Poll the own row every 1 s, 30 s at most: `clean` → download `object_key` (`storage.from('avatars').download`), keep a blob URL in the auth store; `rejected` → message per code; still `uploaded` after 30 s → "processing, come back later" (the next mount polls again). 5. "Supprimer la photo": confirmation dialog → `remove_avatar()` → monogram. Closed failure kinds the UI phrases (`AssetFailure.kind`): `too_large`, `unsupported_type`, `rate_limited`, `storage_full`, `upload_missing`, `invalid_state`, `not_found`, `network`, `unknown`; rejection codes of a row: `unsupported_type`, `decode_failed`, `dimensions`, `processing_error`. No signed URL, no bearer token: the download runs under the SELECT policy and stops at the row flip (D7). The production CSP already allows `blob:` images; `connect-src` must name the Supabase origin (DEPLOY.md). --- ## 5. Operations ### 5.1 Reconciliation (read-only, operator) ```sql -- objects without a row (any status) and rows whose object is missing select o.bucket_id, o.name from storage.objects o where o.bucket_id = 'avatars' and not exists (select 1 from public.assets a where a.object_key = o.name or a.quarantine_key = o.name); select a.id, a.object_key from public.assets a where a.status = 'clean' and not exists (select 1 from storage.objects o where o.bucket_id = a.bucket_id and o.name = a.object_key); ``` Deletions stay row-driven (queue and leftover sweep); the query only reports. ### 5.2 Backups (self-hosted, file backend) Order: database dump first (whole instance, `storage` schema included), files second, within 48 h of each other. With the 48 h purge grace every `clean` row of the dump still has its object in the file snapshot (the invariant the restore checks with the second query above). Restore: stop the worker, restore the database, restore the volume, run the reconciliation, start the worker. Tooling (restic, retention, drill) is the deployment runbook's (plan Step 27). ### 5.3 Managed migration Same migrations (`supabase db push`); rows (`auth.users`, `public`), then the objects driven by the rows (`clean` rows and confirmed sheet assets: download, verify sha256, upload at the same key, verify), write freeze while copying (worker stopped), reconciliation on the destination, switch `SUPABASE_URL` and the keys of the app and the worker. --- ## 6. Development ```bash pnpm db:start # local stack (migrations + seed) pnpm worker:image # faunosteo_worker:dev docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges:true \ --network supabase_network_faunosteo \ -e SUPABASE_URL=http://supabase_kong_faunosteo:8000 \ -e SUPABASE_SERVICE_ROLE_KEY="$(pnpm -s supabase status -o env | sed -n 's/^SERVICE_ROLE_KEY="\(.*\)"/\1/p')" \ faunosteo_worker:dev run pnpm --filter @faunosteo/supabase test:e2e # real-stack end-to-end (needs the worker above) pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace ``` `docker compose --profile worker up worker` runs the same container from `docker-compose.yml` with `.env.worker` (`SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, `WORKER_ID`; `chmod 600`, never in the app `.env`, never in a backup of env files).