# Files foundation: Supabase Storage, the `assets` table and the Rust worker Status: Step 8 / Rust R1, 2026-10-13, branch `feature/step-08-files-foundation`. Design origin: `STEP8_FILES_ANALYSIS.md` (D1 to D9 validated by the owner, D1 amended to the Rust worker of `RUST_ARCHITECTURE_REVIEW.md` Part IV). This file is the implementation contract: table, keys, RPCs, policies, worker behaviour and client flow are what the code and the tests pin. First consumer: the profile photo (avatar). Every other bucket exists, closed. --- ## 1. Model ### 1.1 Statuses ```text pending_upload ─(confirm_asset_upload)─► uploaded ─(complete_avatar_job)─► clean │ │ │ begin supersedes / 24 h expiry ├─(reject_avatar_job, 3rd expired lease)─► rejected ▼ ▼ deleted ◄──────(remove_avatar, replacement by a newer clean row, begin supersedes) ``` `deleted` and `rejected` are terminal: the row stays (history, purge bookkeeping) and matches no read or upload predicate. ### 1.2 `public.assets` | Column | Type | Rule | |---|---|---| | `id` | uuid pk | | | `owner_id` | uuid → `profiles.id` on delete cascade | | | `kind` | text | `'profile_avatar'` only in Step 8 (CHECK) | | `bucket_id` | text | `'avatars'` for that kind (CHECK) | | `object_key` | text unique | final key, server-built: `profiles///avatar.webp` | | `quarantine_key` | text | `quarantine///upload`; null once the worker confirmed the quarantine object is gone | | `status` | `public.asset_status` | enum above | | `declared_mime` | text | `image/jpeg`, `image/png`, `image/webp` (CHECK) | | `declared_size` | integer | 1 … 5 242 880 (CHECK), the transport size announced by the browser | | `byte_size`, `sha256` | integer, text | the clean output, set by `complete_avatar_job` | | `rejection_reason` | text | closed set `unsupported_type`, `decode_failed`, `dimensions`, `processing_error`; non-null iff `rejected` (CHECK) | | `attempts` | smallint | leases taken; the 3rd expired lease rejects with `processing_error` | | `locked_at`, `lock_token` | timestamptz, uuid | the lease: set by `claim_avatar_jobs`, required by `complete` / `reject`, 10 min validity | | `created_at`, `updated_at` | timestamptz | trigger `set_updated_at` | Indexes: unique partial `assets_one_clean_avatar (owner_id) where kind = 'profile_avatar' and status = 'clean'` ("the" avatar of an account is its single clean row: no profile column, D5); `assets_claimable (created_at) where status = 'uploaded'`; `assets_owner_idx (owner_id, created_at desc)`. Privileges: `revoke all` from `public`, `anon`, `authenticated`; `grant select (id, owner_id, kind, bucket_id, object_key, status, declared_mime, declared_size, byte_size, sha256, rejection_reason, created_at, updated_at)` to `authenticated` (the lease columns and `quarantine_key` stay server-side; a client must list its columns, `select=*` is refused). RLS `assets_select_own` (`owner_id = auth.uid()`). No INSERT / UPDATE / DELETE policy: every write is an RPC. ### 1.3 Buckets (all private) | Bucket | Step 8 state | |---|---| | `avatars` | open: SELECT + INSERT policies through `assets`, 5 MiB, `image/jpeg`, `image/png`, `image/webp` | | `logos`, `professional-documents`, `client-files`, `animal-files`, `report-files`, `imports` | closed: row exists, `public = false`, no policy names it, no cap or MIME list yet (their step sets them) | | `anatomical-sheet-assets` | untouched (migration `20261009100000`, read through the sheet) | The migration upserts the rows (`on conflict (id) do update set public = false`) so it converges on the local stack, the VPS and a managed project. ### 1.4 Keys - Built by `begin_avatar_upload` from uuids; the user's filename never reaches the server; one canonical name per kind (`avatar.webp`). - Policies compare whole strings (`name = quarantine_key`, `name = object_key`): no prefix match, no `storage.foldername()`, so PGlite and the real stack evaluate the same expression. - The worker validates every key it receives from the database before putting it in a URL: exactly four segments, `profiles|quarantine`, two uuids, `avatar.webp|upload`. --- ## 2. RPCs Every function is `security definer`, `set search_path = ''`, authorisation before lookup, EXECUTE revoked from `public` / `anon` / `authenticated` and granted to the one role that calls it. Business refusals raise SQLSTATE `P0001` with the closed code as the message (PostgREST answers 400 with `{"code":"P0001","message":""}`); authorisation refusals raise `42501` (403). ### 2.1 Browser (`authenticated`) | RPC | Effect | Refusals (message) | |---|---|---| | `begin_avatar_upload(declared_mime text, declared_size integer) returns jsonb` | locks the caller's `profiles` row (`for update`); checks the type and the size; rate: at most 10 rows created by the caller in the last hour, every status counted; per-account budget: the caller's bytes present under `profiles//` and `quarantine//` plus one cap below `private.account_byte_budget()` (64 MiB) → `rate_limited`; global ceiling: the bytes present in the bucket (sum of sized `storage.objects`, whatever each row's state) plus one 5 MiB cap per `pending_upload` row whose bytes have not landed, below `private.storage_ceiling()` (10 GiB): a claim dies with its row, so begin spam pins nothing; residuals: about 100 to 160 colluding confirmed accounts at their budget fill the ceiling (bytes still streaming for a superseded row weigh 0 until they land, bounded by rate × cap), about 2 000 accounts each keeping one bytes-less pending claim alive do the same, and a burst of accounts fills the filesystem quota, the hard bound; supersedes the caller's `pending_upload` / `uploaded` rows (→ `deleted`); inserts the row; returns `{"asset_id", "bucket_id", "quarantine_key"}` | `unsupported_type`, `too_large`, `rate_limited`, `storage_full` | | `confirm_asset_upload(asset_id uuid) returns void` | active account; own `pending_upload` row with a `storage.objects` row at `(bucket_id, quarantine_key)` whose `metadata->>'size'` is set and within the cap → `uploaded` | `not_found` (unknown or not own), `invalid_state` (not pending), `upload_missing` (no object yet, or an object without a size: fail closed), `too_large` (object over the cap or over the declared size: the row stays pending and expires after 24 h) | | `remove_avatar() returns integer` | every `profile_avatar` row of the caller in `pending_upload` / `uploaded` / `clean` → `deleted`; returns the count; logs `asset.avatar_removed` | none (0 rows is fine) | `auth.uid() is null` → `42501` on all three. ### 2.2 Worker (`service_role`) | RPC | Effect | |---|---| | `claim_avatar_jobs(max_jobs integer) returns table (asset_id, owner_id, bucket_id, object_key, quarantine_key, lock_token, attempts)` | first rejects (`processing_error`) every `uploaded` row whose lease expired with `attempts >= 3`; then claims `uploaded` rows with `locked_at is null or locked_at < now() - 10 min` and `attempts < 3`, oldest first, `for update skip locked`, at most `max_jobs` (capped at 10): `locked_at = now()`, `attempts + 1`, fresh `lock_token`. VOLATILE (a STABLE function would run read-only through PostgREST) | | `complete_avatar_job(asset_id uuid, lock_token uuid, byte_size integer, sha256 text) returns void` | guard: `status = 'uploaded'` and the token matches, else `invalid_state`. In one transaction: the owner's previous `clean` avatar → `deleted` (trigger enqueues its object 48 h later), the row → `clean` with `byte_size` / `sha256`, lease cleared; logs `asset.avatar_set` (actor null = worker, target = owner) | | `reject_avatar_job(asset_id uuid, lock_token uuid, reason text) returns void` | same guard; `reason` in the closed set else `invalid_reason`; → `rejected`; logs `asset.rejected` with the reason | | `mark_quarantine_purged(asset_id uuid) returns void` | `clean` row → `quarantine_key = null` (after the worker deleted the quarantine object); other statuses: no-op | | `claim_purges(max_items integer) returns table (purge_id, bucket_id, object_name)` | queue rows with `not_before <= now()` and no live lease (`locked_at` null or older than 10 min), `for update skip locked`, at most `max_items` (capped at 100); sets `locked_at` | | `ack_purge(purge_id uuid) returns void` | deletes the queue row (a missing object counted as success by the worker) | | `claim_quarantine_leftovers(older_than_seconds integer, max_items integer) returns table (bucket_id, object_name, asset_id)` | `storage.objects` under `quarantine/` in `avatars`, created before `now() - older_than`, whose row is not `pending_upload` / `uploaded` (anti-join on `quarantine_key`); `asset_id` is the matching row when one exists (the worker calls `mark_quarantine_purged` for a `clean` one) | | `expire_pending_uploads(max_age_seconds integer) returns integer` | `pending_upload` rows older than the age → `deleted` (trigger enqueues their keys); returns the count | The three stale-task guards together: a superseded or removed row is `deleted` (never `uploaded`, so no `complete`); a `complete` from an expired lease carries a stale `lock_token` and is refused even while the row is still `uploaded`; the purge of an old key never touches a newer row's object (keys embed the asset id). ### 2.3 `storage.objects` policies (bucket `avatars`) | Policy | Command | Expression | |---|---|---| | `avatars_read` | SELECT, `authenticated` | `bucket_id = 'avatars' and private.can_read_avatar_object((select auth.uid()), name)` = a `clean` row of the caller with `object_key = name` | | `avatars_upload` | INSERT, `authenticated` | `bucket_id = 'avatars' and private.can_upload_avatar_object((select auth.uid()), name)` = a `pending_upload` row of the caller with `quarantine_key = name` | No UPDATE / DELETE policy exists for any API role. The pgTAP census asserts that every policy on `storage.objects` pins exactly one bucket literal and counts them: 2 on `avatars`, 1 on `anatomical-sheet-assets`, 0 elsewhere, total 3. ### 2.4 Purge queue (`private.object_purge_queue`) `(id, bucket_id, object_name, not_before, enqueued_at, locked_at, attempts)`, unique on `(bucket_id, object_name)` (a second enqueue keeps the earliest `not_before`). Fed by triggers on `assets`: | Transition | `object_key` | `quarantine_key` | |---|---|---| | → `deleted` | `now() + 48 h` | `now() + 1 h` | | → `rejected` | `now() + 1 h` (a clean object may exist after a PUT before a 3rd expired lease) | `now() + 1 h` | | row DELETE (account cascade) | `now() + 48 h` | `now() + 1 h` | A `clean` row's `object_key` is never enqueued (test). The 48 h grace is the backup invariant of §5. --- ## 3. The worker (`crates/worker`, container `faunosteo_worker`) ### 3.1 Process model One static musl binary, two roles in one container: - `faunosteo-worker run`: the keyed parent. Holds `SUPABASE_SERVICE_ROLE_KEY`, talks to PostgREST and the Storage API over HTTP, never decodes a pixel. - `faunosteo-worker decode`: the child spawned per job by the parent with an empty environment and piped stdin/stdout (R0 sandbox unchanged: `Threads: 1`, no core dump, `RLIMIT_AS` 512 MiB, `RLIMIT_CPU` 10 s / 12 s, seccomp allowlist, 5 MiB input, 512 KiB output, 20 s deadline). It sees no key, no file, no network. The parent checks the child's output at container level only (`webp_check`: RIFF structure, chunk allowlist, 512 × 512 canvas, no metadata chunk), hashes it and uploads it. ### 3.2 Configuration (environment of the parent only) | Variable | Required | Default | Meaning | |---|---|---|---| | `SUPABASE_URL` | yes | | origin of the stack (`http://supabase_kong_faunosteo:8000` from the Supabase compose network, `http://127.0.0.1:54321` natively). Cleartext `http://` is accepted only for `localhost`, a dotted-quad loopback address, or a single-label host starting with a letter (a Docker service name; inside the musl container Docker's `ndots:0` makes musl skip the search list; glibc, natively, may still try search domains on a mistyped name, so prefer loopback there); any other host must be `https://`, the service key travels in the headers | | `SUPABASE_SERVICE_ROLE_KEY` | yes | | the service key; never printed; the JWT `exp` claim, when the key is a JWT, is logged at start and warned about 30 days ahead | | `WORKER_ID` | no | `worker-` | tag of every log line | | `WORKER_POLL_SECONDS` | no | `2` | idle interval between claims | | `WORKER_SWEEP_SECONDS` | no | `600` | interval of the sweeper | | `WORKER_QUARANTINE_GRACE_SECONDS` | no | `10800` | age before a quarantine leftover is deleted (above the 2 h validity of a signed upload URL) | | `WORKER_PENDING_MAX_AGE_SECONDS` | no | `86400` | age before a never-confirmed `pending_upload` row is expired | | `WORKER_ONCE` | no | unset | `1` = one claim pass and one sweep, then exit (tests) | `faunosteo-worker run --check` prints the resolved configuration without any secret value. Exit 0 when the worker could start (libwebp ≥ 1.6.0, both required variables set, URL parseable) or when no worker variable is set at all (an unconfigured image is a valid artifact check, `scripts/check-all.sh`); a partial or invalid configuration exits 2 naming the variable, a too-old libwebp exits 18. ### 3.3 HTTP endpoints used (service key: `apikey` and `Authorization: Bearer`) | Call | Request | |---|---| | RPC | `POST /rest/v1/rpc/`, JSON body, `Content-Type: application/json`, `Prefer: return=representation` | | download quarantine | `GET /storage/v1/object//`, body read with a 5 MiB + 1 cap; a body over the cap (impossible through the bucket cap and `confirm_asset_upload`, kept as a guard) is a content refusal: the row is rejected `decode_failed` and the worker logs `download_overflow` | | upload clean | `POST /storage/v1/object//`, `Content-Type: image/webp`, `x-upsert: true` (idempotent on retry) | | delete object | `DELETE /storage/v1/object//` (a missing object counts as done: HTTP 404, or the HTTP 400 with `statusCode` 404 this Storage API version answers) | Every call: 30 s global timeout, 5 s connect timeout, no redirect follow, no proxy (the environment's `HTTP_PROXY` is ignored on purpose), `http_status_as_error(false)` so each status is classified by the caller. Log lines carry the closed error kind and the HTTP status, never the library's error text. Keys are validated (§1.4) before any URL is built. ### 3.4 One job ```text claim_avatar_jobs(1) stage claim deadline 35 s GET quarantine stage download 35 s 404 → reject decode_failed (bytes never landed; the row said uploaded) spawn decode child, feed bytes stage decode 25 s Produced(webp) → POST clean (upsert) → complete_avatar_job(asset, token, size, sha256) stage upload 35 s, stage commit 35 s → DELETE quarantine → mark_quarantine_purged (best effort: the sweeper finishes it) Rejected{code} → reject_avatar_job(asset, token, code) code ∈ unsupported_type | decode_failed | dimensions | resource_limit→decode_failed Signalled / Timeout / OutputOverflow / Failed → reject_avatar_job(asset, token, 'processing_error') (logged with signal number / detail) Broken{code} → log level=error event=child_broken, leave the lease (the row is retried, 3rd lease rejects), stop the loop with exit code 20 complete refused (invalid_state: superseded, removed, stale lease) → log, delete nothing: the purge queue owns every object of a terminal row ``` Every network failure in a stage leaves the row to its lease (re-claimed after 10 min, rejected `processing_error` on the 3rd expiry). A stage that exceeds its deadline makes the watchdog thread exit the process (code 21); Docker's restart policy brings it back. ### 3.5 Sweeper (every `WORKER_SWEEP_SECONDS`, and once at start) 1. `expire_pending_uploads(WORKER_PENDING_MAX_AGE_SECONDS)`. 2. `claim_purges(100)` → DELETE each object (404 = done) → `ack_purge`. 3. `claim_quarantine_leftovers(WORKER_QUARANTINE_GRACE_SECONDS, 100)` → DELETE each object → `mark_quarantine_purged(asset_id)` when present. Those are the only deletions the worker ever makes: a key it did not get from the queue or the leftover query is never deleted. ### 3.6 Sheet verification mode (V1-2.3) `faunosteo-worker run --jobs sheet-verify`, service `worker-verify` under the Compose profile `worker` (same image, environment file and hardening as `worker`, but `mem_limit` 1 280 MiB so the child's 1 GiB limit binds first), runs the server verdict of submitted sheet rounds (V1-2 contract §6.4, §6.5) and nothing else: no avatar job, no sweep. The default `run` is unchanged. - Parent: `claim_sheet_verifications(1)` (M6, service role only), then `get_sheet_verification_input` (the frozen document and the base as text, the drawing rows), then each drawing from `anatomical-sheet-assets` at its key `.svg`. A row not uploaded, a key that is not its own sha256, an object missing (404) or over the policy (at most 4 drawings, 2 MiB each, 6 MiB in all: the archive caps) is passed to the core without bytes, so the verdict names it (`drawing_missing`, caps): a business failure, never a worker failure. The parent never parses the document or a drawing. - Child `faunosteo-worker verify`: same sandbox as `decode` (empty environment, one thread, no core dump, the seccomp allowlist) with its own limits: framed stdin input up to 12 MiB, `RLIMIT_AS` 1 GiB, `RLIMIT_CPU` 60 s, 90 s deadline, verdict output up to 3 MiB (under the SQL cap of 4 MiB on `jsonb::text`). It runs the core's `compute_verdict` and writes the verdict JSON; a failed verdict is still a normal exit. The seccomp derivation of 2026-11-01 traced `brk` in the verify child at the caps (musl mallocng growing its heap): added to the allowlist, the only change. - Completion: `complete_sheet_verification` answers `passed`, `failed`, `stale` (nothing written, logged) or `malformed` (reported `verification.worker.verdict_refused`). Child timeout → `verification.worker.timeout`; a signal, an output overflow or an input over its cap → `verification.worker.resource_limit`; a transport error → `verification.worker.unavailable`; a broken child → `unavailable`, then the worker stops (as the avatar loop). A dead lease (10 minutes) is claimed again as `verification.worker.lease_expired`; the fifth attempt closes the round `verification_failed` / `unavailable`. - Retry delay (owner decision 2026-11-01): a failure reported by `fail_sheet_verification` makes the round claimable again after 2 s, 5 s, 10 s, 20 s (after attempts 1 to 4), kept in SQL (`not_before`, hidden from the API roles), so the worker never sleeps for it and the claim of another round is not blocked; the effective delay is that delay plus up to one poll interval (`WORKER_POLL_SECONDS`, 2 s). A verdict (failed or not) is never retried; a dead lease is claimed again at once. The worker code is the same as before. - Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations, 8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict 24 689 bytes; the equine draft in about 60 ms. - Sandbox only: `pnpm db:sandbox verify-e2e` starts a gateway and this worker on the sandbox network (SUPABASE.md §5.1). Never started on DEV (V1-2 contract §16.11 item 3). ### 3.7 Logs One JSON object per line on stdout, fields from an allowlist only: `ts`, `level`, `event`, `worker_id`, `asset_id`, `review_id`, `stage`, `code`, `signal`, `http_status`, `duration_ms`, `count`, `detail` (worker-authored text only). The child's stderr is child-controlled: it is logged under `child_stderr` JSON-escaped and truncated to 200 bytes, never raw. No key, no URL with a token, no user content. --- ## 4. Client flow (browser, anon key + user JWT) 1. Pick: ``. The first bytes are sniffed (`FF D8 FF`, PNG signature, `RIFF….WEBP`); anything else, SVG included, is refused with `unsupported_type` before any decode. `File.size` over 20 MiB → `too_large`. 2. Transport: `createImageBitmap` → canvas with the longest side at 1024 px → `toBlob('image/jpeg', 0.9)`; over 5 MiB after that → `too_large`. The preview shows that blob. Nothing uploaded yet. 3. "Enregistrer la photo": `begin_avatar_upload('image/jpeg', blob.size)` → `storage.from('avatars').upload(quarantine_key, blob, { upsert: false, contentType: 'image/jpeg' })` → `confirm_asset_upload(asset_id)`. 4. Poll the own row every 1 s, 30 s at most: `clean` → download `object_key` (`storage.from('avatars').download`), keep a blob URL in the auth store; `rejected` → message per code; still `uploaded` after 30 s → "processing, come back later" (the next mount polls again). Only a row the section knows at mount resumes: after a full page reload the section mounts before the auth store has read the rows, so it shows the clean photo with « Modifier » and « Supprimer » enabled, and a new save supersedes the waiting row (`deleted`, keys enqueued). An in-app return to Mon compte resumes the polling. Without a running worker (a DEV stack where `docker compose --profile worker up worker` was not started, §3.2 of SUPABASE.md) every new photo stays `uploaded` and ends on that message; the clean photo is kept (AVATAR-01, 2026-10-27, isolated reproduction). 5. "Supprimer la photo": confirmation dialog → `remove_avatar()` → monogram. Closed failure kinds the UI phrases (`AssetFailure.kind`): `too_large`, `unsupported_type`, `rate_limited`, `storage_full`, `upload_missing`, `invalid_state`, `not_found`, `network`, `unknown`; rejection codes of a row: `unsupported_type`, `decode_failed`, `dimensions`, `processing_error`. No signed URL, no bearer token: the download runs under the SELECT policy and stops at the row flip (D7). The production CSP already allows `blob:` images; `connect-src` must name the Supabase origin (DEPLOY.md). --- ## 5. Operations ### 5.1 Reconciliation (read-only, operator) ```sql -- objects without a row (any status) and rows whose object is missing select o.bucket_id, o.name from storage.objects o where o.bucket_id = 'avatars' and not exists (select 1 from public.assets a where a.object_key = o.name or a.quarantine_key = o.name); select a.id, a.object_key from public.assets a where a.status = 'clean' and not exists (select 1 from storage.objects o where o.bucket_id = a.bucket_id and o.name = a.object_key); ``` Deletions stay row-driven (queue and leftover sweep); the query only reports. ### 5.2 Backups (self-hosted, file backend) Order: database dump first (whole instance, `storage` schema included), files second, within 48 h of each other. With the 48 h purge grace every `clean` row of the dump still has its object in the file snapshot (the invariant the restore checks with the second query above). Restore: stop the worker, restore the database, restore the volume, run the reconciliation, start the worker. Tooling (restic, retention, drill) is the deployment runbook's (plan Step 27). ### 5.3 Managed migration Same migrations (`supabase db push`); rows (`auth.users`, `public`), then the objects driven by the rows (`clean` rows and confirmed sheet assets: download, verify sha256, upload at the same key, verify), write freeze while copying (worker stopped), reconciliation on the destination, switch `SUPABASE_URL` and the keys of the app and the worker. --- ## 6. Development ```bash pnpm db:start # local stack (migrations + seed) pnpm worker:image # faunosteo_worker:dev docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges:true \ --network supabase_network_faunosteo \ -e SUPABASE_URL=http://supabase_kong_faunosteo:8000 \ -e SUPABASE_SERVICE_ROLE_KEY="$(pnpm -s supabase status -o env | sed -n 's/^SERVICE_ROLE_KEY="\(.*\)"/\1/p')" \ faunosteo_worker:dev run pnpm --filter @faunosteo/supabase test:e2e # real-stack end-to-end (needs the worker above) pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace (DEV: the owner's call) pnpm db:sandbox avatar-e2e # the same suite and worker on the throwaway stack (SUPABASE.md §5.1): the check of a storage or worker change ``` `docker compose --profile worker up worker` runs the same container from `docker-compose.yml` with `.env.worker` (`SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, `WORKER_ID`; `chmod 600`, never in the app `.env`, never in a backup of env files).