25 KiB
Files foundation: Supabase Storage, the assets table and the Rust worker
Status: Step 8 / Rust R1, 2026-10-13, branch feature/step-08-files-foundation.
Design origin: STEP8_FILES_ANALYSIS.md (D1 to D9 validated by the owner, D1
amended to the Rust worker of RUST_ARCHITECTURE_REVIEW.md Part IV). This
file is the implementation contract: table, keys, RPCs, policies, worker
behaviour and client flow are what the code and the tests pin.
First consumer: the profile photo (avatar). Every other bucket exists, closed.
1. Model
1.1 Statuses
pending_upload ─(confirm_asset_upload)─► uploaded ─(complete_avatar_job)─► clean
│ │
│ begin supersedes / 24 h expiry ├─(reject_avatar_job, 3rd expired lease)─► rejected
▼ ▼
deleted ◄──────(remove_avatar, replacement by a newer clean row, begin supersedes)
deleted and rejected are terminal: the row stays (history, purge
bookkeeping) and matches no read or upload predicate.
1.2 public.assets
| Column | Type | Rule |
|---|---|---|
id |
uuid pk | |
owner_id |
uuid → profiles.id on delete cascade |
|
kind |
text | 'profile_avatar' only in Step 8 (CHECK) |
bucket_id |
text | 'avatars' for that kind (CHECK) |
object_key |
text unique | final key, server-built: profiles/<owner_id>/<id>/avatar.webp |
quarantine_key |
text | quarantine/<owner_id>/<id>/upload; null once the worker confirmed the quarantine object is gone |
status |
public.asset_status |
enum above |
declared_mime |
text | image/jpeg, image/png, image/webp (CHECK) |
declared_size |
integer | 1 … 5 242 880 (CHECK), the transport size announced by the browser |
byte_size, sha256 |
integer, text | the clean output, set by complete_avatar_job |
rejection_reason |
text | closed set unsupported_type, decode_failed, dimensions, processing_error; non-null iff rejected (CHECK) |
attempts |
smallint | leases taken; the 3rd expired lease rejects with processing_error |
locked_at, lock_token |
timestamptz, uuid | the lease: set by claim_avatar_jobs, required by complete / reject, 10 min validity |
created_at, updated_at |
timestamptz | trigger set_updated_at |
Indexes: unique partial assets_one_clean_avatar (owner_id) where kind = 'profile_avatar' and status = 'clean' ("the" avatar of an account is its
single clean row: no profile column, D5); assets_claimable (created_at) where status = 'uploaded'; assets_owner_idx (owner_id, created_at desc).
Privileges: revoke all from public, anon, authenticated; grant select (id, owner_id, kind, bucket_id, object_key, status, declared_mime, declared_size, byte_size, sha256, rejection_reason, created_at, updated_at)
to authenticated (the lease columns and quarantine_key stay server-side;
a client must list its columns, select=* is refused). RLS
assets_select_own (owner_id = auth.uid()). No INSERT / UPDATE / DELETE
policy: every write is an RPC.
1.3 Buckets (all private)
| Bucket | Step 8 state |
|---|---|
avatars |
open: SELECT + INSERT policies through assets, 5 MiB, image/jpeg, image/png, image/webp |
logos, professional-documents, client-files, animal-files, report-files, imports |
closed: row exists, public = false, no policy names it, no cap or MIME list yet (their step sets them) |
anatomical-sheet-assets |
untouched (migration 20261009100000, read through the sheet) |
The migration upserts the rows (on conflict (id) do update set public = false) so it converges on the local stack, the VPS and a managed project.
1.4 Keys
- Built by
begin_avatar_uploadfrom uuids; the user's filename never reaches the server; one canonical name per kind (avatar.webp). - Policies compare whole strings (
name = quarantine_key,name = object_key): no prefix match, nostorage.foldername(), so PGlite and the real stack evaluate the same expression. - The worker validates every key it receives from the database before
putting it in a URL: exactly four segments,
profiles|quarantine, two uuids,avatar.webp|upload.
2. RPCs
Every function is security definer, set search_path = '', authorisation
before lookup, EXECUTE revoked from public / anon / authenticated and
granted to the one role that calls it. Business refusals raise SQLSTATE
P0001 with the closed code as the message (PostgREST answers 400 with
{"code":"P0001","message":"<code>"}); authorisation refusals raise
42501 (403).
2.1 Browser (authenticated)
| RPC | Effect | Refusals (message) |
|---|---|---|
begin_avatar_upload(declared_mime text, declared_size integer) returns jsonb |
locks the caller's profiles row (for update); checks the type and the size; rate: at most 10 rows created by the caller in the last hour, every status counted; per-account budget: the caller's bytes present under profiles/<uid>/ and quarantine/<uid>/ plus one cap below private.account_byte_budget() (64 MiB) → rate_limited; global ceiling: the bytes present in the bucket (sum of sized storage.objects, whatever each row's state) plus one 5 MiB cap per pending_upload row whose bytes have not landed, below private.storage_ceiling() (10 GiB): a claim dies with its row, so begin spam pins nothing; residuals: about 100 to 160 colluding confirmed accounts at their budget fill the ceiling (bytes still streaming for a superseded row weigh 0 until they land, bounded by rate × cap), about 2 000 accounts each keeping one bytes-less pending claim alive do the same, and a burst of accounts fills the filesystem quota, the hard bound; supersedes the caller's pending_upload / uploaded rows (→ deleted); inserts the row; returns {"asset_id", "bucket_id", "quarantine_key"} |
unsupported_type, too_large, rate_limited, storage_full |
confirm_asset_upload(asset_id uuid) returns void |
active account; own pending_upload row with a storage.objects row at (bucket_id, quarantine_key) whose metadata->>'size' is set and within the cap → uploaded |
not_found (unknown or not own), invalid_state (not pending), upload_missing (no object yet, or an object without a size: fail closed), too_large (object over the cap or over the declared size: the row stays pending and expires after 24 h) |
remove_avatar() returns integer |
every profile_avatar row of the caller in pending_upload / uploaded / clean → deleted; returns the count; logs asset.avatar_removed |
none (0 rows is fine) |
auth.uid() is null → 42501 on all three.
2.2 Worker (service_role)
| RPC | Effect |
|---|---|
claim_avatar_jobs(max_jobs integer) returns table (asset_id, owner_id, bucket_id, object_key, quarantine_key, lock_token, attempts) |
first rejects (processing_error) every uploaded row whose lease expired with attempts >= 3; then claims uploaded rows with locked_at is null or locked_at < now() - 10 min and attempts < 3, oldest first, for update skip locked, at most max_jobs (capped at 10): locked_at = now(), attempts + 1, fresh lock_token. VOLATILE (a STABLE function would run read-only through PostgREST) |
complete_avatar_job(asset_id uuid, lock_token uuid, byte_size integer, sha256 text) returns void |
guard: status = 'uploaded' and the token matches, else invalid_state. In one transaction: the owner's previous clean avatar → deleted (trigger enqueues its object 48 h later), the row → clean with byte_size / sha256, lease cleared; logs asset.avatar_set (actor null = worker, target = owner) |
reject_avatar_job(asset_id uuid, lock_token uuid, reason text) returns void |
same guard; reason in the closed set else invalid_reason; → rejected; logs asset.rejected with the reason |
mark_quarantine_purged(asset_id uuid) returns void |
clean row → quarantine_key = null (after the worker deleted the quarantine object); other statuses: no-op |
claim_purges(max_items integer) returns table (purge_id, bucket_id, object_name) |
queue rows with not_before <= now() and no live lease (locked_at null or older than 10 min), for update skip locked, at most max_items (capped at 100); sets locked_at |
ack_purge(purge_id uuid) returns void |
deletes the queue row (a missing object counted as success by the worker) |
claim_quarantine_leftovers(older_than_seconds integer, max_items integer) returns table (bucket_id, object_name, asset_id) |
storage.objects under quarantine/ in avatars, created before now() - older_than, whose row is not pending_upload / uploaded (anti-join on quarantine_key); asset_id is the matching row when one exists (the worker calls mark_quarantine_purged for a clean one) |
expire_pending_uploads(max_age_seconds integer) returns integer |
pending_upload rows older than the age → deleted (trigger enqueues their keys); returns the count |
The three stale-task guards together: a superseded or removed row is
deleted (never uploaded, so no complete); a complete from an expired
lease carries a stale lock_token and is refused even while the row is
still uploaded; the purge of an old key never touches a newer row's
object (keys embed the asset id).
2.3 storage.objects policies (bucket avatars)
| Policy | Command | Expression |
|---|---|---|
avatars_read |
SELECT, authenticated |
bucket_id = 'avatars' and private.can_read_avatar_object((select auth.uid()), name) = a clean row of the caller with object_key = name |
avatars_upload |
INSERT, authenticated |
bucket_id = 'avatars' and private.can_upload_avatar_object((select auth.uid()), name) = a pending_upload row of the caller with quarantine_key = name |
No UPDATE / DELETE policy exists for any API role. The pgTAP census asserts
that every policy on storage.objects pins exactly one bucket literal and
counts them: 2 on avatars, 1 on anatomical-sheet-assets, 0 elsewhere,
total 3.
2.4 Purge queue (private.object_purge_queue)
(id, bucket_id, object_name, not_before, enqueued_at, locked_at, attempts),
unique on (bucket_id, object_name) (a second enqueue keeps the earliest
not_before). Fed by triggers on assets:
| Transition | object_key |
quarantine_key |
|---|---|---|
→ deleted |
now() + 48 h |
now() + 1 h |
→ rejected |
now() + 1 h (a clean object may exist after a PUT before a 3rd expired lease) |
now() + 1 h |
| row DELETE (account cascade) | now() + 48 h |
now() + 1 h |
A clean row's object_key is never enqueued (test). The 48 h grace is the
backup invariant of §5.
3. The worker (crates/worker, container faunosteo_worker)
3.1 Process model
One static musl binary, two roles in one container:
faunosteo-worker run: the keyed parent. HoldsSUPABASE_SERVICE_ROLE_KEY, talks to PostgREST and the Storage API over HTTP, never decodes a pixel.faunosteo-worker decode: the child spawned per job by the parent with an empty environment and piped stdin/stdout (R0 sandbox unchanged:Threads: 1, no core dump,RLIMIT_AS512 MiB,RLIMIT_CPU10 s / 12 s, seccomp allowlist, 5 MiB input, 512 KiB output, 20 s deadline). It sees no key, no file, no network.
The parent checks the child's output at container level only
(webp_check: RIFF structure, chunk allowlist, 512 × 512 canvas, no
metadata chunk), hashes it and uploads it.
3.2 Configuration (environment of the parent only)
| Variable | Required | Default | Meaning |
|---|---|---|---|
SUPABASE_URL |
yes | origin of the stack (http://supabase_kong_faunosteo:8000 from the Supabase compose network, http://127.0.0.1:54321 natively). Cleartext http:// is accepted only for localhost, a dotted-quad loopback address, or a single-label host starting with a letter (a Docker service name; inside the musl container Docker's ndots:0 makes musl skip the search list; glibc, natively, may still try search domains on a mistyped name, so prefer loopback there); any other host must be https://, the service key travels in the headers |
|
SUPABASE_SERVICE_ROLE_KEY |
yes | the service key; never printed; the JWT exp claim, when the key is a JWT, is logged at start and warned about 30 days ahead |
|
WORKER_ID |
no | worker-<pid> |
tag of every log line |
WORKER_POLL_SECONDS |
no | 2 |
idle interval between claims |
WORKER_SWEEP_SECONDS |
no | 600 |
interval of the sweeper |
WORKER_QUARANTINE_GRACE_SECONDS |
no | 10800 |
age before a quarantine leftover is deleted (above the 2 h validity of a signed upload URL) |
WORKER_PENDING_MAX_AGE_SECONDS |
no | 86400 |
age before a never-confirmed pending_upload row is expired |
WORKER_ONCE |
no | unset | 1 = one claim pass and one sweep, then exit (tests) |
faunosteo-worker run --check prints the resolved configuration without
any secret value. Exit 0 when the worker could start (libwebp ≥ 1.6.0, both
required variables set, URL parseable) or when no worker variable is set at
all (an unconfigured image is a valid artifact check, scripts/check-all.sh);
a partial or invalid configuration exits 2 naming the variable, a too-old
libwebp exits 18.
3.3 HTTP endpoints used (service key: apikey and Authorization: Bearer)
| Call | Request |
|---|---|
| RPC | POST <url>/rest/v1/rpc/<name>, JSON body, Content-Type: application/json, Prefer: return=representation |
| download quarantine | GET <url>/storage/v1/object/<bucket>/<key>, body read with a 5 MiB + 1 cap; a body over the cap (impossible through the bucket cap and confirm_asset_upload, kept as a guard) is a content refusal: the row is rejected decode_failed and the worker logs download_overflow |
| upload clean | POST <url>/storage/v1/object/<bucket>/<key>, Content-Type: image/webp, x-upsert: true (idempotent on retry) |
| delete object | DELETE <url>/storage/v1/object/<bucket>/<key> (a missing object counts as done: HTTP 404, or the HTTP 400 with statusCode 404 this Storage API version answers) |
Every call: 30 s global timeout, 5 s connect timeout, no redirect follow, no
proxy (the environment's HTTP_PROXY is ignored on purpose),
http_status_as_error(false) so each status is classified by the caller.
Log lines carry the closed error kind and the HTTP status, never the library's
error text.
Keys are validated (§1.4) before any URL is built.
3.4 One job
claim_avatar_jobs(1) stage claim deadline 35 s
GET quarantine stage download 35 s 404 → reject decode_failed (bytes never landed; the row said uploaded)
spawn decode child, feed bytes stage decode 25 s
Produced(webp) → POST clean (upsert) → complete_avatar_job(asset, token, size, sha256)
stage upload 35 s, stage commit 35 s
→ DELETE quarantine → mark_quarantine_purged (best effort: the sweeper finishes it)
Rejected{code} → reject_avatar_job(asset, token, code) code ∈ unsupported_type | decode_failed | dimensions | resource_limit→decode_failed
Signalled / Timeout / OutputOverflow / Failed
→ reject_avatar_job(asset, token, 'processing_error') (logged with signal number / detail)
Broken{code} → log level=error event=child_broken, leave the lease (the row is retried, 3rd lease rejects), stop the loop with exit code 20
complete refused (invalid_state: superseded, removed, stale lease)
→ log, delete nothing: the purge queue owns every object of a terminal row
Every network failure in a stage leaves the row to its lease (re-claimed
after 10 min, rejected processing_error on the 3rd expiry). A stage that
exceeds its deadline makes the watchdog thread exit the process (code 21);
Docker's restart policy brings it back.
3.5 Sweeper (every WORKER_SWEEP_SECONDS, and once at start)
expire_pending_uploads(WORKER_PENDING_MAX_AGE_SECONDS).claim_purges(100)→ DELETE each object (404 = done) →ack_purge.claim_quarantine_leftovers(WORKER_QUARANTINE_GRACE_SECONDS, 100)→ DELETE each object →mark_quarantine_purged(asset_id)when present.
Those are the only deletions the worker ever makes: a key it did not get from the queue or the leftover query is never deleted.
3.6 Sheet verification mode (V1-2.3)
faunosteo-worker run --jobs sheet-verify, service worker-verify under the
Compose profile worker (same image, environment file and hardening as
worker, but mem_limit 1 280 MiB so the child's 1 GiB limit binds first), runs the server verdict of submitted sheet rounds (V1-2 contract
§6.4, §6.5) and nothing else: no avatar job, no sweep. The default run is
unchanged.
- Parent:
claim_sheet_verifications(1)(M6, service role only), thenget_sheet_verification_input(the frozen document and the base as text, the drawing rows), then each drawing fromanatomical-sheet-assetsat its key<sha256>.svg. A row not uploaded, a key that is not its own sha256, an object missing (404) or over the policy (at most 4 drawings, 2 MiB each, 6 MiB in all: the archive caps) is passed to the core without bytes, so the verdict names it (drawing_missing, caps): a business failure, never a worker failure. The parent never parses the document or a drawing. - Child
faunosteo-worker verify: same sandbox asdecode(empty environment, one thread, no core dump, the seccomp allowlist) with its own limits: framed stdin input up to 12 MiB,RLIMIT_AS1 GiB,RLIMIT_CPU60 s, 90 s deadline, verdict output up to 3 MiB (under the SQL cap of 4 MiB onjsonb::text). It runs the core'scompute_verdictand writes the verdict JSON; a failed verdict is still a normal exit. The seccomp derivation of 2026-11-01 tracedbrkin the verify child at the caps (musl mallocng growing its heap): added to the allowlist, the only change. - Completion:
complete_sheet_verificationanswerspassed,failed,stale(nothing written, logged) ormalformed(reportedverification.worker.verdict_refused). Child timeout →verification.worker.timeout(transient); SIGXCPU, SIGABRT or SIGSYS (the child's own CPU, memory and seccomp budgets), an output overflow or an input over its cap →verification.worker.resource_limit(definitive: the frozen input gives the same end every time); any other signal, SIGKILL first (the OOM killer or a stop, from outside) and a transport error →verification.worker.unavailable(transient); a broken child →unavailable, then the worker stops (as the avatar loop). A dead lease (10 minutes) is claimed again asverification.worker.lease_expired. Retry by cause:resource_limitandverdict_refusedclose the roundverification_failed/unavailableat the first occurrence; the other codes retry up to the fifth attempt, which closes it the same way. - Retry delay (owner decision 2026-11-01): a failure reported by
fail_sheet_verificationmakes the round claimable again after 2 s, 5 s, 10 s, 20 s (after attempts 1 to 4), kept in SQL (not_before, hidden from the API roles), so the worker never sleeps for it and the claim of another round is not blocked; the effective delay is that delay plus up to one poll interval (WORKER_POLL_SECONDS, 2 s). A verdict (failed or not) is never retried; a dead lease is claimed again at once. Only the transient codes wait: the delay applies totimeout,unavailableandlease_expired(SQL decides,private.sheet_worker_final_codes). - Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations, 8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict 24 689 bytes; the equine draft in about 60 ms.
- Sandbox only:
pnpm db:sandbox verify-e2estarts a gateway and this worker on the sandbox network (SUPABASE.md §5.1). Never started on DEV (V1-2 contract §16.11 item 3).
3.7 Logs
One JSON object per line on stdout, fields from an allowlist only: ts,
level, event, worker_id, asset_id, review_id, stage, code, signal,
http_status, duration_ms, count, detail (worker-authored text only).
The child's stderr is child-controlled: it is logged under child_stderr
JSON-escaped and truncated to 200 bytes, never raw. No key, no URL with a
token, no user content.
4. Client flow (browser, anon key + user JWT)
- Pick:
<input type="file" accept="image/jpeg,image/png,image/webp">. The first bytes are sniffed (FF D8 FF, PNG signature,RIFF….WEBP); anything else, SVG included, is refused withunsupported_typebefore any decode.File.sizeover 20 MiB →too_large. - Transport:
createImageBitmap→ canvas with the longest side at 1024 px →toBlob('image/jpeg', 0.9); over 5 MiB after that →too_large. The preview shows that blob. Nothing uploaded yet. - "Enregistrer la photo":
begin_avatar_upload('image/jpeg', blob.size)→storage.from('avatars').upload(quarantine_key, blob, { upsert: false, contentType: 'image/jpeg' })→confirm_asset_upload(asset_id). - Poll the own row every 1 s, 30 s at most:
clean→ downloadobject_key(storage.from('avatars').download), keep a blob URL in the auth store;rejected→ message per code; stilluploadedafter 30 s → "processing, come back later" (the next mount polls again). Only a row the section knows at mount resumes: after a full page reload the section mounts before the auth store has read the rows, so it shows the clean photo with « Modifier » and « Supprimer » enabled, and a new save supersedes the waiting row (deleted, keys enqueued). An in-app return to Mon compte resumes the polling. Without a running worker (a DEV stack wheredocker compose --profile worker up workerwas not started, §3.2 of SUPABASE.md) every new photo staysuploadedand ends on that message; the clean photo is kept (AVATAR-01, 2026-10-27, isolated reproduction). - "Supprimer la photo": confirmation dialog →
remove_avatar()→ monogram.
Closed failure kinds the UI phrases (AssetFailure.kind): too_large,
unsupported_type, rate_limited, storage_full, upload_missing,
invalid_state, not_found, network, unknown; rejection codes of a row:
unsupported_type, decode_failed, dimensions, processing_error.
No signed URL, no bearer token: the download runs under the SELECT policy
and stops at the row flip (D7). The production CSP already allows blob:
images; connect-src must name the Supabase origin (DEPLOY.md).
5. Operations
5.1 Reconciliation (read-only, operator)
-- objects without a row (any status) and rows whose object is missing
select o.bucket_id, o.name from storage.objects o
where o.bucket_id = 'avatars'
and not exists (select 1 from public.assets a where a.object_key = o.name or a.quarantine_key = o.name);
select a.id, a.object_key from public.assets a
where a.status = 'clean'
and not exists (select 1 from storage.objects o where o.bucket_id = a.bucket_id and o.name = a.object_key);
Deletions stay row-driven (queue and leftover sweep); the query only reports.
5.2 Backups (self-hosted, file backend)
Order: database dump first (whole instance, storage schema included),
files second, within 48 h of each other. With the 48 h purge grace every
clean row of the dump still has its object in the file snapshot (the
invariant the restore checks with the second query above). Restore: stop
the worker, restore the database, restore the volume, run the
reconciliation, start the worker. Tooling (restic, retention, drill) is the
deployment runbook's (plan Step 27).
5.3 Managed migration
Same migrations (supabase db push); rows (auth.users, public), then the
objects driven by the rows (clean rows and confirmed sheet assets:
download, verify sha256, upload at the same key, verify), write freeze while
copying (worker stopped), reconciliation on the destination, switch
SUPABASE_URL and the keys of the app and the worker.
6. Development
pnpm db:start # local stack (migrations + seed)
pnpm worker:image # faunosteo_worker:dev
docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges:true \
--network supabase_network_faunosteo \
-e SUPABASE_URL=http://supabase_kong_faunosteo:8000 \
-e SUPABASE_SERVICE_ROLE_KEY="$(pnpm -s supabase status -o env | sed -n 's/^SERVICE_ROLE_KEY="\(.*\)"/\1/p')" \
faunosteo_worker:dev run # the worker beside the local stack, for the manual photo flow (IDENTITY.md); the e2e suite refuses this stack
pnpm e2e:avatar # = pnpm db:sandbox avatar-e2e: the suite and the worker (5 s sweep, no quarantine grace) on the throwaway stack only (SUPABASE.md §5.1); the DEV script was removed on 2026-11-01
docker compose --profile worker up worker runs the same container from
docker-compose.yml with .env.worker (SUPABASE_URL,
SUPABASE_SERVICE_ROLE_KEY, WORKER_ID; chmod 600, never in the app
.env, never in a backup of env files).