Email sending (#3707)

All product email sent by our own code (receipts, conversion results, shares, contact/feedback notifications, admin alerts, marketing) goes through one provider-agnostic library, @accessible/email (packages/email). It chooses the provider, fails over when that is safe, skips suppressed addresses, and writes one row per send to the unified send log. Provider delivery events (delivered, bounced, complained, …) are folded into the same log, and delivery failures reported by the provider (bounced / complained / failed / suppressed-by-provider) raise an admin alert.

Out of scope — Supabase Auth (GoTrue) mail. The main sign-in flow is Supabase Auth signInWithOtp, which GoTrue sends over its own SMTP settings, not through this library, so those sign-in links do not appear as library sends (only as delivery events, category unknown, if they go via Resend). The only sign-in link the library sends is the AccessU promo magic link (POST /api/promo/accessu/claim). Phase 3 points GoTrue’s SMTP at SES (see “Supabase Auth SMTP (SES)”), so its delivery events land in email_log as provider ses, category unknown; it still has no library failover.

This replaces the Resend-only delivery-log design from PR #2979 (resend_email_events, migration 297 — never applied).

Architecture

call site ──► workers/api services/email-sender.ts ──► @accessible/email
                                                          │  1. isSuppressed?  (email_suppressions)
                                                          │  2. primary provider (EMAIL_PRIMARY)
                                                          │  3. fallback provider (EMAIL_FALLBACK), only if "unavailable"
                                                          └► record_email_send()  ──► email_log

Resend ──(Svix webhook, real time)──► supabase/functions/resend-webhook        ┐
Resend ◄─(list API, every 15 min)──── supabase/functions/resend-events-fetcher ┴► record_email_event() ──► email_log
SES ──► config set ──► EventBridge (default bus) ──► POST /api/webhooks/ses-events ┘  (workers/api, alerts in process)
                                                                                    │  (+ email_suppressions on hard bounce / complaint)
                                        alert = true (once per failed send) ◄───────┘
                                                   │
                                                   ▼
                       POST /internal/email-alert (workers/api) ──► dispatchAdminAlert (email + Telegram)

admin.gevell.com/admin/email-logs ──► GET /api/admin/email-log, GET/DELETE /api/admin/email-suppressions

Library (packages/email/src): createEmailClient / sendEmailWithEnv. A send never throws; the result is a union on status (sent / deferred / suppressed / failed). ok means the message was handed to a provider: sent, or deferred (an ambiguous attempt queued for a retry on the same idempotency key, so it goes out exactly once). suppressed and failed are ok: false and always carry error. A caller must never tell a user “sent” unless ok is true. Call sites in workers/api use services/email-sender.ts → sendEmail(request, { resendApiKey }) and log result.error when !result.ok. User-facing responses follow the result: the share route’s recipients counts only emails that went out (the rest are in unsentRecipients), a team invite reports inviteSent: true only on an ok send, and the promo magic-link claim tells the user to request a link themselves when the email didn’t go out.
Supabase hooks (send log + suppression lookup) time out after 2.5 s each. A timed-out or malformed suppression lookup fails open (the send proceeds); a timed-out log write is reported and never changes the send result. When SUPABASE_URL / the service key are missing the library warns once per process and sends without logging or suppression.
Categories: auth (the promo magic link, account deletion), transactional, notification, alert, marketing. Each send also carries a source tag such as api:contact. Rows written only by delivery events (mail sent outside the library) have category unknown.
Providers: Resend (resend) and Amazon SES (ses, SESv2 API signed with aws4fetch, no AWS SDK, so it also runs on Cloudflare Workers). From phase 3, SES is primary and Resend the fallback, set once for every runtime by EMAIL_PRIMARY / EMAIL_FALLBACK in SSM /accessible-pdf/shared (see “Switch providers”). The library default when unset is still resend.
Pinned sends. A request with provider: 'ses' (or 'resend') goes only through that provider and never fails over. Prospect outreach pins SES so cold-mail bounces and complaints never land on the Resend reputation that carries transactional mail. Suppression, logging and deferral still apply.
SES credentials (packages/email/src/aws-credentials.ts): env keys AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY plus AWS_SESSION_TOKEN (the Lambda’s role credentials are temporary; the old outreach client dropped the token, so every Lambda SES send was refused), else the EC2 instance role via IMDSv2. Missing credentials are unavailable (failover, or a clear failure on a pinned send), never a crash.
Tags and configuration sets. tags become SES EmailTags / Resend tags and come back on delivery events. sesConfigurationSet picks the SES configuration set for a send (outreach: accessible-outreach); otherwise EMAIL_SES_CONFIGURATION_SET, else the identity’s default set (gevell-transactional).
Legacy Cloudflare Worker entry: the old wrangler entry of workers/api (src/index.ts, compatibility_date = "2024-09-23") has no process.env. sendEmail(request, { env: c.env }) reads the email settings from the request bindings instead; call sites that don’t pass env send with the Resend key they pass but write no email_log row and check no suppressions there. The EC2 API and the Lambda are the supported runtimes.

Failover rules — never double-send

Each provider error is classified (packages/email/src/types.ts, FailureKind):

Kind	Meaning	What the library does
`rejected`	The provider refused this message (bad address, validation, provider-side suppression). Another provider would refuse it too.	No failover. Result `failed`.
`unavailable`	Definitely not accepted: 503, 429 / quota, bad API key.	Fail over to `EMAIL_FALLBACK` once, immediately.
`ambiguous`	Can’t tell if it was accepted: timeout (15 s per attempt), any network-level error, 500 / 502 / 504, or a 5xx with a non-JSON body (a proxy error page, which the SDK reports as `application_error`).	No failover — the first provider may have sent it. Deferred for a delayed retry on the same idempotency key when a defer hook is configured; otherwise recorded as `failed` with `error_kind = 'ambiguous'`.

SES (classifySesError, providers/ses.ts): throttling / quota / sending paused or suspended / our credentials refused (403, expired token) / an unverified sender identity (MailFromDomainNotVerifiedException, MessageRejected … not verified — Resend may have the domain) are unavailable; 503 is unavailable, other 5xx ambiguous; other 4xx (bad address, rejected content, unknown configuration set) rejected. aws4fetch’s own retries are off (retries: 0) — SES has no idempotency key, so a hidden retry could double-send. SES ignores the idempotency key; it is still logged.

Resend network errors are ambiguous, not unavailable. The Resend SDK (v6) turns every fetch-level failure — connection refused, DNS failure, reset — into an error with statusCode: null, so the library can’t tell “never connected” from “dropped after acceptance”. Those are classified ambiguous and deferred, not failed over. Only an explicit refusal from Resend’s API (503, 429, quota, bad key) proves the message wasn’t accepted; a 500 or a gateway error (502 / 504) can happen after the message was queued.

Timeouts don’t cancel the request. The Resend SDK takes no AbortSignal, so when the 15 s timeout fires the library stops waiting but the underlying HTTP request may still complete and send the message. That is exactly why a timeout is ambiguous.

Every send carries an idempotency key (Resend dedupes on it for 24 h), and the same key is reused on a deferred retry.

Delayed retry queue. Where EMAIL_DEFER_QUEUE_URL is set (the API Lambda), an ambiguous send is put on SQS accessible-pdf-production-email-deferred and returns deferred (ok: true). The accessible-pdf-production-email-deferred Lambda (src/email-deferred-handler.ts) re-sends it through the library with the same idempotency key and provider pin, after 2 min; an ambiguous retry re-queues after 10 min, then 15 min, then gives up and is logged as failed (… defer failed). Each attempt writes its own email_log row, all sharing the idempotency_key. A retry that fails or is suppressed is final (not redriven); only a crash goes back to SQS and, after 3 receives, to the -email-deferred-dlq. Messages over ~250 KB (large attachments) can’t be queued and fail at once. SES has no idempotency key, so an ambiguous SES send can very occasionally arrive twice after a retry. The EC2 API enqueues with its instance role (#3648). Check email_log before re-sending anything by hand.

Sender domain

theaccessible.org and gevell.com are synonyms (every mailbox exists on both), but we no longer send from theaccessible.org. The library rewrites the From address on theaccessible.org or any subdomain (e.g. [email protected]) to the same local part @gevell.com, keeping the display name. Reply-To and recipients are never rewritten: we still receive on theaccessible.org, and convert-by-email depends on replies reaching [email protected] (gevell.com MX is Google Workspace, not the SES intake). The convert-by-email senders (workers/email-intake replies, the workers/batch completion message) send as Gevell PDF <[email protected]> with Reply-To: [email protected], so answering with another PDF starts a new conversion. The API hosts are unrelated to sender rewriting. They moved to api.gevell.com, api-pdf.gevell.com and worker.gevell.com on 2026-09-24; the old *.theaccessible.org hosts stay up as fallbacks until about 2026-10-26, so use the gevell.com hosts everywhere.

Environment

workers/api (Node API on EC2 and the Lambda — keep them in parity):

Name	Purpose
`EMAIL_PRIMARY`	Provider tried first: `resend` (default) or `ses`.
`EMAIL_FALLBACK`	Provider tried when the primary is `unavailable`. Unset = no failover. Ignored if equal to the primary.
`RESEND_API_KEY`	Resend key. Required whenever `resend` is primary or fallback.
`SUPABASE_URL` + `SUPABASE_SECRET_KEY` / `SUPABASE_SERVICE_ROLE_KEY`	Enables the send log and the suppression check. Without them the library still sends, but logs nothing and suppresses nothing (and warns once per process).
`ALERT_EMAIL`, `TELEGRAM_BOT_TOKEN`, `TELEGRAM_CHAT_ID`, `PROXY_SHARED_SECRET`	Existing — used by `/internal/email-alert` and the SES webhook → `dispatchAdminAlert`.
`AWS_REGION` / `SES_REGION`, `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` (+ `AWS_SESSION_TOKEN`)	SES region (default us-east-1) and credentials. Unset keys fall back to the EC2 instance role. The API Lambda, EC2 anchor and batch roles have `ses:SendEmail` on `gevell.com` (`infra/cdk/lib/ses-send-policy.ts`).
`EMAIL_SES_CONFIGURATION_SET`	Optional default configuration set for library SES sends. Unset = the identity’s default (`gevell-transactional`). Named apart from `SES_CONFIGURATION_SET`.
`SES_CONFIGURATION_SET`	Outreach configuration set (default `accessible-outreach`).
`EMAIL_DEFER_QUEUE_URL`	SQS queue for delayed same-key retries of ambiguous sends. Set by CDK on the API Lambda and the consumer; unset on the EC2 API until #3648.
`SES_WEBHOOK_SECRET`	`X-Webhook-Secret` the EventBridge connection sends to `/api/webhooks/ses-events`. SSM `/accessible-pdf/shared/SES_WEBHOOK_SECRET`, created by `infra/ses/setup-ses-events.sh`.

workers/email-intake (Lambda) and workers/batch read the same email variables from process.env (the intake loads /accessible-pdf/shared from SSM at cold start, which carries RESEND_API_KEY).

Supabase edge function secrets (supabase secrets set …):

Name	Purpose
`RESEND_API_KEY`	Bearer key for the list-emails poll.
`RESEND_WEBHOOK_SECRET`	Svix signing secret (`whsec_…`) from the Resend webhook config.
`PROXY_SHARED_SECRET`	Gates the poll (must equal the cron’s `x-proxy-secret`) and authenticates the alert POST to workers/api.
`INTERNAL_ALERT_URL`	`https://api-pdf.gevell.com/internal/email-alert`
`INTERNAL_SEND_EMAIL_URL`	`https://api-pdf.gevell.com/internal/send-email` (used by `audit-digest`)

SUPABASE_URL / SUPABASE_SERVICE_ROLE_KEY are injected automatically. The pg_cron job reads proxy_shared_secret from Supabase Vault; it must equal the edge functions’ PROXY_SHARED_SECRET (already mirrored for the email-notify-sweep job).

Tables (migration 332)

Both are service-role only (RLS on, no policies).

`email_log`

One row per send through the library (a send that failed over writes a single row with failed_over = true, not one row per provider attempt), keyed for events on (provider, provider_message_id). Two writers, either of which may arrive first — both upsert:

record_email_send(p_entry jsonb) — the library: category, source, from/to, subject, outcome, error_kind, failed_over, cost_usd, idempotency key.
record_email_event(p_provider, p_provider_message_id, p_event, …) — webhooks and pollers. Returns (send_id, status, alert).

status only moves forward by rank (email_status_rank()): deferred → sent → delivery_delayed → delivered → opened → clicked; suppressed, failed, bounced, complained are terminal. A late delivered never overwrites bounced. last_event keeps the raw latest provider event even when it doesn’t move status (e.g. scheduled).

alerted_at is claimed atomically by record_email_event the first time a row reaches a provider-reported failure (bounced / complained / failed / suppressed-by-provider), so the webhook and the poll can both observe a failure and only one alert fires. If the alert POST fails, the edge function sets alerted_at back to NULL so the next observation (a Resend webhook redelivery — the webhook returns 502 — or the next poll) alerts again.

POST /internal/email-alert reports what dispatchAdminAlert did:

Dispatcher result	Response	Meaning
`delivered`	200 `{ success: true, delivery: 'delivered' }`	At least one wanted channel (email / Telegram) succeeded.
`disabled`	200 `{ success: true, delivery: 'disabled' }`	Alert type turned off on `/admin/notifications`. Deliberate; not retried.
`unconfigured`	503 `ALERT_CHANNEL_UNCONFIGURED` + error log	A wanted channel is missing its config (`RESEND_API_KEY`/`ALERT_EMAIL`, or the Telegram token/chat) and nothing was attempted. The edge function records this as `not_configured`: it releases its claim but the webhook still answers 200 (no Resend redelivery storm); the next poll re-alerts once config is fixed.
`failed`	502	Channels were attempted and none succeeded. The edge function releases its claim and retries.
alert inbox itself failing (`skipped` / `unconfigured` with the recipient = `ALERT_EMAIL`)	200 `{ …, escalated: 'telegram' }`, else 502 / 503	Email is skipped to avoid a loop. The route escalates to a direct Telegram message (ignoring routing). If Telegram fails (502) or isn’t configured (503) it also writes a `fatal` log (flushed immediately) and the claim is released, so the alert is retried.

Library-side failures do not alert. A send the library itself could not complete (error_kind rejected / unavailable / ambiguous) is recorded in email_log with status failed but raises no admin alert; the calling code logs it.

`email_suppressions`

Addresses the library will not mail (checked before every send; the lookup fails open so a database outage never blocks mail).

How a suppression affects a send depends on its category:

Category	Complaint	Hard bounce	Manual
`auth`	Ignored — the recipient asked for this mail (a sign-in link).	Blocks.	Blocks.
`alert`	Ignored — our own operator inbox.	Ignored — an automatic suppression must never silence the alert inbox.	Blocks.
everything else	Blocks.	Blocks.	Blocks.

Blocked recipients are dropped and the rest are sent to. When every To recipient is blocked nothing is sent and the result is status: 'suppressed', ok: false, error: 'Recipient suppressed: hard bounce' (etc.), in every category, so no caller reports it as sent. The relay routes (/api/send-email, /internal/send-email) return 422 EMAIL_SUPPRESSED for it (not retryable) and 500 EMAIL_FAILED for a failure; audit-digest counts a 422 as a skipped recipient, not a failed run.

`reason`	Added by
`hard_bounce`	`email.bounced` webhook with `data.bounce.type = 'Permanent'`. `Temporary` / `Undetermined` bounces are logged and alerted but not suppressed.
`complaint`	`email.complained` (webhook or poll).
`manual`	Inserted by hand.

Only single-recipient mail is suppressed automatically: a Resend event covers the whole email and doesn’t say which of several recipients bounced. The poll never suppresses on a bounce because the list API carries no bounce type.

Resend event mapping

The webhook strips the email. prefix (email.delivery_delayed → delivery_delayed); the list API’s last_event has none. email.suppressed (Resend’s own account-level suppression list) maps to our suppressed status. email.received (inbound mail) and non-email events (contact.*, domain.*) are acknowledged and ignored. Recipients are stored as bare lowercase addresses. The failure reason (bounce.type/subType: message, failed.reason, suppressed.message) goes to email_log.error; the full event is stored in raw.

The poll only calls the RPC for emails that are new to email_log or whose last_event changed, so repeat polls don’t churn rows (or overwrite the webhook’s richer raw payload).

Backfill never alerts. An email the poll sees for the first time (no email_log row) that was sent more than an hour ago and before email logging began (the oldest email_log.created_at) is recorded — status, suppression and all — but raises no alert, and its alert claim is kept so it never alerts later (isBackfill, _shared/email-events.ts). This is what the first poll after migration 332 looks like: up to 500 old emails with every historic bounce in them. Mail sent after logging began always alerts, so a webhook/poller outage can’t turn new failures into silent backfill; emails the library logged or the webhook recorded always alert however old. The poll reports backfilled in its counts.

Truncation. One run reads at most 5 pages (500 emails). If Resend still reports more, the poll returns ok: false, truncated: true (HTTP 200): events older than the cap were not read that run.

Webhook authentication. authenticateSvixRequest (_shared/svix.ts) rejects a missing svix header, a stale/future timestamp (±5 min) or a bad signature with 400 before anything is recorded. A correctly signed event with no email id is acknowledged with 200 and a warning (a redelivery would carry the same payload). The poll’s x-proxy-secret is compared in constant time.

SES setup and event mapping

Identity. gevell.com is an SES domain identity in us-east-1 (created 2026-09-25) with Easy DKIM (RSA 2048) and custom MAIL FROM bounce.gevell.com. Its DNS lives in the gevell.com Cloudflare zone, DNS only (grey cloud):

Type	Name	Content
CNAME	`<token>._domainkey` × 3	`<token>.dkim.amazonses.com` (tokens: `aws sesv2 get-email-identity --email-identity gevell.com`)
MX	`bounce` (10)	`feedback-smtp.us-east-1.amazonses.com`
TXT	`bounce`	`v=spf1 include:amazonses.com ~all`

The root SPF record is unchanged: SPF is checked on the MAIL FROM domain (bounce.gevell.com), which aligns with gevell.com under DMARC aspf=r, and Easy DKIM signs as d=gevell.com, which the strict adkim=s requires.

Configuration sets and events — infra/ses/setup-ses-events.sh (idempotent; the source of truth, since SES config lives outside CDK): gevell-transactional (the identity’s default; no open/click tracking) and accessible-outreach (adds OPEN / CLICK), both with account-level suppression of bounces and complaints, publish to the default EventBridge bus. Rule gevell-ses-events (source: aws.ses) → API destination gevell-ses-events → POST https://api-pdf.gevell.com/api/webhooks/ses-events with X-Webhook-Secret, retried for up to 24 h.

The webhook (routes/ses-events.ts → services/ses-email-events.ts) records every event with record_email_event, then does outreach bookkeeping (email_sends / prospects / unsubscribes) when the message is a campaign send. If recording fails it answers 500 so EventBridge retries.

SES event	email_log event	Suppresses
Send	`sent`	—
Delivery	`delivered`	—
DeliveryDelay	`delivery_delayed`	—
Bounce, Permanent	`bounced`	the bounced recipients (`hard_bounce`)
Bounce, Transient / Undetermined	`bounce_transient` (unranked: status unchanged, no alert)	—
Complaint	`complained`	the complainants (`complaint`)
Reject, Rendering Failure	`failed`	—
Open / Click (outreach only)	`opened` / `clicked`	—

Failures alert once, in process (services/email-delivery-alert.ts, shared with /internal/email-alert); a claim whose alert didn’t go out is released so the next event re-alerts. Outreach events never alert (mail tagged with the outreach configuration set): cold-mail bounces are expected, and they still suppress the address.

Out-of-band senders (intentionally bypass the library)

These alert on failures of the very system the library depends on (the API, the database, the budget, CI), so they keep their own direct Resend call and must keep working when the API is down. They are allow-listed in workers/api/src/__tests__/email-provider-guard.test.ts (OUT_OF_BAND). Their mail still shows up in email_log via delivery events, with category unknown.

Sender	What it alerts on
`workers/status/src/alerts.ts`	status.gevell.com target down/up
`workers/dmarc/src/alerts.ts`	DMARC failures in aggregate reports
`workers/gemini-alert/src/index.ts`	Gemini budget-email spend jumps
`infra/gcp/budget-killswitch/index.js`	GCP budget kill-switch
`scripts/cost-spike-monitor.mjs`, `scripts/cost-trickle-monitor.mjs`	cost spikes / slow cost leaks (Google request metrics only; not Bedrock)
`scripts/spend-digest-monitor.mjs`	spend digest
`packages/logger/src/alerts.ts`	`error`/`fatal` log alerts (`EmailAlertTransport`)
`packages/github-action/scripts/regression-email.cjs`	accessibility regression emails from the GitHub Action
`workers/phone/src/voicemail.ts`	voicemail notifications to the operator

supabase/functions/audit-digest (customer audit digests) sends through the API’s /internal/send-email, so it gets the send log, suppression and failover. pg_cron runs it (migration 333): audit-digest-daily at 14:00 UTC every day (?cadence=daily) and audit-digest-weekly at 14:00 UTC on Mondays (?cadence=weekly). It is deployed with --no-verify-jwt and answers 403 unless x-proxy-secret matches PROXY_SHARED_SECRET. A run returns 500 when INTERNAL_SEND_EMAIL_URL is unset and 502 (ok: false, with sent / suppressed / failed counts) when any digest failed; one failed recipient doesn’t stop the others. Recipients come from audit_digest_subscriptions (migration 104). Nothing writes that table yet, so the runs send nothing until subscriptions exist.

Every other sender uses the library; the guard’s PENDING list is empty since phase 2. Prospect outreach (services/prospect-sender.ts, campaign test sends) sends from [email protected], pinned to SES.

Tests

packages/email: npm test -w packages/email (library, Resend and SES adapters, MIME builder, AWS credentials, Supabase hooks).
workers/api: services/ses-email-events.test.ts, routes/ses-events.test.ts, services/prospect-sender.test.ts; workers/batch completion-email.test.ts; workers/email-intake handler.test.ts.
Edge functions: npx vitest run --root supabase/functions — Svix verification, the webhook handler (_shared/resend-webhook-handler.ts), delivery-event recording and backfill. Runs in CI as the “Edge function tests” step of .github/workflows/test.yml.
SQL: psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f supabase/tests/email_log.test.sql (manual; rolls back).

Deploy checklist

Apply migration 332 (done in prod 2026-09-25, verified: tables, grants, poll cron, alert preference) (20260925010000_332_email_log.sql) via the Supabase MCP / SQL editor — migrations are not auto-applied. It creates both tables, the RPCs, the email_delivery_failure alert-preference row and the resend-events-poll pg_cron job.
Edge function secrets: supabase secrets set RESEND_API_KEY=… RESEND_WEBHOOK_SECRET=… PROXY_SHARED_SECRET=… INTERNAL_ALERT_URL=https://api-pdf.gevell.com/internal/email-alert INTERNAL_SEND_EMAIL_URL=https://api-pdf.gevell.com/internal/send-email, then redeploy audit-digest
Deploy the edge functions without JWT verification (Resend and pg_net can’t send a Supabase JWT; they authenticate with the Svix signature and the proxy secret): supabase functions deploy resend-webhook --no-verify-jwt supabase functions deploy resend-events-fetcher --no-verify-jwt supabase functions deploy audit-digest --no-verify-jwt (checks x-proxy-secret)
Resend dashboard → Webhooks: add endpoint https://vuvwmfxssjosfphzpzim.supabase.co/functions/v1/resend-webhook, subscribed to all email.* events. Copy its signing secret into RESEND_WEBHOOK_SECRET (step 2) and redeploy resend-webhook if you set a placeholder first.
Deploy workers/api to both runtimes (Node API on EC2 and the Lambda) with EMAIL_PRIMARY / EMAIL_FALLBACK set as intended.
Deploy apps/admin (npm run deploy) for /admin/email-logs, and apps/web for the share dialog’s sent/unsent recipient count (the API change is backward compatible; the old dialog just keeps showing the typed count).
Expect a quiet first poll. The first resend-events-poll run after the migration records up to 500 recent emails. Historic failures in it (sent more than an hour before the poll and before the first email_log row) are recorded as bounced / complained etc. and appear on /admin/email-logs, but do not alert and are not re-alerted later. The poll response shows them as backfilled. Review them on the page rather than waiting for alerts.
Verify with a library send — submit the contact form (gevell.com) or the in-app feedback form — then confirm a row with its source (api:contact / api:feedback) reaches delivered on /admin/email-logs (or query email_log) within a minute, and force a poll (below) to confirm the cron path. Don’t use a normal sign-in link: that goes through GoTrue, not the library.

Phase 2 (SES)

SES identity gevell.com (done 2026-09-25) and its five DNS records (table above). Wait for aws sesv2 get-email-identity --email-identity gevell.com to show VerifiedForSendingStatus: true and MAIL FROM SUCCESS.
Deploy workers/api (Lambda via deploy-aws; the EC2 anchor auto-deploys) — the CDK deploy adds the SES send grants.
Run infra/ses/setup-ses-events.sh — creates the webhook secret in SSM, both configuration sets, the EventBridge rule and API destination, and the identity’s default set. Then force a Lambda cold start (or wait) so it loads SES_WEBHOOK_SECRET.
Deploy workers/email-intake (CDK, same deploy-aws run).
Verify: a campaign test send from /admin (outreach pinned to SES) should reach delivered in email_log with provider ses within a minute.

Phase 3 (SES primary, Resend fallback)

EC2 API on the instance role (#3648): without it the anchor signs with an IAM user that has no SES grant, so every EC2 send would fail over.
Deploy the alarms (monitoring stack, deploy-aws).
Test failover: send through the library (sendEmailWithEnv) with EMAIL_PRIMARY=ses, EMAIL_FALLBACK=resend and bogus AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY to [email protected]. SES answers InvalidClientTokenId (unavailable), so the send fails over. Don’t use a bad region: a network error is ambiguous and defers instead. Expect one EMAIL_FAILOVER line and an email_log row with provider resend and failed_over = true. (Done 2026-09-26.)
Flip with the two SSM parameters (Operating → Switch providers) and restart the senders.
Verify: POST /api/send-email with x-proxy-secret to [email protected] on both api-pdf.gevell.com (Lambda) and api.gevell.com (EC2) reaches delivered with provider ses, and no EMAIL_FAILOVER lines appear. (Flipped and verified 2026-09-26.)
Supabase Auth SMTP → SES (next section), keeping Resend SMTP as the documented manual fallback.

Supabase Auth SMTP (SES)

GoTrue sends sign-in links, password resets and invites over its own SMTP settings, so it does not get the library’s automatic failover.

Run infra/ses/setup-supabase-smtp.sh. It creates IAM user gevell-supabase-smtp (can only ses:SendRawEmail from gevell.com) and stores its SES SMTP credentials in SSM /gevell/supabase-smtp/username and /password. It never prints them.
Supabase dashboard → Authentication → Emails → SMTP Settings: host email-smtp.us-east-1.amazonaws.com, port 587, sender [email protected], and paste the username and password from SSM (aws ssm get-parameter --name /gevell/supabase-smtp/password --with-decryption --query Parameter.Value --output text | pbcopy). Note the Resend values first so you can put them back.
Send yourself a sign-in link. Because gevell.com uses the gevell-transactional configuration set by default, the delivery shows up in email_log as provider ses, category unknown (GoTrue mail doesn’t go through the library, so there is no send row).

Fallback (manual): if SES is down or the account is paused, switch the Supabase SMTP settings back to Resend: host smtp.resend.com, port 465, username resend, password = the Resend API key.

Rotate: ROTATE=1 infra/ses/setup-supabase-smtp.sh, paste the new values, test a sign-in link, then delete the old key with the command it prints.

Operating

View: admin.gevell.com/admin/email-logs — status summary, filters (status, category, provider, exact recipient, subject), per-row provider, failover flag, source and cost; suppression list with Remove.
Un-suppress: the Remove button calls DELETE /api/admin/email-suppressions/:email and writes an email_unsuppress entry to admin_audit_log. Only do it when the address was fixed or the recipient asked for mail again.
Force a poll: curl -X POST -H "x-proxy-secret: $PROXY_SHARED_SECRET" https://vuvwmfxssjosfphzpzim.supabase.co/functions/v1/resend-events-fetcher
Inspect the cron: SELECT jobname, schedule FROM cron.job WHERE jobname = 'resend-events-poll'; SELECT * FROM cron.job_run_details WHERE jobid = (SELECT jobid FROM cron.job WHERE jobname = 'resend-events-poll') ORDER BY start_time DESC LIMIT 5;
Replay a failure alert: clear the claim so the next observation re-alerts: UPDATE email_log SET alerted_at = NULL WHERE provider = 'resend' AND provider_message_id = '…'; (SES events arrive only once; replaying one means re-sending it through EventBridge, so check the row’s raw instead.)
SES event delivery: EventBridge console → Rules → gevell-ses-events (invocations / failed invocations), or CloudWatch metric AWS/Events FailedInvocations for the rule.
Mute or reroute alerts: toggle email_delivery_failure on /admin/notifications.
Switch providers: every sending runtime (API Lambda, email-deferred Lambda, email-intake, EC2 API and course-map-api) loads /accessible-pdf/shared from SSM at start, so one pair of parameters sets them all. No call-site change is needed:
Terminal window
```
aws ssm put-parameter --overwrite --type String --name /accessible-pdf/shared/EMAIL_PRIMARY --value ses
aws ssm put-parameter --overwrite --type String --name /accessible-pdf/shared/EMAIL_FALLBACK --value resend
```
Then restart the senders so they reload SSM: redeploy the anchor (gh workflow run deploy-anchor.yml --ref main -f image_tag=<sha> -f posture=unchanged) and force Lambda cold starts (any configuration update, e.g. bump a description). Roll back by setting EMAIL_PRIMARY=resend the same way.
Alarms (SNS topic accessible-pdf-production-alarms, emailed to the alert address; infra/cdk/lib/email-alarms.ts):
- …-email-failovers: 3+ EMAIL_FAILOVER log lines in 15 minutes across the API Lambda and /gevell/anchor log groups. The primary is probably down or rejecting; check the lines for the error.
- …-ses-bounce-rate (> 2%) and …-ses-complaint-rate (> 0.05%): SES reviews the account at 5% / 0.1%. Look at recent bounces in email_log and the suppression list; pause campaigns first.
- …-email-deferred-dlq: a message exhausted its delayed retries. Check email_log for the recipient before re-sending by hand.