Disaster Recovery Plan

Organization: Strategic Habits Consulting, LLC (DBA Gevell) Plan owner: Larry Anglin, CTO Review cycle: Annually, and after any real recovery event or major architecture change Related: Business Continuity Plan, Incident Response Plan

1. Purpose and scope

This plan explains how Gevell restores its production service after a disruptive event: a provider outage, data loss, loss of a compute instance, credential compromise, or ransomware. It covers the customer-facing Gevell apps (pdf.gevell.com and the other product sites), their APIs, and the data they depend on.

All customer-data processing runs in cloud services (AWS, Cloudflare, Supabase). Gevell does not operate its own data center or on-premises servers for customer data.

How to read this plan. Each item is marked as one of:

In place: exists and operates today.
Target: a goal this plan commits to. Some targets have not been proven by a test yet.
Planned: not done yet. Listed in Actions required.

2. Architecture (what we are recovering)

Layer	Component	Where it runs	State held	Status
Frontends	Static sites (pdf.gevell.com, audit.gevell.com, gevell.com, etc.)	Cloudflare Pages / Workers	None. Built from source in GitHub	In place
Serverless API	Hono API on AWS Lambda behind API Gateway (`api-pdf.gevell.com`), plus supporting Lambdas	AWS us-east-1	None. Config in AWS CDK (Git) and SSM Parameter Store	In place
Node API + processing (“anchor”)	Node API (`api.theaccessible.org`), Docker processing sidecars, and the single job-scheduler leader	AWS EC2, us-east-1: a fixed-size Auto Scaling group (min = max = 1) across 2 Availability Zones. Encrypted EBS. IMDSv2 required. No inbound ports: traffic arrives through an outbound-only Cloudflare Tunnel whose token is held in SSM. Admin access through AWS SSM Session Manager (no SSH, no key pair). Container images from private Amazon ECR; logs to CloudWatch Logs	Documents only while processing. No durable state. Secrets pulled from SSM (KMS-encrypted SecureStrings) into memory at boot	In place
Job queues	Amazon SQS pipeline, ingest and bulk queues, each with a dead-letter queue (DLQ)	AWS us-east-1	Job pointers (IDs and storage keys). Never document content. 7-day retention	In place
Database and auth	Supabase Postgres + Supabase Auth	Supabase, AWS us-east-2	Accounts, job and document metadata, billing ledger, audit logs	In place
Document storage	Cloudflare R2	Cloudflare, US region	Customer uploads and converted outputs	In place
Transient AWS storage	S3 upload-staging bucket for malware scanning (1-day lifecycle). DynamoDB key-value table (point-in-time recovery enabled in production)	AWS us-east-1	Short-lived	In place
AI inference	AWS Bedrock (Anthropic Claude), zero data retention	AWS	None	In place
Payments	Stripe	Stripe	Stripe is the system of record	In place
Email	Resend (outbound transactional email)	Resend	None	In place
Source code	GitHub (private repositories)	GitHub	All code, infrastructure-as-code, migrations	In place
Monitoring	Health-monitor daemons (email + Telegram), Amazon CloudWatch Logs and alarms, AWS GuardDuty and CloudTrail (alerts to the CTO), status.gevell.com (endpoint checks every minute and the public status page)	Mixed	Operational logs	In place

Design properties that help recovery:

Stateless compute (in place). No compute tier holds durable customer state. Any instance can be thrown away and replaced from Git, container images and SSM.
Self-healing anchor (in place). If the anchor instance fails a health check, or its Availability Zone fails, the Auto Scaling group replaces it automatically in a healthy AZ. The replacement pulls secrets from SSM and reconnects the tunnel.
Queue-based work (in place). If the processing host dies mid-job, the SQS message becomes visible again and is retried once the host is replaced. After 3 failed attempts it moves to a DLQ, which raises a CloudWatch alarm.
Infrastructure as code (in place). AWS resources are defined in AWS CDK (infra/cdk/). The database schema is defined in versioned migrations.

3. Recovery objectives (RTO / RPO), target state

RTO (Recovery Time Objective) is how long it takes to restore the service. RPO (Recovery Point Objective) is how much data can be lost, measured in time.

All figures are targets. None has been confirmed by a formal restore test yet (§7). “Provider-dependent” means recovery waits on the provider. Our own action in that case is to communicate and to fail over where a failover path exists.

System	RPO target	RTO target	Basis / notes
Frontends (Cloudflare Pages/Workers)	0 (rebuilt from Git)	4 h to redeploy. Provider-dependent if Cloudflare itself is down	Source in GitHub and the nightly GitHub backup
Lambda API (`api-pdf.gevell.com`)	0 (stateless)	4 h (roll back to the prior commit, or `cdk deploy`)	Secrets restored automatically from SSM
Anchor EC2 (Node API + processing): instance or AZ failure	0. In-flight jobs return to SQS	1 h (automatic Auto Scaling group replacement plus boot)	Queued work waits in SQS (7-day retention) during replacement
Anchor EC2: bad deploy or corrupted configuration	0	2 h (redeploy the previous image or commit, then refresh the instance)
AWS us-east-1 regional outage	0 for stateless tiers. SQS messages are delayed, not lost, if the region recovers	Provider-dependent. Best effort 72 h to redeploy into a second region if the outage lasts more than 24 h	Single region by design. Second-region redeploy is not rehearsed
Supabase Postgres + Auth	24 h (daily provider backup). Improves to minutes if point-in-time recovery (PITR) is enabled (Planned)	8 h (restore from backup). Provider-dependent for a platform outage	Restore time grows with database size. Not yet tested
Customer documents (Cloudflare R2)	Hardware failure: covered by R2 durability. Accidental or malicious deletion: no independent copy today	Provider-dependent	Customers keep their source files. Outputs can be regenerated by re-running conversion. Independent copy is Planned
SQS queues	Messages kept 7 days (DLQ 14 days)	Provider-dependent	Queue definitions recreated by CDK
Secrets and config (SSM Parameter Store)	24 h (with the planned encrypted export)	4 h	After a compromise, rotate rather than restore
Source code (GitHub)	24 h (nightly backup). Near 0 in practice through working clones	24 h to re-host and restore CI/deploy	See §5.7
Stripe / Resend	Provider is system of record	Provider-dependent	We hold no card data. Email may be delayed
Logs (CloudWatch Logs, application log store)	Loss of up to 30 days of operational logs is acceptable	Best effort	Not customer data. Evidence is exported first during incidents (IR plan §8)

Context: Our SLA is 99.5% monthly availability, or about 3.6 hours of downtime per month. A full-RTO event in several rows above would breach the SLA for that month. Service credits apply under the customer agreement.

4. Roles

Role	Person	Responsibilities in DR
Recovery lead	Larry Anglin (CTO). The only person with production access	Declares a disaster, performs all technical recovery, verifies restoration
Communications lead	Laura Hopkins (Chief Accessibility Officer). No production access	Customer and institution communication and tracking customer impact (BCP §6)
Backup technical responder	Designated. Known to Larry and Laura; not named in this document	Performs recovery using the sealed break-glass kit if Larry is unavailable (BCP §5)

5. Recovery procedures by scenario

Every scenario starts the same way:

Open an incident record (IR plan §5).
Post a status update on status.gevell.com.
Laura notifies affected customers per BCP §6.
If there is any sign of compromise, follow the IR plan first and preserve evidence before rebuilding.

5.1 Lambda / API Gateway failure, or AWS regional outage

Confirm the scope. Check the AWS Health Dashboard, CloudWatch 5xx alarms and health-monitor alerts. Decide whether the cause is us (bad deploy or configuration) or AWS.
If we caused it (bad deploy): revert the PR, or redeploy the last known-good commit from main. Verify https://api-pdf.gevell.com/health.
If the stack or account is damaged: run cd infra/cdk && npx cdk deploy using SSO credentials. If SSO is unavailable, use the break-glass `cdk-deployer` AWS profile (`docs/admin/access/aws-access.md`).
If the region is down: both the Lambda API and the anchor are affected, so the whole processing service is down.
- Wait on AWS and keep customers updated.
- If the outage is forecast to last more than 24 h, consider deploying the CDK stacks to us-east-2, copying the SSM parameters, confirming that the Bedrock models are available there, and repointing the tunnel and API hostnames. This has not been rehearsed. Treat 72 h as best effort.
Validate: smoke-test upload → convert → download on pdf.gevell.com, and confirm the SQS queue age is back to normal.

5.2 Supabase outage or data loss

Outage (platform down): check status.supabase.com and open a Supabase support ticket. Our APIs fail closed (no data is served without the database). Communicate, wait, and verify after recovery.
Data loss or corruption (bad migration, erroneous bulk update, malicious change):
1. Stop writes. Disable the affected feature, or pause processing by scaling the anchor’s queue consumer to 0.
2. Identify the last known-good time.
3. With PITR enabled (Planned): restore to the minute before the event through the Supabase dashboard. Today (no PITR): restore the most recent daily backup. Anything written after that backup is lost. Reconcile payments from Stripe (the system of record) and job state from R2 object listings where possible.
4. If only some tables are affected, restore into a new project and copy back only the affected rows, instead of rolling back the whole database.
5. Re-apply any migrations that came after the backup, if needed. Verify row-level security (RLS) policies and that auth sign-in works.
Before any risky migration (in place, by practice): take a manual supabase db dump and store it in R2.

5.3 Cloudflare / R2 outage

Edge or Pages outage: frontends and the tunnels into the Node API are unavailable, so the whole customer-facing service is down. This depends on the provider (accepted risk). Communicate by direct email, because the status page is also hosted on Cloudflare.
R2 outage: uploads and downloads fail. Jobs in progress retry through SQS and then land in the DLQ. After recovery, re-drive DLQ messages and check for stuck chunk_jobs / large_conversion_jobs conversion jobs.
R2 data deletion: today there is no independent copy. Recovery is to (a) regenerate outputs from source files that still exist and (b) ask affected customers to re-upload sources. Deletion by a compromised credential is also an incident (IR plan).

5.4 Anchor EC2 instance failure, or rebuild

Automatic:
- The Auto Scaling group sees the failed health check and launches a replacement, in another AZ if needed.
- Watch the ApproximateAgeOfOldestMessage metric in CloudWatch and the anchor ASG activity history.
If the replacement fails to become healthy:
1. Connect with SSM Session Manager and check the cloudflared and docker compose status.
2. Roll back to the previous image or commit.
Scheduler single-leader rule: only the anchor runs the job-scheduler leader. Never run two leaders during recovery (see migration runbook §5).
Verify https://api.gevell.com/health and complete one test conversion.

5.5 Credential compromise

Follow the IR plan (at least SEV2). Revoke first, then investigate. Use the secrets inventory in docs/admin/security/secrets-inventory-and-rotation.md.
Priority order:
1. Cloud admin identities: AWS SSO / root, Cloudflare, Supabase, GitHub, Stripe.
2. Supabase service-role key.
3. R2 / S3 keys.
4. Cloudflare Tunnel tokens.
5. Third-party API keys (Resend, Mathpix, Telegram).
Update the new values in SSM, then refresh the anchor instance so it picks them up.
Review the provider audit logs for actions taken with the credential: AWS CloudTrail, Cloudflare audit log, Supabase logs, GitHub audit log, Stripe events.
If data may have been accessed, move to breach assessment and notification (IR plan §6).

5.6 Ransomware or malware on the processing instance (anchor)

Isolate:
1. Put the instance in Standby or detach it from its ASG, so it is not terminated before evidence is captured.
2. Swap its security group for one that denies all traffic.
3. Revoke the Cloudflare Tunnel token.
Preserve: snapshot its EBS volumes for evidence (IR plan §8). Then terminate the instance.
Rotate: every secret the instance role could read from SSM. Assume all of them are compromised.
Rebuild: the ASG launches a clean replacement from the base image, Git and the rotated SSM secrets. Nothing is restored from the infected instance.
Assess exposure: processing instances hold customer documents in transit. Decide which documents were present during the exposure window, and treat the event as a potential breach (IR plan §6).
Backups are out of reach of processing instances:
- GitHub backups sit in an R2 bucket with Bucket Lock (write-once retention).
- The instance roles have no credentials for the backup stores (to verify, DR-9).

5.7 Loss of GitHub (outage, or account loss / lockout)

Short outage: wait. Manual deploys can run from a working clone. Replacement anchor instances must not depend on GitHub at boot. They should start from prebuilt images in Amazon ECR (DR-6).
Account loss or long outage:
1. Restore the repositories from the nightly backup in the R2 github-backups bucket (8-day daily, 35-day weekly and 370-day monthly tiers, with Bucket Lock), or from local clones.
2. Push them to a new organization or another Git host.
3. Re-create the GitHub Actions secrets and the AWS OIDC deploy-role trust.
4. Re-register the CI runners. Target: 24 h.

6. Backup inventory

What	Mechanism	Frequency	Retention	Location	Encryption	Monitored	Status
Supabase Postgres (all app data + auth)	Supabase-managed backups	Daily	7 days (Supabase Pro plan)	Supabase (AWS)	Provider-managed at rest	Supabase dashboard	In place
Supabase PITR	Supabase add-on	Continuous	Per add-on setting	Supabase	Provider-managed	—	Not enabled (Planned)
Independent database export	`pg_dump` to storage we control (R2 or B2)	Nightly	30 days	—	Encrypted	Backup-status panel	Planned. Today, manual dumps are taken only before risky migrations
DynamoDB key-value table	AWS point-in-time recovery	Continuous	35 days	AWS	AWS-managed	—	In place (production)
Customer documents (R2)	Provider durability only	—	Until customer deletes	Cloudflare R2	Provider-managed at rest	—	In place. No independent copy
GitHub repositories (all)	Nightly backup job → Cloudflare R2 (write-once Bucket Lock)	Nightly	8 d / 35 d / 370 d (GFS) with Bucket Lock	Cloudflare R2	Provider-managed at rest	Backup-status panel + email	In place
Runtime secrets and config	AWS SSM Parameter Store (SecureString, KMS)	On change	Parameter history	AWS	KMS	—	In place
Encrypted export of SSM parameters	Export to an encrypted restic repository	Nightly	30 days	Backblaze B2	Client-side AES-256	Backup-status panel	Planned. Protects against loss of the AWS account
Anchor images	Container images in private Amazon ECR (last 10 kept)	Every build	10 images	AWS	AWS-managed	—	In place
AWS infrastructure definition	AWS CDK code	Every change	Git history	GitHub + backup	—	CI	In place
Database schema	Versioned migrations	Every change	Git history	GitHub + backup	—	CI drift check	In place

Every backup job reports to an admin backup-status panel. A job not heard from in 36 hours is flagged, and the nightly audit emails the operator (docs/admin/backup-status-reporting.md).

7. Annual restore test

Frequency: At least once every 12 months, and after major changes to backup or compute architecture. Owner: Larry Anglin. Recorder: Laura Hopkins.

Status: no formal restore test has been performed and recorded under this plan yet.

#	Test	Pass criteria
R1	Restore the latest Supabase backup into a scratch project (never over production)	Completes. Key-table row counts are within 24 h of production. RLS policies are present. A test user can sign in. Time recorded
R2	(After PITR is enabled) PITR restore to a chosen timestamp in a scratch project	Data matches that timestamp
R3	Restore one repository from the R2 GitHub backup and build it on a clean machine	Build succeeds
R4	Restore the encrypted SSM export to a temp location	Secrets are recoverable. Decrypted material is removed afterwards
R5	Terminate the anchor instance on purpose	The ASG replaces it. `/health` returns ok and a test conversion completes within the 1 h RTO
R7	Lambda API redeploy of a prior commit	Health ok. Time recorded

Clean up scratch projects, instances and decrypted material afterwards. Any target missed gets an owner and a date.

Restore test log (template)

Date	Test #	Performed by	Backup / snapshot used	Start	End	Elapsed vs RTO	Data point restored vs RPO	Result (Pass/Fail)	Issues found	Follow-up (owner, due)
YYYY-MM-DD	R1

8. Plan maintenance

Review annually alongside the BCP and IR tabletop exercises.
Update after any real recovery, restore test, or architecture change.

Actions required before this plan is true

#	Action	Why
DR-1	Build the anchor EC2 deployment. Run the Node API, Docker sidecars and scheduler leader on a fixed-size (min = max = 1) Auto Scaling group across 2 AZs in us-east-1, per `docs/admin/deployment/node-to-ec2-migration-runbook.md`. Required parts: encrypted EBS, IMDSv2, SSM Session Manager and no SSH, no inbound security-group rules, Cloudflare Tunnel token in SSM, `SCHEDULER_ROLE=leader` only on the anchor. Then cut `api.theaccessible.org` over Done 2026-09-24. The anchor runs on EC2 as described in §2	Done
DR-2	~~Enable EBS encryption and require IMDSv2 on the anchor~~ Done 2026-09-24	Done
DR-3	~~Confirm the Supabase plan tier and write the exact backup-retention days into §6~~ Done 2026-09-24. Pro plan, daily backups retained 7 days	Done
DR-4	Decide on and enable Supabase PITR, or keep and state the 24 h database RPO	PITR is not enabled
DR-5	Add a nightly independent encrypted `pg_dump` of production to R2 or B2 (30-day), reporting to the backup-status panel	Removes sole reliance on the provider’s backups
DR-6	~~Build anchor images into ECR, so that a replacement instance boots without GitHub~~ Done 2026-09-24. Anchor images come from private Amazon ECR	Done
DR-7	Add a nightly encrypted export of the SSM parameters to B2	Secrets are otherwise recoverable only while the AWS account is intact
DR-8	(Plan tracked in #3542.) Decide on an independent copy of R2 customer documents (replication with retention lock), or formally accept the risk and document the “regenerate / re-upload” stance	R2 deletion is not recoverable today
DR-9	Confirm that the anchor instance role cannot write to or delete the backup stores (R2 `github-backups`, B2)	Needed for the ransomware claim in §5.6
DR-10	Confirm AWS CloudTrail is enabled with at least 90-day retention	Needed for credential-compromise investigation
DR-11	Run and log the first annual restore test (R1, R3–R7) after migration	No test is recorded. All RTO/RPO targets are unproven
DR-12	Write the us-east-2 regional redeploy steps (including Bedrock model availability), or remove the 72 h best-effort figure	Regional recovery is unrehearsed
DR-13	Renew the GitHub backup token before it expires on 2026-12-05	Otherwise the nightly GitHub backup stops
DR-14	Create the break-glass kit (BCP-2). The backup technical responder was designated 2026-09-23	Until the kit exists, the responder has no way to act if Larry is unavailable
DR-15	Update the other security documents to match this plan. `security-whitepaper.md` §V.1/§V.5 and `hecvat-draft.md` describe an older architecture and say restores are “tested quarterly”, but no test record exists	Consistency in what customers are told