Skip to content
AnglinAI Docs

EC2 anchor: verification and cutover runbook

Four tasks, in order. Tasks 1–3 are verification with production still on .4; task 4 is the cutover. Background and decisions: ec2-migration-plan-2246.md. Anchor files: infra/anchor/README.md.

State at the start (2026-09-24)

.4 (10.1.1.4)EC2 anchor
Serves api.theaccessible.orgyes (tunnel api.theaccessible.org, id 4a92034e…)no
Test hostname—api-anchor.theaccessible.org (tunnel gevell-anchor, id a0272978…)
Takes queue + chunk workyes (both containers)yes (both containers, since 2026-09-24 13:09Z)
Runs singleton timersyes (both containers until task 3)no (follower)
Instance—i-0c1bdfda3c875a300, ASG AccessiblePdfProd-Anchor-AnchorAsgASG4C7EF9B0-bq5T5LrOHcwz

Commands below assume export AWS_PROFILE=accessible-sso and the repo root as the working directory. Anchor shell commands go through SSM (the instance has no SSH):

Terminal window
anchor() {  # run a shell command on the anchor, print its output
  local id; id=$(aws ssm send-command --region us-east-1 --instance-ids i-0c1bdfda3c875a300 \
    --document-name AWS-RunShellScript --parameters "commands=[\"$1\"]" \
    --query Command.CommandId --output text)
  sleep 8
  aws ssm get-command-invocation --region us-east-1 --command-id "$id" \
    --instance-id i-0c1bdfda3c875a300 --query '[StandardOutputContent,StandardErrorContent]' --output text
}

Prerequisite for all four tasks: the startup-reclaim fix (PR “Startup chunk reclaim: only take chunks whose owner stopped heartbeating”) is merged and deployed to both hosts — anchor via task 1 step 1, .4 via task 3. Without it, restarting either host resets chunks the other host is still processing.

Task 1 — Confirm real work runs on EC2

Goal: production jobs are being claimed and completed on the anchor, with no errors or memory trouble.

  1. Deploy current main to the anchor (brings the reclaim fix; posture stays work-on/follower). Wait for Build anchor images to finish for the merge commit, then:
    Terminal window
    gh workflow run deploy-anchor.yml --ref main -f image_tag=$(git rev-parse origin/main) -f posture=unchanged
    If the latest main commit did not touch the image paths, use the most recent SHA that Build anchor images built (gh run list --workflow "Build anchor images" --limit 3).
  2. Let it run for at least 24 hours (the hourly smoke test plus organic jobs).
  3. Which workers claimed chunks. Get the anchor’s container hostnames:
    Terminal window
    anchor 'for c in $(docker ps -q); do docker inspect -f "{{.Config.Hostname}} {{.Name}}" $c; done | grep api-node'
    Then in Supabase SQL (anchor hostnames = the two printed above; everything else is .4): 
    select worker_id, status, count(*), max(completed_at)
    from chunk_jobs
    where created_at > now() - interval '24 hours'
    group by 1, 2 order by 1, 2;
  4. Errors on the anchor — CloudWatch Logs Insights, log group /gevell/anchor, last 24 h: 
    fields @timestamp, @logStream, @message
    | filter @message like /"level":"(error|fatal)"/
    | sort @timestamp desc | limit 50
  5. Memory and OOM kills:
    Terminal window
    anchor 'docker stats --no-stream --format "{{.Name}} {{.MemUsage}}"; journalctl -k --since -24h | grep -ci "out of memory"'

Pass: anchor workers completed chunks (and/or queue jobs) with no failures attributable to the anchor; no new error signatures that .4 doesn’t also show; zero OOM kills; api-node memory well under the 7 GB cap. If it fails: gh workflow run deploy-anchor.yml --ref main -f image_tag=<same sha> -f posture=parallel takes the anchor off the queues immediately; .4 carries on alone.

Task 2 — Large-document memory test

Goal: the anchor’s 6 GB api-node heap (8 GB on .4, #1865) handles the largest documents we see.

  1. Start a memory watch on the anchor (samples every 10 s for 20 minutes):
    Terminal window
    anchor 'nohup sh -c "for i in \$(seq 120); do date +%T; docker stats --no-stream --format \"{{.Name}} {{.MemUsage}}\" | grep -E \"api-node|flyer\"; sleep 10; done" > /tmp/memwatch.log 2>&1 &'
  2. Upload the largest known PDF (a few hundred pages) to pdf.gevell.com and convert it.
  3. Find where it ran: in Supabase, the newest large_conversion_jobs row for that file, then select worker_id, status from chunk_jobs where job_id = '<id>'. Assembly runs on whichever node completes the last chunk. If every chunk ran on .4, repeat the upload (either host may claim work).
  4. When it completes: anchor 'sort -k3 -h /tmp/memwatch.log | tail -5; journalctl -k --since -1h | grep -i "out of memory"'.

Pass: the conversion completes; peak api-node memory stays under ~6 GB; no OOM kill. If it fails: move to r7i.xlarge (32 GB, ≈ +$46/mo): set anchorInstanceType: 'r7i.xlarge' in infra/cdk/lib/env-config.ts, raise API_NODE_HEAP_MB to 8192 and the api-node caps to 9g in infra/anchor/generate-compose.py, regenerate the compose file, merge, then replace the instance: aws autoscaling start-instance-refresh --auto-scaling-group-name <asg> --preferences MinHealthyPercentage=0 (the anchor is briefly absent; .4 still serves production during this phase).

Task 3 — Rebuild .4 onto the new code

Goal: .4 runs the scheduler-role gate and the reclaim fix, so api-node-2 stops running every timer and a .4 restart can’t reset the anchor’s in-flight chunks.

  1. Confirm the reclaim-fix PR is merged, then from the Mac:
    Terminal window
    npm run rebuild
    (Rolling: api-node-1, wait healthy, api-node-2.) Rebuild succeeded only if it ends with “SUCCESS: All services healthy”; if it fails at image export, re-run it (#3210).
  2. Verify the roles on .4:
    Terminal window
    ssh -i ~/.ssh/nightly-audit [email protected] 'for n in 1 2; do docker logs accessible-pdf-converter-api-node-$n 2>&1 | grep "Roles:" | tail -1; done'
    Expect singletons=leader work=enabled for api-node-1 and singletons=follower work=enabled for api-node-2.

Pass: both lines as above; https://api.gevell.com/health healthy.

Task 4 — Cutover

Preconditions: tasks 1–3 pass; the anchor and .4 run the same commit; no form_conversions in refining (they have no orphan recovery):

select count(*) from form_conversions where status = 'refining';
select count(*) from large_conversion_jobs where status in ('processing', 'assembling');

Pick a quiet time (early morning US Central) and tell Laura.

  1. Find every hostname routed to .4. Cloudflare dashboard → theaccessible.org → DNS, filter cfargotunnel.com: note each record whose target is 4a92034e-0443-4c91-ad16-51fb3fead80e.cfargotunnel.com. Expected: api.gevell.com only. Anything else needs the same flip in step 5 (or a decision).
  2. Stop .4’s queue work and timers, leaving it serving HTTP. On .4, add ~/accessible/docker-compose.override.yml: 
    services:
      api-node-1:
        environment:
          - NODE_WORK_ENABLED=false
          - SCHEDULER_ROLE=follower
      api-node-2:
        environment:
          - NODE_WORK_ENABLED=false
    then roll the two containers (each drains in-flight work on SIGTERM):
    Terminal window
    ssh -i ~/.ssh/nightly-audit [email protected] 'cd ~/accessible && docker compose up -d --no-deps api-node-1 && sleep 60 && docker compose up -d --no-deps api-node-2'
    Check both show singletons=follower work=disabled (task 3 step 2 command).
  3. Promote the anchor (work on, api-node-1 leader):
    Terminal window
    gh workflow run deploy-anchor.yml --ref main -f image_tag=<sha from task 1> -f posture=live
    curl -s https://api-anchor.theaccessible.org/capacity
    /capacity answers from either node; repeat until you have seen "runSingletons":true once.
  4. Smoke the anchor once more at https://api-anchor.theaccessible.org/health and /course-map-api/health.
  5. Flip the hostname (from the Mac, with ~/.cloudflared/cert.pem for theaccessible.org):
    Terminal window
    cloudflared tunnel route dns --overwrite-dns gevell-anchor api.theaccessible.org
    Verify it now lands on the anchor — the uptimes should match the anchor’s, not .4’s:
    Terminal window
    for h in api.gevell.com api-anchor.theaccessible.org; do curl -s https://$h/health | python3 -c "import json,sys;print('$h', json.load(sys.stdin)['data']['uptime'])"; done
    Remove api.theaccessible.org from .4’s tunnel public hostnames in Zero Trust → Networks → Tunnels, so the dashboard doesn’t recreate the old record.
  6. Stop (don’t remove) .4’s product stack, keeping it for rollback:
    Terminal window
    ssh -i ~/.ssh/nightly-audit [email protected] 'cd ~/accessible && docker compose stop'
    The CI runners are systemd services and keep running.
  7. Watch for 48 hours: status.gevell.com, the hourly smoke test, Task 1’s error query, and select status, count(*) from large_conversion_jobs where created_at > now() - interval '1 day' group by 1;

Rollback (any time in the 14 days after cutover):

Terminal window
ssh -i ~/.ssh/nightly-audit [email protected] 'cd ~/accessible && rm docker-compose.override.yml && docker compose up -d'
cloudflared tunnel route dns --overwrite-dns 4a92034e-0443-4c91-ad16-51fb3fead80e api.theaccessible.org
gh workflow run deploy-anchor.yml --ref main -f image_tag=<sha> -f posture=parallel

Re-add api.theaccessible.org to .4’s tunnel public hostnames if you removed it in step 5.

After 14 clean days: Phase 5 of the plan — rotate every secret that lived on .4, wipe and reinstall .4 as the CI host (Media Handling policy; log it), give the Shelly cron its own scoped keys, and update the questionnaire-facing docs (BCP/DRP wording “one instance, automatically replaced”).