Force-wake (use 2 to start both)

What this is

Up to two large EC2 spot instances dedicated to CI, managed by a min-0/max-2 Auto Scaling Group (accessible-ci-runner) defined in infra/cdk/lib/stacks/ci-runner-stack.ts. It sits at 0 instances (no cost) until something wakes it. Each instance registers 6 runner processes (ec2-ci-runner-<instance-id>-1 through -6, label ec2-build) and terminates itself once idle (see “How idle scale-in works”, below, for the current thresholds — shortened from a flat 2h on 2026-10-05). Usually one instance is up; the second starts only when the first is saturated with jobs still waiting (see Scaling, below). Raised from max 1 on 2026-10-03 (#4086): 6 slots were full of PR test jobs, 11 jobs were queued, and the post-merge production deploys waited about 10 minutes.

Why 6 processes: a runner process takes one job at a time regardless of core count. The commands inside a job (turbo, vitest, Playwright) use every core, but the ~30 small jobs in test.yml would otherwise run one after another. Change the count with runnerCount on CiRunnerStack. Raised from 4 on 2026-10-03: with 4, a full run kept 7 jobs queued while CPU averaged 35-55% and only ~4 GB of 31 GB memory was in use. Going much higher slows test and engine, which each use every core they can get.

Workflows that run here: all of test.yml (#4067), plus ci-presence, dependency-audit (daily), engine-full-corpus (daily, #4077), db-tests, docs and audit, and the deploys: deploy-aws, build-anchor-images, deploy-anchor, deploy-flyer-tagger and deploy-formspec-lambda. A deploy that finds the instance asleep waits through a cold start (3-5 min); after a merge it is usually still awake from the PR.

Deliberately not here:

• Frequent schedules: pdf-upload-smoke (hourly) and smoke (hourly) would wake the ASG on every run and (pre-2026-10-05, with the flat 2h idle window) never let it go back to sleep, so the instance would run 24/7. Even with the shorter idleSeconds (20min) the instance would still boot and cold-start on an hourly cadence it was never meant to carry — pdf-upload-smoke stays on the office theaccessible-test pool; smoke, smoke-ai, migration-drift and rotation-reminder stay on GitHub-hosted.
• contrast-sweep stays on the office theaccessible-test pool: Cloudflare 403s GitHub-hosted IPs, and it has not been tried from AWS.

No workflow targets the office theaccessible-ci label any more.

10.1.1.4 runs no CI. Its runners (theaccessible-runner-4/-4b) were uninstalled on 2026-10-03 along with deploy-sidecars.yml, the only workflow pinned to it (pdf-stack label).

db-tests uses Supabase’s fixed local ports, so it has a cross-PR concurrency group (db-tests-ec2-host) and runs one at a time.

Instance types

16 vCPU / 64GB spot only: m6a.4xlarge (launch template default), m6i.4xlarge, m7i.4xlarge (launchTemplateOverrides on the ASG’s mixedInstancesPolicy, capacity-optimized, 0% on-demand). All three are x86_64 and compatible with the launch template’s Amazon Linux 2023 AMI (AmazonLinuxCpuType.X86_64). Cost Explorer showed m6a.4xlarge already running 41.9 of 50.7 billed hours this month at about $0.33/h before this change — the idle-timer change above is what cuts that down, not the instance type swap.

Changed from the 32GB c6a.4xlarge/c6i.4xlarge/m6a.4xlarge mix on 2026-10-05: on 2026-10-05 all 6 concurrent jobs on a c6a.4xlarge (32GB, no swap) stalled in npm ci (each ~4.8 GB RSS, load ~28, ~250 MB free) and hit the 15-minute job timeout, while an m6a.4xlarge (64GB) in the same run passed everything. 16 vCPU stays fixed — only the “a/i/7i” CPU family and generation vary, all 64GB.

Spot prices (us-east-1, last-hour sample taken 2026-10-05): m6i.4xlarge $0.19-0.30/h, m6a.4xlarge $0.28-0.35/h, m7i.4xlarge $0.31-0.62/h — all in the same range as the retired c6a.4xlarge ($0.27-0.32/h) and c6i.4xlarge ($0.40-0.63/h), so this change is cost-neutral on the hourly rate; the savings are entirely from the shorter idle windows above. Spot prices move; re-check before relying on these numbers. On-demand pricing for this family was not sampled — check the current rate before setting onDemandPercentageAboveBaseCapacity: 100.

Things that differ from the LAN runners:

• No turbo remote cache. It lives on 10.1.1.4, unreachable from AWS, so TURBO_API is no longer set. The test job’s actions/cache step still restores .turbo between runs.
• Root-owned files can’t wedge a runner. Jobs that bind-mount the workspace into root containers (pdf-enhance, docx2acroform, census) leave root-owned files behind; that broke office runner-4’s checkouts on 2026-10-02. Each EC2 runner has a job-started hook (/usr/local/bin/ci-runner-job-started.sh) that chowns its work dir back to ec2-user before every job.
• Waking is automatic, with no always-on machine. A repo webhook (id 691356561, workflow_job and workflow_run events) calls the CiRunnerWaker Lambda’s function URL. When a job targeting ec2-build is queued and the ASG is at 0, it sets desired capacity to 1, and it adds the second instance on backlog (Scaling, below). Requests not signed with the webhook secret get a 401. Check recent calls under Settings > Webhooks > Recent Deliveries, or the Lambda’s CloudWatch logs (each call logs its decision and reason, e.g. no change for queued test: 2 runner(s) free).

One-time setup (not done by CDK — do these before first deploy)

1. Create a GitHub PAT (classic, repo scope — needed for the runner registration-token API) under an account with admin rights on LarryAnglin/theaccessible-platform.
2. Store it in SSM in us-east-1:

   aws ssm put-parameter --name /ci-runner/github-pat --type SecureString --value '<PAT>' --region us-east-1

3. Webhook secret in SSM (/ci-runner/webhook-secret, SecureString, e.g. openssl rand -hex 32). Created 2026-10-03.
4. The stack (AccessibleCiRunner, production config only) is in infra/cdk/bin/app.ts, so deploy-aws.yml’s cdk deploy --all deploys it with every infra change. To deploy it by hand:

   cd infra/cdk
   npx cdk deploy AccessibleCiRunner --app "npx ts-node --prefer-ts-exts bin/ci-runner-only.ts"

   bin/ci-runner-only.ts synthesizes just this stack, so it deploys without building every Lambda asset bin/app.ts needs (a fresh checkout has none). With AWS SSO, CDK may not find your login: run eval "$(aws configure export-credentials --format env)" first.
5. Register the webhook (repo Settings > Webhooks): payload URL = the stack’s WakerWebhookUrl output, content type application/json, secret = the SSM value, events = “Workflow jobs” and “Workflow runs”. Done 2026-10-03 (id 691356561). If the stack is ever destroyed and recreated, the function URL changes: update the webhook’s URL.

Using it in a workflow

Set runs-on: [self-hosted, ec2-build]. Queuing the job fires the webhook, which wakes the instance; the job waits until a runner is free (3-5 min on a cold start). Nothing else is needed.

If jobs sit queued on ec2-build with the ASG at 0, the webhook didn’t wake it: check its Recent Deliveries for a non-200, then force-wake (Manual controls).

Which jobs a change starts

Runner slots are the bottleneck, so test.yml starts only the jobs a change needs (#4079):

• Source changes: each job’s path filter in the changes job, as before.
• Edits to test.yml itself: only the jobs whose own definition changed. scripts/ci-forced-jobs.mjs parses the workflow at the base and head commits and compares each job. Comments, if: conditions and the path filters don’t count; steps, env, needs, runs-on and the rest do. A changed gated step (the flipbook and contrast-sweep e2e steps inside test) is forced too. A change to top-level env, permissions or defaults forces every job, and so does a base commit that can’t be read. Exception: js lists test.yml in its filter (its tests read the file), so any edit starts it.
• Push to main: only changes, deploy-filters, gate and the a11y/mobile jobs whose app paths the merge touched.
• Escape hatch: the ci:full PR label (labels are read when a run starts, so add it, then push) or a manual dispatch with full checked forces every job. A plain manual dispatch still runs every a11y/mobile job.

Scaling: when the second instance starts

The waker’s rule (infra/cdk/lib/ci-runner-waker/scale.mjs): wake 0 -> 1 on any queued ec2-build job; add one more instance only when every runner on the instances already up is online and busy and an ec2-build job is still waiting. In detail, on each ec2-build workflow_job event:

• Desired 0: a queued job sets it to 1 (no GitHub calls).
• Desired 1 (below max): one GET /actions/runners call. If fewer than 6 ec2-ci-runner-* runners are online, instance 1 is still booting and nothing changes, so one job queued during a cold start never starts a second instance. If any is free, nothing changes. If all 6 are busy: a queued event is itself the waiting job, so desired goes to 2; an in_progress event (a runner just took the last slot) lists that run’s jobs (one more call) and goes to 2 if any ec2-build job is still queued. The in_progress check covers a burst like test.yml’s ~30 jobs, whose queued webhooks can all arrive before GitHub marks any runner busy.
• Desired 2 (max): nothing, no GitHub calls.

The waker only ever raises desired capacity. Scale-in belongs to each instance’s idle watcher, so there is nothing to flap. GitHub API use is about one call per ec2-build job event while exactly one instance is up, on the /ci-runner/github-pat token (5,000 requests/hour). A failed GitHub call logs scale-out check failed and leaves capacity alone; instance 1 still drains the queue, just slower.

Cost: the second instance runs only after a backlog and then lives at least its burstIdleSeconds window (10min default, down from the flat 2h every instance used before 2026-10-05): see “Idle thresholds” above and “Instance types” below for the current per-hour spot price range.

Runner names and stale registrations

Each runner registers as ec2-ci-runner-<instance-id>-<n> (n = 1..6), with the instance id also as a label. Unique names are required with two instances: fixed names plus --replace let a second instance take over the first one’s registrations mid-job. Everything that matches runners keys on the ec2-ci-runner- prefix: the waker’s auto-rerun (spot.mjs) and scale-out check, and the cleanup below. No workflow refers to runner names.

An instance that goes away without deregistering (spot reclaim, a manual scale-to-0) leaves offline entries behind. On every boot, after registering its own runners, an instance deletes offline ec2-ci-runner-* runners whose name contains no instance id currently in the ASG; a peer that is still booting is in the ASG (Pending), so its runners are kept. The pre-#4086 fixed names (ec2-ci-runner-1 .. -6) contain no instance id, so the first boot after deploy removes them once they are offline. The cleanup never fails the boot; it logs ci-runner cleanup: ... in the user-data output.

How idle scale-in works

A systemd timer on each instance (ci-runner-idle-watch.timer, installed by infra/cdk/lib/ci-runner-userdata.sh) runs every 10 minutes. If no Runner.Worker process is active on that instance and more than its idle threshold has passed since its last job finished, it deregisters its own runners and calls aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <self> --should-decrement-desired-capacity. That removes only this instance and lowers desired capacity by one (2 -> 1, or 1 -> 0); a busy peer is never touched. The instance role can terminate instances in this ASG but can no longer set its desired capacity. If the call fails, the next tick retries it.

Idle thresholds (changed 2026-10-05, #ci-runner-idle-and-64gb): every wake-up previously kept an instance up for a flat 2h after its last job, so a ~20-minute CI run (test + deploy) cost roughly 4-4.5 instance-hours once a second instance’s own 2h window is counted. Measured on 2026-10-05: three short CI bursts (01:13-03:26, 03:39-05:45, 09:22-…), each briefly running two instances, burned 13 instance-hours, almost all of it idle. The timer now picks one of two thresholds, both CiRunnerStack props:

• idleSeconds (default 20min, was 2h) — the primary instance: the earliest-launched InService instance in the ASG at check time. There is always exactly one once any instance is up.
• burstIdleSeconds (default 10min) — any other instance (the one the waker added on backlog, scale.mjs). It only exists because of a real backlog, so it is let go sooner once idle.

Each tick, the script asks the ASG (DescribeAutoScalingGroups, already permitted) for the InService instance ids. Alone, it’s primary. With more than one, a second call (ec2:DescribeInstances, new permission) gets each one’s LaunchTime; the earliest is primary, everyone else is burst. Both instances independently compute this from the same AWS-recorded state, so there’s nothing to coordinate and no race — the one failure mode is a tied or unreadable lookup, which the script resolves to the short threshold (it defaults to burst and only promotes to primary on a clean single-instance or clear-oldest result): two instances that can’t tell they’re alone cost less terminating sooner than both riding out the long window, and either way an idle instance only ever terminates itself.

Effect: a typical PR cycle (test job run + deploy that follows soon after) should now cost roughly 1-1.5 instance-hours instead of 4-4.5.

Trade-off: a deploy workflow that lands more than idleSeconds (or burstIdleSeconds, if it was the burst instance) after the preceding jobs finished will find the ASG back at 0 and pay the same 3-5 minute cold start a fully-asleep ASG always has — just more often now that the window is shorter. Nothing currently chains a deploy closely enough after test.yml for this to matter in practice (deploys wait on gate, which waits on the same run), but a workflow that queues a follow-up job well after the triggering run finishes would see it.

Scale-in protection was considered instead (protect busy instances, let the group scale in), but it needs a hook to toggle protection around every job, and protected instances also ignore a manual set-desired-capacity 0. Self- termination keeps one place that decides an instance is done, the same timer that already ran here.

The ASG’s AZRebalance process is suspended (a CDK custom resource, CiRunnerSuspendAzRebalance): with two instances it could otherwise terminate a busy runner to even out the AZ spread. Instances leave only by idle self-termination, spot reclaim or a manual scale.

Rolling out a change to this (one time, #4086): an instance launched before the deploy keeps its old user-data: fixed runner names and an idle watcher that calls set-desired-capacity 0, which its role may no longer do. Left alone it would deregister its runners after 2 idle hours, fail to terminate, and hold desired capacity at 1 with no runners, so queued jobs would never wake a new instance. After the deploy, once no job is running on it, retire it: aws autoscaling set-desired-capacity --auto-scaling-group-name accessible-ci-runner --desired-capacity 0 --region us-east-1. The next queued job boots an instance with the new user-data, and its cleanup deletes the old ec2-ci-runner-1..6 registrations.

2026-10-05 idle/instance-type change: same caveat, lower stakes — an instance already running when this deploys keeps its old flat 2h timer and c6a.4xlarge/c6i.4xlarge/m6a.4xlarge launch config until it idles out and a fresh one boots with the new user-data and instance types. No manual retire is required (the old idle watcher still terminates itself correctly; it just takes up to 2h instead of 20min), but force a faster rollover with the same set-desired-capacity 0 once no job is running on it.

Scaling back up launches a fresh instance and reprovisions the toolchain via user-data from scratch — there is no persistent “stopped” state, matching how WorkerAsg already behaves in ComputeStack.

To change either idle window, edit idleSeconds and/or burstIdleSeconds passed to CiRunnerStack in infra/cdk/bin/app.ts and redeploy.

The ASG deliberately sets no desiredCapacity in CDK. With one set, every cdk deploy (including deploy-aws.yml’s --all on each merge) reset the ASG to 0 and killed in-flight CI jobs. A deploy that changes the launch template only affects the next boot; the running instance keeps the old one.

Spot reclaims: auto re-run and how often they happen

The instances are 100% spot, so AWS can reclaim one mid-run (2 minutes’ warning). Every job on it then fails; jobs on the other instance carry on. Re-run jobs land on whichever runner is free, and the ASG replaces the reclaimed instance (desired capacity is unchanged), whose boot cleanup deletes the dead instance’s offline registrations. Measured on 2026-10-03: one reclaim killed 4 jobs of a #4067 run. The waker Lambda handles both halves:

• Auto re-run. On each workflow_run completed with failure, it lists the run’s jobs and picks those on an ec2-ci-runner-* runner that have a step cancelled and no step failure (a killed job; continue-on-error steps report success, so engine’s geometry-oracle step doesn’t count). If any, it calls rerun-failed-jobs, which also re-runs gate. Capped at run attempt 3 so a run that keeps failing can’t loop. Real test failures are never re-run. Uses the /ci-runner/github-pat token.
• Metrics (CloudWatch namespace CiRunner):
  • SpotInterruptions: AWS reclaim warnings for this ASG’s instances (EventBridge rule on EC2 Spot Instance Interruption Warning).
  • JobsKilledByRunnerLoss: jobs killed, per failed run.
  • RunsAutoRerun: runs re-run automatically.
  • RerunsSkippedAtCap: runs that hit the attempt cap and need a person.

How bad is it? Sum over a period:

for m in SpotInterruptions JobsKilledByRunnerLoss RunsAutoRerun RerunsSkippedAtCap; do
  printf '%s: ' "$m"
  aws cloudwatch get-metric-statistics --region us-east-1 --namespace CiRunner --metric-name "$m" \
    --start-time "$(date -u -v-30d +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" --period 2592000 \
    --statistics Sum --query 'Datapoints[0].Sum' --output text
done

The ASG’s own history also shows reclaims (“an instance was taken out of service in response to an EC2 health check indicating it has been terminated”): aws autoscaling describe-scaling-activities --auto-scaling-group-name accessible-ci-runner --region us-east-1. Each record shows the run IDs and job names (Lambda logs, 1-month retention).

If reclaims prove frequent, switch the ASG to on-demand: set onDemandPercentageAboveBaseCapacity: 100 in ci-runner-stack.ts — see “Instance types” below for current on-demand vs. spot pricing.

Manual controls

aws autoscaling set-desired-capacity --auto-scaling-group-name accessible-ci-runner --desired-capacity 1 --region us-east-1

# Force-sleep: terminates every instance, killing any running jobs. Their
# offline runner entries are deleted by the next boot's cleanup.
aws autoscaling set-desired-capacity --auto-scaling-group-name accessible-ci-runner --desired-capacity 0 --region us-east-1

# Retire one instance only (what the idle watcher does)
aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <id> --should-decrement-desired-capacity --region us-east-1

# Check current state
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names accessible-ci-runner --region us-east-1 --query 'AutoScalingGroups[0].[DesiredCapacity,MaxSize,Instances[].[InstanceId,LifecycleState]]'

# Registered EC2 runners
gh api 'repos/LarryAnglin/theaccessible-platform/actions/runners?per_page=100' --jq '.runners[] | select(.name | startswith("ec2-ci-runner-")) | [.name, .status, .busy] | @tsv'

Debugging a stuck boot

No SSH — the instance role only has AmazonSSMManagedInstanceCore. Use Session Manager:

aws ssm start-session --target <instance-id> --region us-east-1
sudo journalctl -u cloud-final -f   # user-data output
sudo journalctl -u ci-runner-idle-watch.service