Incident Response Plan
Organization: Strategic Habits Consulting, LLC (DBA Gevell) Plan owner: Larry Anglin, CTO Review cycle: Annually, with a tabletop exercise (§11), and after every SEV1/SEV2 incident Related: Business Continuity Plan, Disaster Recovery Plan (same folder)
1. Purpose and scope
This plan sets out how Gevell detects, handles, communicates and learns from security incidents and major service incidents. It covers:
- Customer data: documents, account data, metadata.
- Production systems: Cloudflare, AWS, Supabase, the EC2 processing instances, GitHub, Stripe, Resend.
- The AI features, which run on Anthropic Claude through AWS Bedrock.
Security incident: any event that threatens, or actually harms, the confidentiality, integrity or availability of customer data or production systems. Data breach: a confirmed unauthorized acquisition of, or access to, customer data.
2. Roles
Gevell has two people. One person may hold several roles.
| Role | Who | Responsibilities |
|---|---|---|
| Incident Commander (IC) / Technical Lead | Larry Anglin (CTO). The only person with production access | Declares incidents and sets severity. Carries out containment, eradication and recovery. Preserves evidence. Decides whether an event is a breach |
| Communications Lead | Laura Hopkins (Chief Accessibility Officer). No production access | Drafts and sends customer notifications (content approved by the IC). Is the single point of contact for affected institutions. Keeps the communications log |
| Scribe | Laura, when available; otherwise the IC | Keeps the incident timeline (who, what, when) |
| Backup Technical Responder | Designated. Known to Larry and Laura; not named in this document | Acts as IC through the break-glass procedure (BCP §5) if Larry is unavailable |
| External support | Provider support teams; outside counsel as needed | No IR retainer is in place. Forensic help would be engaged ad hoc |
3. Severity levels
| Level | Definition | Examples | Response target | Customer comms |
|---|---|---|---|---|
| SEV1: Critical | Confirmed or strongly suspected compromise of customer data, or a full outage of core service | Unauthorized access to R2 documents or the database. Leaked Supabase service-role key. Ransomware on a processing instance. Cross-tenant data exposure. All conversions down | Start within 1 h, 24/7. Continuous work until contained | Yes. See §6 |
| SEV2: High | Credible threat to customer data with no confirmed access, or a major partial outage | Leaked lower-privilege credential. Exploitable vulnerability in production. Malware detected on upload and blocked, but with a pattern of attempts. Major feature (e.g., sign-in) down for more than 1 h. AI provider sending data outside the zero-data-retention path | Start within 4 h | Status page for outages. Direct notice if customer data may be involved |
| SEV3: Medium | Limited impact. Contained, or no customer data at risk | Single-customer job failures from a bug. Denial-of-service or abuse attempt blocked by controls. Spend-guardrail trip. Non-critical dependency outage | Next business day | Affected customer only, if needed |
| SEV4: Low | Minor. Informational | Phishing email received and not acted on. Low-severity dependency advisory. Failed login noise | Within 5 business days | None |
Severity can be raised or lowered as facts emerge. When unsure, pick the higher level.
4. Detection sources (in place)
- Health-monitor daemons that probe the production APIs every 60 s and alert by email and Telegram after 3 failures.
- Application alert rules (error rate, fatal errors, missing heartbeat), with logs in Amazon CloudWatch Logs, and status.gevell.com, which checks every production endpoint each minute.
- AWS GuardDuty threat detection and AWS CloudTrail on the AWS account, with alerts to the CTO.
- AWS CloudWatch alarms: dead-letter-queue messages, queue backlog, API 5xx spikes.
- Bedrock spend guardrail and budget alerts. The AI kill switch trips automatically at the caps.
- Malware scanning of uploads (AWS GuardDuty for S3).
- Backup-status staleness alerts. Nightly security audit email (
npm audit,.envfiles in git). - Dependabot, CI dependency audit, Supabase security advisors.
- External reports to [email protected], customer support, and provider abuse notices.
5. Response phases
For every incident at SEV2 or above, open an incident record: a private
GitHub issue labeled incident, or a dated file in a restricted location if
GitHub itself is affected. The timeline is kept there.
| Phase | Key actions | Output |
|---|---|---|
| 1. Detect | Receive the alert or report. Acknowledge it. Write down the time first seen | Timestamped entry |
| 2. Triage | Is it real? What systems and data are involved? Which customers? Assign severity. Is customer data possibly affected? If yes, start the evidence clock (§8) | Severity, initial scope |
| 3. Contain | Stop the harm without destroying evidence. Options: revoke or rotate the credential; block the IP or account; disable the feature; trip the AI kill switch (§9); revoke the Cloudflare Tunnel token and isolate the processing instance (security-group swap, ASG standby); scale the queue consumers to 0; put a frontend into maintenance | Harm stopped. Actions logged |
| 4. Eradicate | Remove the cause: patch the vulnerability, remove malware or rebuild the host from clean sources (DR §5.6), rotate all possibly exposed secrets (the secrets inventory and rotation runbook), close the misconfiguration | Root cause removed |
| 5. Recover | Restore service and data (DR plan). Monitor closely for recurrence (at least 72 h of heightened watch for SEV1). Confirm integrity before re-enabling any disabled features | Service restored, verified |
| 6. Notify | Assess breach status. Notify customers per §6 and keep a log of all notices. Update the status page for outages | Notifications sent and logged |
| 7. Post-incident review | Hold a blameless review within 10 business days for SEV1/SEV2 (§10). Record actions as GitHub issues with owners and dates | Review document, action items |
6. Customer notification
6.1 Timelines
- Confirmed breach of customer data: we notify affected customers within 72 hours of confirming the breach.
- Contracts take precedence. If a customer agreement (including a data security or FERPA addendum) sets a shorter window or different trigger, such as notice of a suspected incident, that term governs for that customer. The IC checks for any such terms at triage for every affected institution.
- Early notice. When we have reasonable belief that an institution’s data is involved but have not confirmed it, we aim to send an initial heads-up within 24 hours. This lets the institution meet its own reporting duties. Many Texas public institutions have short internal and state reporting timelines.
- Role split. The customer institution controls its data (the FERPA “school official” model). We notify the institution. Unless the contract says otherwise, the institution decides on notice to individuals and regulators, and we support it with the facts it needs.
6.2 Contents of a notification
- What happened, in plain language, and when (discovered, and the estimated window of exposure).
- What data was involved (types of data and documents, and how many records, if known) and which accounts or tenants.
- What we have done to contain it and prevent recurrence.
- What the customer should do, if anything (for example, rotate API keys, review users).
- Whether law enforcement was notified or has asked for a delay.
- The contact person (Laura for coordination, Larry for technical detail), and [email protected].
- When the next update will come. Updates follow as material facts change, and a final summary is sent after the post-incident review.
Notices go by direct email to the customer’s named security or business contacts. They do not go only through the product’s email system.
7. Communications discipline
- Only the IC and the Communications Lead speak about an incident outside the company.
- Do not speculate. State what is confirmed and what is still under investigation.
- Do not put customer data or secrets in chat, Telegram, status posts or tickets.
8. Evidence preservation
Collect evidence before rebuilding or deleting anything, where safe to do so.
| Source | How to preserve | Normal retention (why it matters) |
|---|---|---|
| Processing instances (EC2) | EBS snapshot before termination, plus docker ps, container logs, /var/log, and running processes and network connections via SSM Session Manager | Instance storage is lost on termination |
| Application logs (Amazon CloudWatch Logs, processing host and API) | Export the incident time range | Per log-group retention |
| AWS CloudTrail, CloudWatch Logs (Lambda) | Export or copy to a dedicated evidence bucket | Per configuration (verify, see Actions) |
| Cloudflare audit log, access and WAF events | Export | Plan-dependent |
| Supabase logs (auth, API, Postgres) | Export from the dashboard | Plan-dependent, short |
| GitHub audit log | Export | Per GitHub plan |
Application audit tables (admin_audit_log, cost_ledger, job tables) | SQL export | Durable |
| Stripe events | Export | Stripe-retained |
- Store evidence in a restricted, access-logged location separate from the affected system.
- Record a SHA-256 hash of each file, and who collected it and when (chain-of-custody note in the incident record).
- Keep incident evidence for at least 1 year, or longer if litigation or a contract requires it.
9. AI incident handling
AI incidents include:
- Runaway cost or looping.
- Prompt-injection effects.
- Customer data sent to a model path outside zero-data retention.
- Harmful or clearly wrong output at scale (for example, systematically wrong alt text).
- A provider-side incident at AWS Bedrock or Anthropic.
- Kill switch (in place). Disable Bedrock-backed PDF conversions
immediately. Use the admin dashboard (AI Controls → Disable AI, with a
reason) or, if the API is down, a direct Supabase SQL update. See
the AI kill-switch runbook. Queued jobs are held
(
paused_budget), not lost. Each action is written toadmin_audit_log. The guardrail also trips automatically at spend caps. - For AI paths the kill switch does not cover, disable the specific feature through its flag or route, or revoke the model credentials.
- Triage as for any incident. If customer data went to a non-approved AI path, treat it as a potential breach (§6).
- Re-enable only after completing the checklist in the kill-switch runbook (root cause fixed, spend reviewed, abuse blocked).
10. Post-incident review
Required for SEV1 and SEV2, and optional for SEV3. The review is blameless and held within 10 business days.
Post-incident review template
Incident ID / title:Severity (initial → final):Date/time detected: Date/time contained: Date/time resolved:IC: Comms lead: Scribe:
Summary (3–5 sentences):
Customer impact (customers, data types, records, duration):
Timeline (UTC): - hh:mm event / action / who
Root cause:Contributing factors:Detection — how was it found? How could it be found sooner?What went well:What went poorly:
Notifications sent (to whom, when, channel):Breach determination (Y/N, rationale):Evidence collected (location, hashes):
Action items: | # | Action | Owner | Due | GitHub issue |
Plan changes needed (IR/DR/BCP):11. Annual tabletop exercise
Frequency: Once every 12 months. It may be held together with the BCP tabletop. Participants: Larry, Laura and the backup technical responder. Duration: 1–2 hours.
Status: no IR tabletop has been performed and recorded yet.
- Pick a scenario (rotate each year):
- A leaked Supabase service-role key found in a public paste.
- Ransomware on a processing instance.
- An institution reports that it can see another institution’s document.
- AI output routed to a non-zero-retention provider.
- A suspicious Cloudflare login while Larry is travelling.
- Walk through phases 1–7 against the clock. Draft the actual customer notice using §6.2.
- Confirm that the contacts in §12 are current and that the kill switch and evidence export steps still work (dry run where safe).
- Record the results. Turn every gap into an action with an owner and date.
Tabletop log (template)
| Date | Scenario | Participants | Time to contain (simulated) | Notice drafted within 72 h (Y/N) | Gaps found | Actions (owner, due) | Plan updated (Y/N) |
|---|---|---|---|---|---|---|---|
| YYYY-MM-DD |
12. Contact list
| Contact | Details |
|---|---|
| Security reports (external) | [email protected] |
| Incident Commander | Larry Anglin, [email protected] |
| Communications Lead | Laura Hopkins, [email protected] |
| Backup Technical Responder | Designated. Known to Larry and Laura; not named in this document |
| Legal counsel | In the offline contact list |
| AWS | AWS Support Center (console → Support). Abuse reports: [email protected] |
| Cloudflare | Dashboard → Support |
| Supabase | Dashboard → Support / [email protected] |
| Stripe | Dashboard → Support |
| Resend | [email protected] |
| GitHub | support.github.com |
| Backblaze | help.backblaze.com |
| Law enforcement (if required) | FBI IC3 (ic3.gov), or the local FBI field office (fbi.gov/contact-us/field-offices) |
Phone numbers, provider account identifiers and legal counsel details are kept in an offline contact list held by Larry and Laura, not in this document.
Keep a printed or offline copy of this list. It must be usable when email or GitHub is unavailable.
Actions required before this plan is true
| # | Action | Owner |
|---|---|---|
| IR-1 | Formally adopt this plan (sign and date) and brief Laura on her role | Larry |
| IR-2 | Keep the offline contact list current: phone numbers, provider account identifiers, legal counsel and the backup technical responder. Both Larry and Laura hold a copy | Larry + Laura |
| IR-3 | Confirm [email protected] exists, is monitored, and is routed to both Larry and Laura | Larry |
| IR-4 | Brief the backup technical responder (designated 2026-09-23) on this plan and their role | Larry |
| IR-5 | Create the incident issue label and a private location for incident records and evidence (access-logged storage) | Larry |
| IR-6 | Verify that audit-log sources are enabled and retained long enough for investigation: AWS CloudTrail (90 days minimum), Lambda log-group retention, Cloudflare audit logs, Supabase log retention on the current plan | Larry |
| IR-7 | Review customer contracts and addenda for notification windows shorter than 72 hours. Keep a per-customer list the IC can check at triage | Laura |
| IR-8 | Run and record the first annual tabletop exercise (§11) | Larry + Laura |
| IR-9 | List which AI features the kill switch does not cover, and document how to disable each one | Larry |
| IR-10 | Identify a forensic or IR firm that could be engaged on short notice (no retainer), and record the contact in §12 | Larry |