incident-response

# Incident Response

Structured incident management from detection through postmortem, with resilience patterns for preventing and containing cascading failures.

## When to Use

- Production incident in progress (outage, degradation, data loss)
- Designing circuit breakers, bulkheads, or fallback strategies
- Conducting or planning chaos engineering exercises
- Writing or reviewing postmortem documents
- Establishing on-call procedures and escalation paths

Avoid when:
- The issue is a development-time bug with no production impact
- Designing general system architecture (use system-design instead)

## Quick Reference

| Topic | Load reference |
| --- | --- |
| **Triage Framework** | `skills/incident-response/references/triage-framework.md` |
| **Postmortem Patterns** | `skills/incident-response/references/postmortem-patterns.md` |

## Incident Response Workflow

### Phase 1: Detect

- Alert fires or user report received
- Confirm the issue is real (not a false positive)
- Identify affected services and user impact scope

### Phase 2: Triage

- Classify severity (P0-P3)
- Assign incident commander
- Open communication channel (war room, Slack channel)
- Begin status page updates

### Phase 3: Contain

- Stop the bleeding: rollback, feature flag, traffic shift
- Prevent cascade: circuit breakers, load shedding, bulkhead isolation
- Communicate: stakeholder updates every 15 minutes for P0/P1

### Phase 4: Resolve

- Implement fix (minimal viable fix first)
- Validate in staging if time permits
- Deploy with monitoring and rollback plan ready
- Confirm recovery with metrics returning to baseline

### Phase 5: Postmortem

- Document timeline within 48 hours
- Conduct blameless review with all participants
- Identify root cause and contributing factors
- Assign action items with owners and deadlines
- Update runbooks and alerting based on lessons learned

## Severity Framework

| Level | Impact | Response Time | Examples |
|-------|--------|---------------|---------|
| **P0** | Complete outage, data loss, security breach | Immediate (< 5 min) | Service down, data corruption, credential leak |
| **P1** | Major feature broken, significant user impact | < 30 min | Payment processing failed, auth broken for region |
| **P2** | Degraded performance, partial feature loss | < 4 hours | Elevated latency, non-critical feature unavailable |
| **P3** | Minor issue, workaround available | Next business day | UI glitch, slow report generation, cosmetic error |

## Output

- Incident timeline and severity classification
- Containment actions taken
- Postmortem document with action items
- Updated runbooks and alerting rules

## Common Mistakes

- Skipping severity classification and treating everything as P0
- Making changes without a rollback plan
- Forgetting to communicate status to stakeholders
- Writing postmortems that assign blame instead of identifying systemic issues
- Not following up on postmortem action items