Skip to main content

Incident Response Lifecycle and Postmortems

Manage incidents from detection to resolution to learning. Blameless postmortems prevent recurrence.

TL;DR​

Incident lifecycle: detect (alerts fire) → declare (classify severity) → mitigate (stop customer impact) → resolve (fix root cause) → postmortem (extract systemic learning). Blameless postmortems assume everyone has good intent: no one intentionally breaks production. Postmortems investigate systems, not scapegoat people. Questions: what failed? Why didn't monitoring catch it? What processes failed? What changes prevent recurrence? Action items: improve alerting, add tests, change deployment procedures. Incidents reveal gaps in systems. Treat them as learning gifts, not shameful failures to hide. Organizations with healthy incident cultures improve fastest.

Learning Objectives​

  • Manage incident lifecycle from detection through resolution
  • Declare severity appropriately; coordinate response by severity
  • Facilitate effective incident response and communication
  • Run blameless postmortems that drive systemic change
  • Convert postmortems into concrete, actionable improvement items
  • Track action items to completion
  • Build a learning culture that treats incidents as opportunities

Motivating Scenario​

A deploy introduces a subtle bug that causes 10% of API calls to timeout after 30 seconds. The on-call engineer is paged at 2 AM. After 45 minutes, the issue is identified: a new database connection pool configuration is exhausting connections. Rollback happens, service recovers. Users experienced 45 minutes of degradation.

In a blame-focused culture, the postmortem focuses on the engineer: "Why didn't you test this?" In a learning culture, the postmortem asks: "Why didn't our tests catch this? Why didn't our monitoring alert us immediately? Why was rollback not faster? What processes can prevent this class of bug?"

Learning-focused postmortems lead to: automated load tests that catch connection pool issues, monitoring that alerts on timeout rates within 2 minutes (not 45), and automated rollback procedures. The next similar bug is caught in CI/CD, never reaching production.

Core Concepts​

Incident Response Lifecycle: Detection to Learning

Incident Severity Levels​

SEV-1 (Critical): Service entirely down or severely degraded. Customer-facing, significant business impact. Declare immediately, page all on-call, escalate to leadership. Target MTTR: <15 minutes.

SEV-2 (Major): Significant customer impact, but service partially available. Page on-call + service lead. Target MTTR: <1 hour.

SEV-3 (Minor): Isolated customer impact, non-critical functionality affected. Create ticket, investigate during business hours. Target MTTR: <24 hours.

SEV-4 (Degradation): No customer impact yet, but metrics concerning. No page. Create ticket for investigation.

Blameless Postmortem Structure​

Timeline: Chronological events. "2:15 AM: Alert fired. 2:17 AM: On-call checked monitoring. 2:25 AM: Root cause identified (connection pool exhausted). 2:32 AM: Rollback initiated. 2:38 AM: Service recovered."

Contributing factors: What conditions allowed this incident? "Lack of connection pool monitoring, high QA traffic during peak load, no load testing before deploy."

Root cause: Systemic gap. "New connection pool config not tested under load. Monitoring did not alert on connection pool exhaustion. Runbook didn't cover this failure mode."

Action items: Concrete changes. "Add connection pool monitoring (owner: Bob, due in 1 week). Add load testing to CI/CD (owner: Alice, due in 2 weeks). Update runbook with connection pool troubleshooting (owner: Carol, due in 3 days)."

Practical Example​

# Postmortem: API Timeout Incident - 2025-02-14

## Summary
Production API experienced timeout errors (10% of requests) for 45 minutes due to database connection pool exhaustion.

## Impact
- Duration: 02:15 - 03:00 UTC (45 minutes)
- Severity: SEV-2
- Services affected: User API, Orders API
- Customers impacted: ~5% of active users experienced timeouts
- Estimated revenue impact: $12K

## Timeline
| Time    | Event |
|---------|-------|
| 02:15   | Alert fires: "API error rate > 5%" |
| 02:17   | On-call (John) acknowledged alert |
| 02:20   | John checked error logs: "timeout exceeded" |
| 02:25   | John identified: database connection pool full |
| 02:27   | John escalated to DBA (Sarah) |
| 02:32   | Sarah found root cause: new connection pool config from deploy at 01:45 |
| 02:35   | Decision made: rollback the deploy |
| 02:38   | Rollback initiated |
| 02:42   | Error rate returned to normal |
| 03:00   | Service stable, all-clear given |

## Root Cause Analysis

### What Went Wrong
1. **Insufficient load testing before deploy**: New connection pool configuration reduced pool size from 100 to 50, assuming similar load. Testing with realistic load not done.
2. **Missing monitoring**: No alerts on connection pool utilization or exhaustion.
3. **Unclear runbook**: Runbook didn't include connection pool troubleshooting steps. On-call spent 5 minutes identifying the issue.

### Contributing Factors
- High baseline traffic during this time window (peak usage)
- New code path added more queries per request (higher pool demand)
- No peer review of infrastructure changes (connection pool config change approved without discussion)

## Blameless Analysis
The engineer who deployed the config change acted with good intent: they were optimizing connection pool overhead. The lack of systems (monitoring, load testing) enabled the problem, not the engineer's competence.

## Action Items

| Action | Owner | Due Date | Priority |
|--------|-------|----------|----------|
| Add connection pool monitoring and alerts | Sarah (DBA) | 2025-02-21 | P0 |
| Add load testing to CI/CD pipeline | Alice (DevOps) | 2025-02-28 | P0 |
| Update runbook: connection pool troubleshooting | John (On-call) | 2025-02-17 | P1 |
| Require peer review for infrastructure changes | Bob (Platform Lead) | 2025-02-21 | P1 |
| Add connection pool metrics dashboard | Sarah | 2025-02-28 | P2 |

## Prevention for Future

1. **Monitoring**: Alert on connection pool utilization >80%, exhaustion, timeout latency
2. **Testing**: Load tests validate connection pool sizing for realistic peak load
3. **Process**: Infrastructure changes require peer review, especially resource pools
4. **Documentation**: Expanded runbook with connection pool failure modes and remediation

## Learning
This incident revealed that while our database is reliable, our observability around resource constraints is weak. We now have a system to prevent similar incidents for other pooled resources (memory pools, thread pools, etc.).

When to Use / When Not to Use​

Blameless, Learning-Focused Culture
  1. Growing, innovative organizations
  2. Systems with complex interactions
  3. Teams experimenting with new technologies
  4. High-reliability requirements
  5. Organizations that value long-term improvement
Different Approaches
  1. Highly regulated industries (may need compliance focus)
  2. Organizations optimizing for short-term stability
  3. Siloed teams with clear boundaries
  4. Systems with single points of failure
  5. Cultures with existing trust issues

Patterns and Pitfalls​

Pattern: Five Whys to Find Root Cause
Don't stop at surface explanations. Ask 'Why?' repeatedly. Surface: 'Config was wrong.' Why? 'No validation.' Why? 'No test for this scenario.' Why? 'Tests don't cover edge cases.' Root cause: 'Testing strategy insufficient for this type of failure.' Systemic improvement: strengthen testing.
Pitfall: Postmortems That Blame Individuals
If postmortem concludes 'John was careless,' you've failed. That's blame, not learning. Even if John made a mistake, the system allowed it. Root cause should be: 'No peer review process, insufficient automation, unclear procedures.' Action: add peer review.
Pattern: Action Items for Monitoring Gaps
Most incidents reveal monitoring gaps. If you didn't detect the problem for 45 minutes, your monitoring was insufficient. Action items should include: 'Alert on X metric within Y minutes of issue start.' Monitoring improvements prevent similar incidents faster.
Pitfall: Forgotten Action Items
Postmortem concludes with 10 action items. Six months later, none are done. Incidents repeat. Treat action items with urgency. Track them. Review monthly. Close or explicitly defer. If action items aren't done, the postmortem failed.
Pattern: Incident Metrics Show Culture Health
Track: incidents per month (decreasing = improving), MTTR (decreasing = improving), action item completion rate (should be >90%), repeat root causes (should be 0). These metrics show if your learning cycle works.
Pitfall: Postmortems Only for Severe Incidents
Some teams only postmortem SEV-1 incidents. But SEV-3 incidents reveal patterns too. A SEV-3 today might become SEV-1 tomorrow if not addressed. Postmortem on patterns: 5 SEV-3 incidents from authentication? Postmortem auth service.

Design Review Checklist​

  • Are incident severity levels clearly defined and understood?
  • Is there a documented process for incident response?
  • Are postmortems scheduled within 24-48 hours of incident resolution?
  • Do postmortems focus on systems, not individual blame?
  • Are action items created with owners and due dates?
  • Is action item progress tracked monthly?
  • Do you measure MTTR, incidents per month, action completion rate?
  • Are postmortems shared transparently with the team?
  • Do you celebrate learning from incidents (not hide them)?
  • Are repeat root causes identified and escalated?

Self-Check​

  • What was your most recent incident?
  • What's the longest action item still pending?
  • Have you had repeat root causes? Why?
  • How many incidents did you have last quarter?
  • Is your team eager to report incidents or hiding them?

Next Steps​

  1. Define severity levels: Create SEV-1/2/3 definitions with response expectations
  2. Create postmortem process: Document who, when, format, action item tracking
  3. Train facilitators: Ensure someone understands blameless postmortem principles
  4. Establish tracking: Use spreadsheet or Jira for action item progress
  5. Measure and improve: Track metrics; celebrate improvements

References​

  1. Beyer, B., et al. (2016). Site Reliability Engineering. O'Reilly Media ↗️
  2. Edmondson, A. (2018). The Fearless Organization. Wiley ↗️
  3. Weick, K. E., & Sutcliffe, K. M. (2015). Managing the Unexpected. Wiley ↗️