3am. The pager goes off. Checkout is down. Your heart rate spikes. What do you do first?
The difference between a calm 20-minute recovery and a chaotic two-hour scramble is rarely talent — it's having a plan you've practised instead of improvising under pressure. A good incident-response process turns "everyone panic" into "everyone knows their next step." Here's a playbook you can adopt.
The lifecycle of an incident
Every incident moves through the same stages. Name them, and the chaos gets a structure:
Detect ──► Triage ──► Mitigate ──► Resolve ──► Review
| Stage | Goal | Key question |
|---|---|---|
| Detect | Know something's wrong | "What's broken, and since when?" |
| Triage | Size it up | "How bad is this, and who's needed?" |
| Mitigate | Stop the bleeding | "How do we reduce impact now?" |
| Resolve | Restore service | "Is it actually fixed and confirmed?" |
| Review | Learn from it | "How do we stop it recurring?" |
1. Detect
You can't respond to what you don't know about. Detection is the job of good monitoring: frequent checks, multiple locations, and alerts that actually reach a human. The faster you detect (low MTTD), the smaller every other number gets.
2. Triage
Before diving in, size it up — a 30-second judgement that shapes everything after:
- How severe? A full outage (SEV1) and a minor glitch (SEV3) deserve very different responses.
- Who's affected? All users, or one region/feature?
- Who do we need? For anything serious, name an incident commander — one person who coordinates (they don't have to be the one fixing it). Clear ownership prevents the "I thought you had it" trap.
3. Mitigate (before you fix)
This is the step people skip — and it's the most important. Reducing impact comes before finding the root cause. Your users don't care why it broke; they care that it's broken.
- Can you roll back the recent deploy? Often the fastest cure.
- Can you fail over to a healthy instance or region?
- Can you disable the broken feature and keep the rest running?
Stop the bleeding first. Diagnose the wound after.
4. Communicate (throughout)
Communication runs alongside every stage, not after. Two audiences:
- Customers — post to your status page early and often. "We're investigating" within minutes beats silence, every time.
- Your team — keep a running log of what's been tried and found. The next person (or the postmortem) will need it. Keep this internal and candid, separate from the public updates.
5. Resolve
The incident is over when service is confirmed restored — not when a fix is deployed and hoped to work. Watch the metrics recover before you call the all-clear, then update the status page to Resolved.
6. Review
Once the dust settles, run a blameless postmortem. What happened, why (systemically), what went well, and — crucially — owned, dated action items so the same incident doesn't come back. This stage is what turns a bad night into a more reliable system.
Common mistakes
| Mistake | Do instead |
|---|---|
| Hunting root cause while users suffer | Mitigate first, diagnose second |
| No one clearly in charge | Name an incident commander |
| Going silent with customers | Post early, update often |
| Calling "resolved" on hope | Confirm recovery in the metrics |
| Skipping the review | Always run a blameless postmortem |
| No runbooks | Pre-write steps for common incidents |
The bottom line
| Stage | In one line |
|---|---|
| Detect | Know fast — that's what monitoring is for. |
| Triage | Size it, assign an owner. |
| Mitigate | Stop the bleeding before chasing the cause. |
| Resolve | Confirm recovery, don't assume it. |
| Review | Blameless postmortem with real action items. |
A practised incident process is the difference between a team that dreads the pager and one that handles it with quiet confidence. Write the playbook before the 3am call — your future self will thank you.
Pair this with On-call best practices and fast detection via alerting.