It's 3am, the pager is screaming, and you're staring at an alert for a system you barely remember. This is the worst possible moment to be figuring things out from scratch. A good runbook is the difference between calm, procedural recovery and frantic, error-prone guessing. Here's how to write ones that actually help.
What a runbook is
A runbook is a documented, step-by-step procedure for handling a specific situation — usually a known type of incident or routine operation. Think of it as the recipe a responder follows so they don't have to invent the response under pressure.
A runbook's real job: make a stressed, half-asleep human effective. It's written for the person who's not the expert, at the worst possible time. Optimise for that reader, not for the person who already knows the system.
What a good runbook contains
| Section | What it gives the responder |
|---|---|
| What it's for | Which alert/situation this runbook handles |
| How to confirm | How to verify the problem is real |
| Impact | Who/what is affected (helps set severity) |
| Steps to fix/mitigate | The actual numbered procedure |
| Escalation | Who to call if these steps don't work |
| Rollback | How to undo, if needed |
The heart of it is the numbered steps — specific, in order, with exact commands or links, not vague advice.
What makes runbooks worthless
Most bad runbooks fail the same ways:
| Mistake | Why it hurts |
|---|---|
| Vague steps | "Investigate the issue" — how? Useless at 3am |
| Out of date | Steps reference systems that changed; trust evaporates |
| Assumes expertise | Written for the author, not a tired stand-in |
| Buried | Can't find it when the alert fires |
| No confirmation step | Responder acts before knowing it's the right problem |
The two killers are vagueness and staleness. A vague runbook helps no one; an out-of-date one is worse than none, because it sends people the wrong way confidently.
How to write a good one
- Be specific. Exact commands, exact links, exact thresholds. Assume the reader knows nothing about this system.
- Start with confirmation. Step 1 should verify the problem is real (check the health endpoint, the dashboard), so no one fixes a phantom.
- Order by likelihood. Most common cause first.
- Include the escape hatch. When to roll back, and who to escalate to (tie into on-call).
- Link it from the alert. The alert that fires should link directly to its runbook — finding it shouldn't be a scavenger hunt.
- Keep it alive. Review runbooks after each incident; if a step was wrong or missing, fix it now while it's fresh.
Runbooks and your incident process
Runbooks slot directly into the incident response flow — they're the "how" behind the mitigate step. And after an incident, the postmortem is the perfect moment to create or improve the relevant runbook, so the next person has it easier.
The bottom line
| In one line | |
|---|---|
| What | A step-by-step procedure for a known incident. |
| For | A stressed, non-expert responder at 3am. |
| Good | Specific, confirmed first, escalation + rollback included. |
| Deadly | Vagueness and staleness — keep them alive. |
A runbook is a gift to your future self (and your teammates): the calm, specific instructions you'll desperately want when the pager goes off. Write them for the tired stranger, link them from your alerts, and update them after every incident.
Related: Incident response playbook, on-call best practices, blameless postmortem.