The real outage began at 3:47am. The alert fired right on time. So did the next one, and the next — but by then, nobody was reading them.
For months the channel had been a firehose: disk-usage warnings that always resolved themselves, a flaky health check that cried wolf nightly, deploy notifications, a third-party blip that pinged everyone at once. The team had quietly learned the only sane response to that channel: mute it. So when the message that actually mattered arrived, it landed in a sea of noise everyone had trained themselves to ignore. The outage ran for three hours.
This is alert fatigue, and it's one of the most dangerous — and most common — failures in operations. The cruel irony: it's caused by too much alerting, not too little.
What alert fatigue is
Alert fatigue is what happens when people receive so many alerts — especially noisy, non-actionable ones — that they stop paying attention to all of them, including the important ones.
The core law of alerting: when everything is urgent, nothing is. Every noisy alert quietly lowers the value of every other alert in the queue.
It's not laziness. It's a completely rational human response to noise. Which is why the fix is never "tell people to try harder" — it's fixing the alerts.
Why it happens
Almost every case traces back to the same handful of culprits:
| Cause | What it looks like |
|---|---|
| Too many alerts | Every metric has a threshold and every threshold pages someone. |
| Non-actionable alerts | "CPU is at 81%." Okay… and? Nothing for a human to do. |
| Duplicate alerts | One root problem fires ten alerts across ten checks. |
| Flapping | A check bounces up/down/up/down, paging on every flip. |
| Thresholds set too tight | Alerting at 70% when trouble doesn't start until 95%. |
| No severity | A minor blip and a full outage arrive looking identical. |
| Alerting on causes, not symptoms | Ten internal warnings fire, but none say "users can't check out". |
What it actually costs
The danger isn't the annoyance — it's what the noise hides and breaks:
| The cost | Why it bites |
|---|---|
| Missed incidents | The one real alert drowns in the noise (our 3am story). |
| Slow response | Even when seen, "probably noise again" delays action. |
| Burnout | A pager that cries wolf nightly wrecks sleep and morale. |
| Attrition | Good engineers leave teams with a brutal on-call. |
| Eroded trust | Once people distrust the pager, they route around it entirely. |
How to fix it
The goal is simple to state and worth tattooing on the wall: every alert that reaches a human should be both real and actionable. Here's how to get there.
1. Alert on symptoms, not causes
Page on what the user feels — "checkout is failing", "the API is returning errors" — not on every internal metric that might contribute. One good symptom alert replaces a dozen cause alerts.
2. Make every alert actionable
Before an alert exists, answer: "What will a human do when this fires?" If the answer is "nothing" or "just watch it", it's not an alert — it's a dashboard metric. Demote it.
3. Use severity, and route by it
Not everything deserves to wake someone up. Split the urgent from the merely interesting, and send them to different places:
| Severity | Example | Where it goes |
|---|---|---|
| Critical | Site down, checkout broken | Phone / pager — wake someone |
| Warning | Latency creeping up, cert expires in 10 days | A team channel — handle in hours |
| Info | Deploy finished | A log or feed — no notification |
4. Confirm before firing
A single failed check from one location is often a network blip, not an outage. Confirming from multiple locations — and optionally requiring a problem to persist for a short duration — eliminates a huge category of false alarms before they ever reach a human.
5. Deduplicate and group
When one outage trips ten checks, your team should get one notification about one incident, not ten. Grouping related alerts turns an alert storm back into a single, readable signal.
6. Tune thresholds to reality
Look at how your system actually behaves. If normal load is 80% CPU, alerting at 70% guarantees nightly noise. Set thresholds where trouble starts, not where comfort ends.
7. Use maintenance windows
Deploying or doing planned work? Silence the affected alerts for the window. Predictable, self-inflicted noise is the easiest kind to remove.
8. Review and delete noisy alerts regularly
Treat your alerts like code: review them. Once a month, look at what fired most and ask, "did anyone act on this?" If not, fix it or delete it. A deleted noisy alert is a feature, not a loss.
9. Track alert volume as a metric
What gets measured gets managed. If your team gets 200 alerts a week and only 5 were actionable, that ratio is your alert-fatigue score. Watch it trend down.
The 3am test
Here's a single question that kills most bad alerts before they're born:
"If this fires at 3am, will I be glad I woke up?"
If yes — it's a real, critical alert; route it to the pager. If no — it belongs in a channel you check during the day, or on a dashboard, or nowhere at all. Run every alert through that test and the noise drains away.
Good alert vs. bad alert
| A bad alert | A good alert |
|---|---|
| "Memory at 78%" | "Checkout error rate above 5% for 5 min" |
| Fires every night, always self-resolves | Fires rarely, means real user impact |
| Goes to everyone, all severities mixed | Routed by severity to the right people |
| Nothing to do but watch | Comes with a clear next step |
| One outage = ten pages | One outage = one grouped incident |
The bottom line
| Principle | In one line |
|---|---|
| The law | When everything alerts, nothing does. |
| The bar | Every alert must be real and actionable. |
| The test | "Would I be glad this woke me at 3am?" |
| The habit | Review, tune and delete noisy alerts — on purpose. |
Fixing alert fatigue isn't about caring more. It's about sending fewer, better alerts — so that when the pager does go off, everyone knows it means something. A quiet pager you trust beats a loud one you've learned to ignore, every single time.
If you want to put this into practice, our alerting guide covers rules, severities and confirmation, and the notifications guide shows how to route the right alerts to the right people.