---
title: Forty monitors fail at once. Slack gets one message.
description: Alert grouping, escaping and retries: what happens after an outage is confirmed, the bug that held a page back for minutes, and the trade-offs we kept.
canonical: https://watchfor.io/blog/alert-grouping-last-mile
---

# Forty monitors fail at once. Slack gets one message.

2026-10-20 · WatchFor Team · Engineering · engineering, alerting, architecture, incidents

Alert grouping, escaping and retries: what happens after an outage is confirmed, the bug that held a page back for minutes, and the trade-offs we kept.

At 02:14 a DNS provider has a bad two minutes. Forty monitors that depend on
it fail, get confirmed from several locations, and open forty incidents inside
the same minute. The team has Slack, email and Telegram routed to all of
them.

Without anything in between, that is 120 messages before anyone has opened a
laptop, followed by another 120 "recovered" messages when the provider comes
back. Nobody reads message 37. Most people mute the channel by message 12, and
the channel stays muted on the night it matters. That slow slide has a name,
[alert fatigue](/blog/alert-fatigue), and a lot of it starts with storms like
this one.

With grouping, Slack gets one message: *40 new incidents*, with the list. Email
gets one. Telegram gets one. The recoveries arrive the same way. The on-call
person's pager still gets forty separate events, on purpose, and that
difference is most of this post.

This is the third piece in a line of posts about how WatchFor is built.
[The first](/blog/scheduling-25-million-checks-one-go-binary) was about
deciding which checks run when. [The second](/blog/confirm-outage-before-paging)
was about deciding whether a failed check is an outage. This one starts after
that decision, when the answer is "yes, tell someone". Deduplication,
silences, inhibition, flapping and escalation chains are explained there and
not repeated here; this post is only about the part people actually see: the
message.

## Two kinds of reader

A notification goes to one of two kinds of reader, and they want opposite
things.

**People in chat and email** want to know what is going on. Forty messages
about one cause is noise; one summary that says "forty things broke at 02:14"
is the useful version. So Slack, Teams, Discord, Google Chat, Mattermost,
Rocket.Chat, Telegram and email go through a short **grouping window**.

**Machines** want one event per incident. PagerDuty and Opsgenie track each
alert by its own key: the event that opens an incident and the event that
resolves it must carry the same key, or the pager never auto-resolves.
Webhooks and Zapier feed scripts that expect one payload per incident. Push
apps show one notification per thing that needs a tap. Grouping those would
break their own deduplication, so they skip the window and get each incident
the moment it is confirmed. PagerDuty and Opsgenie then group on their side,
by their rules.

| Channel | During a storm of 40 incidents |
| --- | --- |
| Slack, Teams, Discord, Google Chat, Mattermost, Rocket.Chat, Telegram, email | One summary per channel, recoveries batched the same way |
| PagerDuty, Opsgenie | 40 events, each with its own key, each resolved by its own recovery |
| Webhooks, Zapier | 40 payloads, one per incident |
| Push (ntfy, Pushover, the WatchFor app…) | One notification per incident |

Which channel to use for what, and how to set each one up, is in
[uptime alerts in Telegram, ntfy and Slack](/blog/uptime-alerts-telegram-ntfy-slack).

A storm of one is not a storm. If only one incident arrives in the window, the
channel gets the normal, fully detailed message for it, not a one-row summary.

## The alert grouping window, and the bug that made it slow

The window has two timers, borrowed from how Prometheus Alertmanager does it.

- **group_wait, 30 seconds.** When the first message for a channel arrives,
  it waits 30 seconds for company. Anything else that fires in those 30
  seconds joins it, and the channel gets one message.
- **group_interval, 5 minutes.** After a channel has been sent a message, it
  stays "warm" for five minutes. Follow-ups for incidents it already told you
  about (their recovery, a repeat reminder) collect until the warm period
  ends and go out as one more summary. A storm that develops slowly produces
  one follow-up every five minutes, not one per wave.

The first version applied the warm period to *everything* on the channel. It
looked tidy in tests and was wrong in production. Picture it: at 02:14 the
DNS storm goes out as one summary, and the channel turns warm. At 02:15 a
completely unrelated monitor, the payments API, goes down. Its first message
joined the warm window and waited for it to end. The page that mattered most
that night arrived almost five minutes late, behind a batch about something
else.

![Two timelines for one Slack channel. Before the fix: the storm summary goes out at 02:14:30 and the unrelated payments incident from 02:15 waits until the warm window ends at 02:19:30, four and a half minutes late. After the fix: the payments incident cuts the warm window short and goes out at 02:15:30, and the forty recoveries still batch into one summary at the end of the next warm period.](/blog/alert-grouping-last-mile-window.webp)

The fix is a rule that reads like common sense once you have been bitten: **a
brand-new incident never waits behind follow-ups.** Each channel remembers
which incidents it has already delivered in the current warm period. A
message for an incident the channel has not seen yet is *fresh*; a fresh
message pulls the window forward so it flushes within 30 seconds at the
latest, and it takes any queued follow-ups along with it. Only messages about
incidents the channel already knows about wait for the warm period.

So the payments API page goes out at 02:15:30. When the DNS monitors recover
at 02:16:30, those forty "back up" messages are news about incidents the
channel already knows, so they wait for the warm period and arrive as one
more summary. A slow storm still collapses into one message every five
minutes; a new problem never queues behind it.

The memory of "what this channel already delivered" is pruned lazily: entries
older than the warm period are dropped the next time the channel flushes, so
it doesn't grow with channels and incidents that are long gone.

## The site you monitor writes part of your alert

A good alert says what broke in the words of the thing that broke: the SMTP
server's reply, the certificate's subject, the text a browser check expected
and the text it got, the Playwright error that names the button it could not
find. That is also text a stranger controls. Whoever runs the server chooses
its banner.

Paste that text into a chat message as-is and three things can go wrong.

1. **The message is lost.** Telegram parses a small subset of HTML in bot
   messages. An error that contains `<button class="pay">` is not valid in
   that subset, so Telegram refuses the *whole message* with "can't parse
   entities". The alert never arrives, and the reason is a quote from the
   site that was down.
2. **The message pings everyone.** In Slack, `<!channel>` notifies a whole
   channel and `<@U…>` mentions a person. Discord has `@everyone`. A server
   that puts those strings in a reply can ping your team through your own
   monitoring.
3. **The message carries a link you didn't write.** Markdown channels turn
   `[verify your account](https://…)` into a clickable link. Inside an alert
   from your monitoring tool, a link like that looks trustworthy.

Email has its own version: a newline in a subject line can start a new mail
header.

There is no single escaping function for all of this, because the channels
speak different dialects. So every value that came from the outside goes
through an escaper for its channel's style:

- **HTML-subset channels** (Telegram, Pushover, Google Chat cards) get `&`,
  `<`, `>` and quotes escaped.
- **Slack** gets backticks replaced so the text cannot escape its code span,
  `&`, `<` and `>` escaped so no mention or link can form, and `@channel`
  words broken.
- **Markdown channels** (Discord, Mattermost, Rocket.Chat, Teams, Webex) get
  backticks replaced, mentions broken, and the `](` of a would-be link split
  so it never becomes one. The detail that took longest: inside a code span,
  these renderers print backslashes literally, so the classic "escape it with
  a backslash" leaves junk in the alert. Outside code spans, the same
  characters are backslash-escaped, because there they render as emphasis,
  strikethrough or spoilers.
- **Email** subjects lose control characters before they reach a header.

All channels build their title and body from one shared builder, and each
escaping style has tests that feed it hostile text and check what comes out. When we
add a channel, it inherits the builder and the tests, not a fresh chance to
get this wrong.

## When delivery fails

Slack has outages. Webhook URLs get deleted. Someone rotates a Telegram bot
token and forgets one place. A notifier tries the request, and if the
provider fails or times out, tries once more after a short pause. If that
fails too, the failure is recorded, and the dashboard says so.

![The Delivery log: each notification with its time, channel, status, severity, monitor and incident.](/blog/alert-grouping-last-mile-delivery-log.webp)

Providers explain failures in their own words: an SMTP code, a JSON error, a
"see our documentation" that means *their* documentation. The **Delivery
log** shows the raw answer, and above it one sentence you can act on:
"410 Gone: the endpoint was removed, update the URL." A *Send test* button on
every channel writes to the same log, and every test message is marked
**[TEST]** in its title, so a test never looks like a real incident in a busy
channel.

## The trade-offs, itemized

**Chat is 30 seconds slower than it could be.** The first message to a chat
or email channel waits for the grouping window. Pagers, webhooks and push
don't wait, which is why on-call routing should go to those.

**The window lives in memory.** If the alert manager crashes in the middle of
a window, the messages in that window are lost. The incidents are not: they
are stored before they reach the buffer, so the dashboard, the incident list
and the on-call chain still have them. A normal restart flushes every open
window first.

**One retry, not a durable outbox.** A provider that is down for ten minutes
means missed messages to that channel, visible in the Delivery log. We chose
that over a queue that redelivers a pile of stale "down" messages after the
incident has already recovered.

**A summary hides detail.** "40 new incidents" with a list is the right first
message and the wrong place to debug. Each row links to its incident, where
the root cause and every check result live.

## What we deliberately didn't build

No per-user learning of which alerts you ignore, no AI deciding what is
"important enough" to send, no digest that holds alerts back for an hour.
Every rule in this post is a window, a set of delivered incidents or an
escaper per channel. When a message arrives late or not at all, the incident
timeline and the Delivery log show exactly which rule did it, and a person
can check that at 3 a.m. without guessing.

If you run notifications for your own system, the two lessons that cost us
the most are short: treat brand-new problems differently from news about old
ones, and treat every word your monitoring quotes as untrusted input.

Channels, routing and the grouping window from the user's side:
[alerting integrations](/alerting-integrations) and
[incident management](/incident-management).

---

Canonical page: https://watchfor.io/blog/alert-grouping-last-mile · All articles: https://watchfor.io/blog/all.md · Site guide: https://watchfor.io/llms.txt