---
title: How we confirm an outage before paging anyone
description: A single failed check is a rumour, not an outage. Here's the pipeline that decides which is which: rotating locations, accelerated re-checks driven back into the scheduler, a severity cascade, and an alert manager that deduplicates, silences, inhibits, groups and escalates. Real defaults from the code, and the honest cost: about a minute of extra latency, and blind spots we chose to keep.
canonical: https://watchfor.io/blog/confirm-outage-before-paging
---

# How we confirm an outage before paging anyone

2026-10-03 · WatchFor Team · Engineering · engineering, alerting, architecture, incidents

A single failed check is a rumour, not an outage. Here's the pipeline that decides which is which: rotating locations, accelerated re-checks driven back into the scheduler, a severity cascade, and an alert manager that deduplicates, silences, inhibits, groups and escalates. Real defaults from the code, and the honest cost: about a minute of extra latency, and blind spots we chose to keep.

The first post in this series ended at the moment a check leaves the
scheduler. This one starts when the result comes back, and it is about one
decision: **should a human be woken up?**

Getting that decision wrong in either direction is expensive. Page too
eagerly and people learn to ignore the pager, which is how real outages get
missed (we wrote about that in [alert fatigue](/blog/alert-fatigue)). Page
too late and the customer tells you first. Most of what follows is machinery
for being confident before being loud.

The short version of our policy: **one failed check marks a monitor
Degraded. An incident, a page and lost uptime need a confirmed failure.**
Here is how that confirmation actually works, with the defaults straight
from the code, and the places where it costs you something.

## Why one failed check is not an outage

A probe runs from a machine in a data center somewhere, talking to your
service across the public internet. When it fails, there are at least four
candidates for what broke:

1. Your service.
2. The network path between that one location and your service.
3. The probe location itself: its resolver, its uplink, a full disk.
4. Something in the middle that only fails sometimes: a load balancer node,
   one anycast edge, a TLS handshake that loses a race.

Only the first one is your outage. The other three are real failures that
deserve to be recorded, but paging someone about them teaches them that the
pager lies. So a failed check gets stored as check history, shown as
**Degraded**, and treated as a question to answer quickly.

## The pipeline, end to end

![WatchFor alert pipeline: the scheduler dispatches checks to probers at rotating locations; results go to the writer for check history and to the alerting engine, which gates failed probes, evaluates rules, applies the severity cascade and counts failures in Valkey. While a failure is unconfirmed it tells the scheduler to re-check faster. Confirmed alerts go to the alert manager, which runs dedup, silence, inhibit, flapping, acknowledgement and grouping before notifying channels and the on-call escalation chain](/blog/confirm-outage-before-paging-pipeline.svg)

Five Go services sit on one message bus. The scheduler dispatches due
checks. Probers in 20 locations run them and publish results. Two consumers
read every result independently: the **writer** batches them into ClickHouse
as check history, and the **alerting engine** evaluates the monitor's alert
rules. When a rule is confirmed, the engine sends a *fire* to the **alert
manager**, which owns the incident lifecycle and every notification.

The split matters. The alerting engine knows about checks and locations but
nothing about people. The alert manager knows about people, channels,
maintenance and on-call, but never looks at a raw check. Each can be
restarted, tested and reasoned about without the other.

## Step one: the engine evaluates every rule, carefully

Each alert rule is an [Expr](https://expr-lang.org/) expression compiled
once and evaluated against every result for its monitor. The built-in
availability rule is literally `success == false`. A status-code rule for an
API monitor compiles to something like
`!((metrics['api.status_code'] >= 200 and metrics['api.status_code'] < 300) or (...))`.
Expr gives us a sandboxed language with no loops and no side effects, so a
rule written in the dashboard can't do anything but return true or false.

Two lessons are baked into how evaluation runs.

**A failed probe only runs rules that can be about it.** When a request
times out, most metrics are missing, and the ones that are present are
misleading: the recorded duration of a timed-out HTTP check is simply the
timeout. Evaluate a "response time over 2 s" rule against that and you get a
latency incident on top of the availability one, for the same event. So on
`success == false` the engine only evaluates rules that reference `success`
itself, or rules whose every metric the prober actually emitted. Some probes
do emit authoritative metrics on failure (a traceroute that never reached the
destination, a DNS check's error count), and rules on those still fire.

**An evaluation error never counts as healthy.** If an expression fails
because a metric is absent (`nil >= 500`), the engine skips the rule for
that result and leaves its state alone. Treating the error as a pass would
resolve an open incident on a result that said nothing about the problem.

Results older than 15 minutes are dropped outright. If the bus ever backs up,
replaying a quarter-hour-old backlog should not page anyone about the past.

### The severity cascade

Some presets create two rules on one metric, for example a warning when
responses take over 1 second and a critical alert over 3. When both match
the same result, you want one incident, not two. The engine evaluates every rule first, groups the ones that
triggered by the left-hand side of their expression, and lets the highest
severity win:

```go
// "results.duration_ms > 3000"            → "results.duration_ms"
// "metrics['icmp.packet_loss_percent'] > 20" → "metrics['icmp.packet_loss_percent']"
func extractMetricKey(expression string) string { /* split on the operator */ }
```

The losing rule is treated as not triggered. If it had already opened an
incident (responses were merely slow for ten minutes before they got very
slow), the engine sends a **silent resolve**: the warning incident closes
with no notification, and the critical one carries the story from there. Complex expressions don't match a
simple key and simply don't cascade, which is the safe default.

## Step two: counting toward a confirmed failure

State lives in Valkey, keyed by rule and monitor. For every triggered rule
the engine updates a failure counter (with the time of the first failure),
the set of distinct locations that saw it, and an ordered log of each
confirming check. Simplified, the decision looks like this:

```go
count, failingFor, _ := state.IncrementFailure(ctx, rule.ID, monitorID)
locations, _ := state.AddFailureMember(ctx, rule.ID, monitorID, result.ProberName)
state.AddConfirmation(ctx, rule.ID, monitorID, result.ProberName, result.RunID, result.ExecutedAt)

isCountMet := count >= int64(rule.FailureThreshold) // default 3
isDurationMet := rule.Duration == 0 || failingFor >= rule.Duration
isConsensusMet := rule.MinFailedProbes <= 1 || locations >= int64(rule.MinFailedProbes)

if isCountMet && isDurationMet && isConsensusMet {
	fireAlert(ctx, rule, result, failingFor)
}
// Unconditionally: ask the scheduler to re-check sooner.
sendOverrideForFailure(ctx, rule, result, failingFor)
```

Any passing result before the incident opens wipes all of it. The threshold
means "N failures **in a row**", not "N failures this hour".

Where do the other locations come from? Monitors default to **round robin**:
each check runs from the next location in the monitor's pool, and the
cursor advances on every dispatch, including the extra ones below. With the
default threshold of 3 and three or more locations, the three consecutive
failures needed to open an incident come from three different places. The
location set is there for when you want to be stricter: with the parallel
strategy every location checks at once, and `min_failed_probes` can demand
that a given number of *distinct* locations agree.

Keys carry TTLs (seven days for the counter, a day for the location set), so
a monitor deleted mid-failure doesn't leave state behind forever.

## Step three: the scheduler speeds up

A threshold of 3 on a 5-minute monitor would mean 10 more minutes before
anyone hears about a real outage. That's why the code above sends an
override on every failure, confirmed or not. The engine publishes a message
back to the scheduler on its own subject, and the scheduler re-arms that job
on its time wheel with a shorter interval:

```json
"default_confirmation_schedule": [
  { "interval": "30s", "limit": "5m" },
  { "interval": "1m",  "limit": "15m" },
  { "interval": "5m",  "limit": "1h" }
]
```

Counted from the first failure: re-check every 30 seconds for five minutes,
then every minute up to 15, then every 5 minutes up to an hour. After that
the override is left to expire and the monitor returns to its normal cadence,
incident or not. The override carries a TTL equal to the last limit, so if
the alerting engine went silent the scheduler would drift back on its own.
On the scheduler side, the override can only ever make a check *more*
frequent:

```go
effective := job.OverrideInterval
if effective > job.IntervalSeconds { effective = job.IntervalSeconds } // never slower
if effective < floor { effective = floor }                             // never below the global floor
```

The same mechanism runs in reverse during recovery: an open incident keeps
the fast cadence, so the fix is noticed in seconds rather than at the next
5-minute tick.

Not every check type should be re-run every 30 seconds, and the policy is
per probe type in the engine's config. Core Web Vitals audits take a minute
or two each, so they get no accelerated phase at all; overlapping audits
just queue behind each other. Traceroutes re-check every 2 minutes for up to
10. Domain expiry, data breach and blacklist lookups hit rate-limited
upstreams and change slowly, so they're exempt. Heartbeats have nothing to
re-run.

![What a 60-second HTTP check does during a short outage with default rules: two passing checks a minute apart, then the first failure marks the monitor degraded and switches re-checks to every 30 seconds from rotating locations; the third failure opens the incident, on-call tools get it at once and chat 30 seconds later; after the fix three passing checks resolve it](/blog/confirm-outage-before-paging-timeline.svg)

Here is a real incident from our demo organization. The monitor checks a public
API every 2 minutes. The first failure arrived at 10:08:40, the re-checks
came 30 seconds apart instead of 2 minutes, and the third consecutive failure
opened the incident at 10:09:40. This monitor checks from a single location,
so all three confirmations name the same city; with several locations, round
robin would have spread them.

![The Activity Timeline of a real incident in WatchFor, threshold 3: Issue First Detected at 10:08:40, then Prober Confirmation 1/3 at 10:08:40, 2/3 at 10:09:10 and 3/3 at 10:09:40, each with location, latency and status 403, Incident Confirmed and Active at 10:09:40, Incident Resolved at 10:13:10 and Marked as False Positive at 10:48:44, with a footer reading "Opened after 3 consecutive failed checks, a single failure only marks the monitor degraded"](/blog/confirm-outage-before-paging-incident.webp)

The last row is the honest part. The API was answering `403` because it was
rate-limiting us, not because it was down, and that reproduces perfectly on
every re-check. Confirmation filters out failures that don't repeat. It can't
tell you whether a failure that does repeat is the one you care about, which
is why an incident can be
[marked as a false positive](/docs/alerting/resolving#false-positives) after
the fact and drops out of the uptime numbers.

## Recovery needs confirming too

Flipping a pager to green on one lucky response is its own kind of false
alarm, so recovery is symmetric. A previously alerted rule needs a run of
consecutive passes to resolve (three by default, one for heartbeats and a
few types measured against a fixed baseline). Any failure in between resets
that count.

When a rule demands agreement from several locations, recovery follows the
same logic in reverse. A healthy result removes its location from the
failure set and adds it to a short-lived resolution set (15-minute TTL, so
only recent opinions count). The incident resolves once enough distinct
locations report success, or once too few are still failing to have met the
bar. A single healthy location does not close an outage the others still see.

## The keep-alive, or how an open incident proves it's still open

The engine fires once, then stays quiet while the rule remains failing. But
the alert manager has to tell the difference between "still broken, nothing
new to say" and "the engine lost track of this", for example after state was
lost. So while a rule is alerted, the engine re-sends the firing alert at
most once an hour. The alert manager folds it into the existing incident
(fire count up, `updated_at` bumped) without adding a timeline row or a
notification. Incidents with no keep-alive for much longer than that (a day,
in production) are closed as stale.

## Step four: the alert manager decides who hears about it

Every fire and resolve from the engine passes through the same short
pipeline before anything leaves the building.

**Dedup.** An incident's identity is a fingerprint of its rule:
`sha256("rule:<id>")`. One rule has at most one open incident. Keep-alives
and repeated fires land on it instead of opening new ones.

**Silence.** Maintenance windows and the per-monitor *Mute* both create
silences, matched on monitor, rule or severity. A silence is a flag on the
notification step, never on the lifecycle: the incident is still created,
counted and timed, the timeline records "Notification silenced", and no
escalation starts. Keeping the incident means a muted monitor that goes down
still has a record, and its keep-alives keep the incident open while it is
still failing. If an incident is still open when
its silence ends, a sweeper (every minute) sends its first notification, so
a maintenance window that overruns doesn't hide an outage. The design
reasoning behind the windows is in
[maintenance windows done right](/blog/maintenance-windows-done-right).

**Inhibit.** While a monitor has a critical incident open, its warning alerts
are held back. Slow responses during an outage are a symptom, not a second
problem. The engine's severity cascade handles two rules on the *same*
metric; inhibition handles different metrics on the same monitor.

**Flapping.** A rule that opens 5 incidents within an hour, or 10 within a
day, is marked flapping. Its incidents are still recorded, with flapping
start and end on the timeline, but neither the "down" nor the matching "up"
notification goes out. Without that second half, every flap cycle would drip
an orphan recovery message.

**Acknowledge and repeat.** Acknowledging stops everything for that incident:
repeat reminders and the on-call chain. Repeat reminders themselves are off
by default and set per monitor (every 1, 6, 12 or 24 hours) for teams that
want them.

**Group.** Chat and email deliveries wait in a per-channel buffer for 30
seconds (`group_wait`). If a shared dependency takes ten monitors down, the
Slack channel gets one "10 new incidents" summary instead of ten pings, and
follow-ups in the next 5 minutes (`group_interval`) batch into one more. A
lone incident still arrives as its normal, fully detailed message. PagerDuty,
Opsgenie, webhooks and push apps skip the buffer, because their own
deduplication and auto-resolve depend on one event per incident.

**Escalate.** A new critical incident that isn't silenced or flapping starts
its escalation chain, if one is routed. The chain state lives in Postgres,
not in a timer, so it survives restarts. A sweeper runs every 15 seconds and
re-reads the incident's status and active silences before each step, so an
acknowledgement in the dashboard stops the chain without any coupling
between the two. Warnings never wake the on-call person. How the rotations
themselves work is in
[on-call scheduling in WatchFor](/blog/on-call-scheduling-in-watchfor).

## Why uptime only counts confirmed incidents

All of that would be undone if the uptime number ignored it. A monitor that
shows 99.95% uptime next to "no incidents this month" teaches people that one
of the two numbers is wrong.

So uptime has one formula everywhere, from the monitor card to
[reports](/blog/honest-sla-reports), the API and the Prometheus endpoint:
observed time minus time spent in **confirmed critical incidents**, with
maintenance treated as neither up nor down. Warning incidents show the
monitor as Degraded and cost no uptime. A failed check that never became an
incident costs nothing either. It shows up in a separate number, **check
success**, which is the right place to look for a flaky location. Details:
[how uptime is calculated](/docs/monitors#how-uptime-is-calculated).

Confirmation delays the *page*, not the *record*. An incident's start time
is the first failed check, not the moment it was confirmed, so the
confirmation window doesn't quietly shave downtime off your SLA.

## The trade-offs, itemized

**Latency.** Take a 60-second HTTP monitor with the defaults. The outage
starts; the first failing check lands within a minute; the next two arrive
within 30 seconds each. So PagerDuty gets the incident up to about 60
seconds after the first failed check, and Slack 30 seconds after that. A
system that pages on the first failure would have sent it at second zero.
That is arithmetic from the defaults, not a benchmark, and it's the price of
not paging people for one location's bad minute. Teams that want speed over
certainty can set the threshold to 1 on a monitor.

**Recovery is slower too, and it's counted.** Three passes 30 seconds apart
means an incident closes about a minute after the fix, and that minute counts
as downtime. We'd rather slightly overstate an outage than end it on one
lucky response.

**Regional outages don't page by default.** With round robin and one bad
location out of five, the healthy checks in between keep resetting the
counter, so the threshold is never met. The monitor shows Degraded and check
success drops for that location, but nobody is paged. That's deliberate, and
it's the wrong default if one region *is* your business. For that, use the
parallel strategy, a monitor scoped to those locations, or both.

**When every location shares an upstream, confirmation can't help.** Several
locations agreeing only proves the failure isn't local to one of them. If
your DNS provider, your CDN or a cloud region all of them depend on fails,
they all agree, and they're right that your users can't reach you. The same
holds for any monitoring service: if its locations share a provider, a
problem at that provider can look like your outage from several places at
once. Multi-location confirmation removes single-location noise.
It does not make the vantage points independent.

**Flapping goes quiet.** A service bouncing every few minutes stops producing
messages after the fifth incident in an hour. That's usually right, but it
means a flapping monitor needs a human to look at the dashboard. We chose
silence over a notification storm, and the timeline makes the choice visible.

**Grouping can lose a window.** The 30-second buffer is in memory. If the
alert manager crashes mid-window, those notifications are lost, while the
incidents themselves are already persisted. A graceful restart flushes the
buffer first.

## What we deliberately didn't build

No distributed consensus between probers, no per-location state machines, no
machine-learned anomaly scores deciding when to wake people. Every decision
above is a counter, a set or a timestamp in a key-value store, and a person
debugging a missed page at 3 a.m. can read the incident timeline and see
exactly which checks counted and why. One question we're still weighing is
whether latency and packet-loss rules should default to agreement from more
than one location, since those metrics are noisier from a single vantage
point than a refused connection is.

---

Canonical page: https://watchfor.io/blog/confirm-outage-before-paging · All articles: https://watchfor.io/blog/all.md · Site guide: https://watchfor.io/llms.txt