--- title: How incidents work description: The lifecycle of a confirmed problem — firing, acknowledged, resolved — what an incident page shows, faster checks while open, grouping and flapping. canonical: https://watchfor.io/docs/alerting/incidents --- # How incidents work The lifecycle of a confirmed problem — firing, acknowledged, resolved — what an incident page shows, faster checks while open, grouping and flapping. An **incident** is the story of one confirmed problem, from the moment WatchFor is sure it is real until it is resolved. It keeps everything about that problem in one place: what failed, from where, who was notified, what your team did and when it recovered. Incidents live under **Incidents** in the sidebar, filterable by status (firing / acknowledged / resolved) and severity (critical / warning), and on every monitor's page — you can [acknowledge and resolve straight from the list](#acting-from-the-list), one row or a whole selection at a time. **Also via API:** [`/v1/incidents`](/docs/api/incidents) lists, reads, acknowledges and resolves incidents; the MCP server and A2A expose the same. ## From a failed check to an incident A single failed check is not an incident. When a check fails, the monitor turns **Degraded** and WatchFor re-checks on an accelerated schedule, rotating through the monitor's locations, until the rule's confirmation threshold is met — three consecutive failed checks by default. Only then does an incident open. A failure that stops reproducing never becomes an incident and never pages anyone. This is why **Degraded** and **Down** are different things: Degraded is "a check failed, not confirmed" (or an open *warning* incident); **Down** is "a confirmed *critical* incident is open". The full status model is in [Core concepts](/docs/core-concepts#monitor-status); the confirmation cadence and the types exempt from it are in [Locations & scheduling](/docs/monitors/scheduling#how-a-failure-becomes-an-incident). ## The lifecycle ``` firing → acknowledged → resolved ``` - [Firing](#): The problem is confirmed and active. Notifications have gone out; for a critical incident the escalation chain (if any) has started. - [Acknowledged](#): A person has taken ownership. Reminders, repeat notifications and the escalation chain stop; the incident stays open until the condition recovers. - [Resolved](#): The condition recovered (or someone resolved it). A recovery notification goes to the same channels that were alerted. ### Acknowledging Any member can **Acknowledge** a firing incident from its page — the same convention as PagerDuty or Opsgenie: "I see it, stop paging me about it". Acknowledging: - stops **repeat reminders** and further notifications for this incident, - stops the **escalation chain**, - is recorded on the incident timeline and in the [activity log](/docs/organization/activity-log). It does not change the monitor's checks or its status, and the incident **keeps counting as downtime** in uptime and SLA figures until it resolves. If the condition is still failing when the incident is later resolved by hand, a new incident opens on the next confirmed failure. ### Resolving Incidents resolve **automatically** when the condition recovers (three consecutive successful checks; one for heartbeat), when their source goes away (the monitor is paused, the rule disabled or deleted), or **by hand** — owners and admins can resolve from the incident page. Every path notifies the original channels with the reason. Details: [Resolving incidents](/docs/alerting/resolving). An incident that recovers and then fails again is **re-opened**: the timeline records it, and the incident's fire count grows instead of a new incident being created for every wave. ### Acting from the list You do not have to open an incident to act on it. On the **Incidents** list (and on a monitor's Incidents tab) the **⋯** menu on each row offers **View details**, **Acknowledge** (on a firing incident — any member) and **Resolve…** (on any open incident — owner/admin, with the same confirmation as the incident page). Tick several rows and the bulk bar appears: **Acknowledge (n firing)** and **Resolve… (n open)** act on the selection in one go, with one confirmation for the batch, a toast counting what succeeded and an entry per incident in the [activity log](/docs/organization/activity-log). Rows whose incidents are already resolved only offer *View details*. ## In your bell and Inbox Every member of the workspace also sees the incident in the dashboard's bell and [Inbox](/docs/organization#inbox-and-in-app-notifications), without any channel configured: **Down: api.acme.dev** when it opens, **Acknowledged: …** when someone takes it, and **Resolved: …** when it clears. Each one links to the incident page. In the Inbox the badge follows the status model rather than the event: a **critical** incident is badged red **Down**, a **warning** one amber **Degraded** — the same distinction the monitors list makes, so a warning never reads as an outage. The wording distinguishes how an incident ended, too: *Recovered after 12m* when the checks turned green by themselves, *Resolved manually by Jane after 12m* when a person closed it, and *Resolved after 12m — monitor paused by Jane* when it closed because the monitor was paused. The monitor's target sits on its own line under the summary, and a heartbeat monitor shows its **name** there instead of its private ping URL. Silenced, inhibited and flapping incidents stay quiet here too, exactly like on your channels. Members who don't want incidents in their Inbox switch them off under Settings → Notifications. ## Faster checks while an incident is open While an incident is active, the monitor is checked **more often than its normal interval** so recovery is noticed within seconds rather than at the next scheduled check. The cadence returns to normal once the incident resolves. Types without an accelerated confirmation phase — Core Web Vitals, domain expiry, data breach, blacklist and MCP — keep their normal interval, and heartbeat monitors have nothing to re-run: they wait for your job's next ping. ## What's on an incident page - **Header** — monitor, severity, status, when it started and how long it has been open (with "ongoing" while firing), a **Maintenance** badge if it happened inside a scheduled window, and the actions: **Acknowledge**, **Resolve** (owner/admin) and **Mute** the monitor for 1 hour, 8 hours, 24 hours or 7 days (owner/admin). - **[Root cause analysis](/docs/alerting/root-cause)** — what happened in plain language, which locations confirmed it and which still saw the site fine, expected vs. detected, why it usually happens, copy-paste diagnostics. - **Response-time chart** around the incident, with the monitor's warning and critical latency thresholds drawn in (logarithmic scale available). - **Confirming probes** — the actual checks that confirmed the failure, per location, with timing, status code, error, and response headers / body snippet or raw output where the type provides them. - **[Post-mortem](/docs/alerting/postmortems)** card — start one, or see the summary of the one already written. - **[Internal notes](/docs/alerting/internal-notes)** — team-only comments, never shown publicly. - **Activity timeline** — *Issue First Detected*, then one **Prober Confirmation** row per confirming check with its own timestamp and a counter that climbs 1/3 → 2/3 → 3/3 towards the monitor's [confirmation threshold](/docs/alerting#confirmation-settings), then *Incident Confirmed & Active* at the moment confirmation actually completed (not at first detection), *Incident Acknowledged*, *Incident Resolved*, escalation steps ("step 2 — Production escalation, paged …"), and every notification with its channel and outcome (sent, failed, silenced). All of it in your profile time zone and 12/24-hour format. - **Similar incidents** — earlier incidents of the same monitor with their durations, so you can see whether this is a pattern. ## Critical outranks warning Severity is decided per metric and per monitor, so one problem never turns into two incidents: - A preset that creates **two rules on one metric** — a warning and a critical threshold for response time, LCP, CLS and the like — opens only the **critical** incident when both thresholds are crossed; the warning rule does not fire on top of it. - While a **critical incident is open on a monitor**, **warning** alerts for that monitor are held back: their rules are still evaluated, but no separate warning incident opens and nothing is notified until the critical incident resolves. A warning that fires in the very same evaluation as the critical one is still recorded, because neither existed when the other was decided. The monitor shows **Down** throughout. A warning condition that coincides with the outage — say, slow responses while the site is returning errors — is visible in the check history, not as a second alert. ## Grouping and noise control When several incidents open (or resolve) within a short window — a host goes down and takes ten monitors with it — chat and email channels receive **one summary message per channel** instead of ten separate pings: > 🔴 **10 new incidents** > 🔴 api.acme.com — Probe failed to connect > 🔴 www.acme.com — Unexpected HTTP Status > … A channel's first window is short (about 30 seconds), so a lone incident still arrives promptly as a normal, fully detailed notification. Follow-ups batch for a few minutes into one further summary; recoveries group the same way. PagerDuty, Opsgenie, webhooks, Zapier and push apps stay **one event per incident** so their own lifecycle and automation keep working. Full details and the channel list: [Grouping during a storm](/docs/notifications#grouping-during-a-storm). ## Flapping A monitor that keeps opening and closing incidents within a short window is marked **flapping**. Incidents are still recorded and the timeline shows when flapping started and ended, but notifications are suppressed while it lasts, so a service bouncing every minute does not generate a message every minute. Fix the underlying instability or raise the monitor's confirmation threshold in its [Alerting tab](/docs/alerting#confirmation-settings). ## Maintenance and incidents During a [maintenance window](/docs/monitors/maintenance) checks keep running and incidents are still recorded, but their notifications are silenced and the incident carries a **Maintenance** badge; the window can also exclude the period from uptime. A finished outage that should not have counted — a deploy that went wrong — can be reclassified afterwards through the API: [Mark an incident as maintenance](/docs/api/incidents#mark-an-incident-as-maintenance). ## Related - [Root cause analysis](/docs/alerting/root-cause): What the incident page tells you about why it failed. - [Resolving incidents](/docs/alerting/resolving): Automatic recovery, source changes and manual resolve. - [On-call & escalations](/docs/alerting/on-call): How critical incidents page people until someone acknowledges. - [Incidents API](/docs/api/incidents): List, read, acknowledge and resolve incidents from scripts and AI agents. --- Canonical page: https://watchfor.io/docs/alerting/incidents · All docs: https://watchfor.io/docs · Site guide: https://watchfor.io/llms.txt