---
title: Who changed the monitor? Why your monitoring needs an audit trail
description: The postmortem question that has no answer in most monitoring setups: who disabled that alert, when, and what did it look like before? Here's why an audit trail on your monitoring config is as important as one on your production data — for debugging, for teams, and for the SOC 2 auditor.
canonical: https://watchfor.io/blog/monitoring-audit-trail
---

# Who changed the monitor? Why your monitoring needs an audit trail

2026-08-23 · WatchFor Team · Monitoring · monitoring, reliability, security, sre

The postmortem question that has no answer in most monitoring setups: who disabled that alert, when, and what did it look like before? Here's why an audit trail on your monitoring config is as important as one on your production data — for debugging, for teams, and for the SOC 2 auditor.

Every postmortem template has a section called "timeline." Here's a timeline entry that appears in more postmortems than anyone admits:

> *14:00 — Database alerts didn't fire because the rule had been disabled. Nobody knows when, or by whom, or why.*

The incident wasn't the outage — the incident was the **silent config change three weeks earlier**. A threshold "temporarily" relaxed during a noisy deploy and never restored. An alert muted during maintenance and forgotten. A notification channel deleted because "nobody uses that Slack channel anymore" — except the escalation policy did.

Your production *data* almost certainly has an audit trail. Your monitoring config — the thing that decides whether anyone finds out production broke — usually doesn't. That's backwards: monitoring configuration is **safety-critical configuration**, and changes to it deserve the same treatment as schema migrations.

## The three audiences of an audit trail

### 1. The debugging engineer: "what changed?"

Half of monitoring mysteries are config archaeology. Alerts stopped arriving — was the rule changed, the channel edited, the monitor's locations reduced? An alert fired that "can't fire" — did someone tighten the threshold yesterday? Without a log, these investigations are Slack archaeology and guesswork. With one, they're a filtered search: *show me every change to this monitor in the last month*.

The killer feature here isn't recording — it's **recording the before and after**. "Threshold changed" is trivia; "threshold changed *from 2s to 10s*" is the answer. Same for an alert rule's version at the moment an incident fired: if the rule was edited *after* the incident opened, honest diagnostics show the version that actually triggered, not today's version. (WatchFor pins the firing rule version to each incident's root cause for exactly this reason.)

### 2. The team: ownership without surveillance

On a team of five, monitoring config is shared state with no merge conflicts — the last writer silently wins. An activity log turns that from a trust problem into a coordination tool:

- **Handoffs stop losing context.** The on-call engineer seeing a muted monitor can check *who* muted it and *when* — and distinguish "muted 20 minutes ago by the person doing the deploy" from "muted six weeks ago and forgotten."
- **Bulk operations become reviewable.** Importing 40 monitors from a CSV, retargeting maintenance windows, bulk-deleting contacts — one log entry with counts turns "what just happened to our config?" into a non-question.
- **Nobody has to ask "did you...?"** The log answers before the question is asked. That's not surveillance — it's the same courtesy as a git history.

The requirement this creates: the log must be **readable by humans**. "user 8f3a…e2 updated object 41" is technically an audit trail and practically useless. Entries need names, monitor titles, locations as city names, values with units — a sentence a teammate can read, not a foreign key dump.

### 3. The auditor: compliance without a scramble

If you sell to companies with security reviews, SOC 2 / ISO 27001 questions eventually reach monitoring: *how do you ensure alerting isn't silently disabled? Who can modify it, and is that logged?* Teams without a trail answer this with an awkward paragraph about "process." Teams with one answer with a screenshot.

The auditor's checklist maps directly onto what the trail must capture:

| Requirement | What it means concretely |
| --- | --- |
| **Completeness** | *Every* mutating action logged — monitors, alert rules, channels, contacts, status pages, maintenance windows, on-call schedules, member changes. A trail with gaps is a trail an auditor discounts. |
| **Attribution** | Which human (not which API key) made the change. |
| **Immutability** | The log is append-only from the app's perspective — no edit-your-own-history. |
| **Sensitive-value hygiene** | Config changes are logged, but secrets inside them (webhook URLs, tokens) are redacted in the trail itself. An audit log that leaks credentials is its own finding. |
| **Access control** | Reading the trail is itself a privileged operation (owners/admins). |

## What "good" looks like in practice

A monitoring audit trail earns its keep when it's *used*, and usage follows a few design choices we've landed on building WatchFor's [activity log](/docs/organization/activity-log):

- **Plain-language entries** — "*Maria paused monitor api.acme.com*", "*threshold: 2s → 10s*", "*locations: 🇩🇪 Frankfurt, 🇺🇸 Ashburn → 🇩🇪 Frankfurt*". Before/after diffs on every update, in the units the UI uses.
- **Everything in one stream, filterable** — by entity type (monitors, rules, channels, on-call, maintenance…) and by action, because "show me every *deleted* thing this month" is the question after a mystery.
- **Lifecycle events included** — incidents acknowledged and resolved (by whom, with what note), maintenance windows started and extended, on-call overrides created and removed. Config *and* operations, one timeline.
- **Cross-referenced where it matters** — incident numbers link to incidents; a rule change shows which monitor it belongs to; notification history answers *what was sent* while the activity log answers *who changed what* — two different questions, deliberately separate views.

None of this is exotic engineering. It's the decision that monitoring config is production config — followed by the same discipline you already apply to production: record every change, show who and what, keep it readable, keep it complete.

The next time an alert doesn't fire, the question won't be "who changed the monitor?" It'll be a search box. That's the entire feature.

---

Canonical page: https://watchfor.io/blog/monitoring-audit-trail · All articles: https://watchfor.io/blog/all.md · Site guide: https://watchfor.io/llms.txt