---
title: Uptime monitoring in Grafana: scrape WatchFor with Prometheus
description: Your Grafana already shows CPU, queues and error rates. What it usually can't show is whether a customer in another country can reach you right now. Here's how to put WatchFor's outside view (confirmed status, incident-based uptime, certificate and domain expiry, heartbeats) into your own Prometheus: the five-line scrape config, the PromQL worth keeping, alert rules that don't double-page, the dashboard, and when the open-source exporter is the better fit.
canonical: https://watchfor.io/blog/prometheus-grafana-uptime-monitoring
---

# Uptime monitoring in Grafana: scrape WatchFor with Prometheus

2026-10-02 · WatchFor Team · Monitoring · prometheus, grafana, uptime, observability, integrations

Your Grafana already shows CPU, queues and error rates. What it usually can't show is whether a customer in another country can reach you right now. Here's how to put WatchFor's outside view (confirmed status, incident-based uptime, certificate and domain expiry, heartbeats) into your own Prometheus: the five-line scrape config, the PromQL worth keeping, alert rules that don't double-page, the dashboard, and when the open-source exporter is the better fit.

Most teams that run Prometheus have a Grafana screen on the wall, and it is
very good at one kind of question: *is our stuff healthy from the inside?*
CPU, memory, request rates, queue depth, p99 latency. What it is bad at is
the question customers actually ask: *can I reach you from here, right now?*

Those two disagree more often than you'd think. A broken certificate chain on
one edge, a DNS change that hasn't reached half of Asia, a login page whose
JavaScript dies after a deploy: every internal graph stays green, because
from inside your network everything works.

WatchFor checks from the outside, from 20 locations in 7 regions, and only
calls something down once a failure is confirmed from more than one of them.
Since this week that view is also a Prometheus target. This post shows how
to wire it in, which queries are worth keeping, how to alert without paging
twice, and what the ready-made Grafana dashboard looks like on a real
organization.

![The WatchFor Overview dashboard in Grafana for a demo organization: fleet health 96, 22 monitors up and one down, monitors by status, health score and open incidents over the night, and a Needs attention table listing the failing login check](/blog/prometheus-grafana-uptime-monitoring-dashboard.png)

## The five-minute version

On the Pro plan and above, `GET /api/v1/metrics` answers in the Prometheus
text format. Create a **read** API key under **Settings → API keys**, save it
in a file Prometheus can read, and add one job:

```yaml
scrape_configs:
  - job_name: watchfor
    scrape_interval: 60s
    scheme: https
    metrics_path: /api/v1/metrics
    authorization:
      credentials_file: /etc/prometheus/watchfor-api-key
    static_configs:
      - targets: ["watchfor.io"]
```

Reload Prometheus, and the target turns green on the next scrape. Grafana
Alloy, the OpenTelemetry Collector's Prometheus receiver and VictoriaMetrics
`vmagent` take the same three things: the URL, a 60-second interval and the
bearer token.

Sixty seconds is not a guess. The response is built at most once a minute
per organization and served from a cache in between, so scraping every 10
seconds only stores the same numbers six times. The endpoint also has its
own limit of 30 requests a minute per organization, which a normal scrape
never comes near.

## What actually comes back

Here is one SSL monitor from the demo organization in the screenshots, exactly
as the endpoint returns it:

```text
watchfor_monitor_info{monitor_id="f6757003-…",name="wikipedia.org certificate",type="ssl",target="en.wikipedia.org",tags="production,ssl",strategy="roundrobin"} 1
watchfor_monitor_up{monitor_id="f6757003-…"} 1
watchfor_monitor_state{monitor_id="f6757003-…"} 2
watchfor_monitor_check_interval_seconds{monitor_id="f6757003-…"} 3600
watchfor_monitor_last_check_success{monitor_id="f6757003-…"} 1
watchfor_monitor_last_check_duration_seconds{monitor_id="f6757003-…"} 0.067
watchfor_monitor_uptime_ratio{monitor_id="f6757003-…",window="30d"} 1
watchfor_ssl_certificate_expiry_timestamp_seconds{monitor_id="f6757003-…"} 1793733340
```

Two design decisions are worth knowing before you write a single query.

**Every per-monitor series carries one label: `monitor_id`.** The name,
type, target and tags live only on `watchfor_monitor_info`, which is always
`1`. That is the standard Prometheus *info metric* pattern, and it's there
for two reasons. Renaming a monitor in WatchFor doesn't split its history
into two series. And cardinality stays flat: the organization above has 23
monitors and 4 hosts, and the whole response is 436 lines, roughly 15 series
per monitor, whatever you name things. When you want names in a result, join
them in:

```text
watchfor_monitor_up * on(monitor_id) group_left(name, type) watchfor_monitor_info
```

**"Up" means what it means in WatchFor.** `watchfor_monitor_up` is `1`
unless a *confirmed critical* incident is open on the monitor. A degraded
monitor still counts as up, and a single failed check from one location does
not make it `0`. Uptime follows the same rule: `watchfor_monitor_uptime_ratio`
counts confirmed critical downtime only, with maintenance windows left out,
so the number in Grafana is the number on the monitor's page and in your
monthly report. If you want the raw signal anyway, it's there as
`watchfor_monitor_last_check_success`; just don't page anyone on it.

## Six queries worth keeping

All of these run as-is against the endpoint (or against the exporter, which
adds an `organization` label you can ignore or filter on).

**What is down right now, by name:**

```text
(watchfor_monitor_up == 0) * on(monitor_id) group_left(name, type) watchfor_monitor_info
```

**Days until each certificate expires.** Sorting this ascending is the
cheapest way to never have a certificate outage again:

```text
round((watchfor_ssl_certificate_expiry_timestamp_seconds - time()) / 86400)
  * on(monitor_id) group_left(name) watchfor_monitor_info
```

`watchfor_domain_expiry_timestamp_seconds` works the same way for domain
registrations, which tend to expire on a Saturday, a year after the person
who registered them left.

**Error budget left against a 99.9% SLO,** where `1` is untouched, `0` is
spent and anything negative is a breach:

```text
(watchfor_monitor_uptime_ratio{window="30d"} - 0.999) / (1 - 0.999)
```

**The slowest HTTP and API endpoints over the last day.** Filter on type
here: a browser journey that signs in and checks out legitimately takes
seconds, and would otherwise top every list:

```text
topk(5,
  watchfor_monitor_avg_duration_24h_seconds
    * on(monitor_id) group_left(name) watchfor_monitor_info{type=~"http|api"}
)
```

**How long ago each cron job checked in.** For jobs and backups that ping
WatchFor, this is the age of the last real check-in, which is the column you
want on a wall screen:

```text
(time() - watchfor_heartbeat_last_ping_timestamp_seconds)
  * on(monitor_id) group_left(name) watchfor_monitor_info
```

Don't build the "it's late" alert on this yourself. WatchFor knows each
job's schedule, including cron expressions and time zones, and opens an
incident the moment a run is missed or reports a failure, so
`watchfor_monitor_up == 0` already covers it.

**The whole fleet in one number:** `watchfor_health_score` is the same 0–100
score the dashboard shows, and `watchfor_monitors{status="down"}` is the
count behind it. Both are organization totals, so they ignore any filter you
put on the scrape.

## Alerting without paging twice

WatchFor already confirms an outage from several locations before it opens
an incident, and it can page people itself. So decide where paging lives
before you write rules, or the same outage wakes the on-call engineer twice.

A setup that works well: let WatchFor page for outages (it knows about
confirmation, maintenance windows and on-call schedules), and use
Prometheus for the slow-burning things that belong in a ticket queue or a
team channel:

```yaml
groups:
  - name: watchfor
    rules:
      - alert: CertificateExpiresSoon
        expr: (watchfor_ssl_certificate_expiry_timestamp_seconds - time()) / 86400 < 14
        labels: { severity: ticket }
        annotations:
          summary: "Certificate on {{ $labels.monitor_id }} expires in {{ $value | humanize }} days"

      - alert: DomainExpiresSoon
        expr: (watchfor_domain_expiry_timestamp_seconds - time()) / 86400 < 30
        labels: { severity: ticket }

      - alert: ErrorBudgetBurning
        expr: watchfor_monitor_uptime_ratio{window="30d"} < 0.999
        labels: { severity: ticket }
```

If Alertmanager *is* your paging system, alert on
`watchfor_monitor_up == 0` with a short `for: 1m`, and turn off paging in
WatchFor for those monitors. What you should never do is alert on
`watchfor_monitor_last_check_success`: that is one check from one location,
and the whole point of confirmation is that one bad network path is not an
outage.

## The dashboard

The screenshot at the top is the **WatchFor Overview** dashboard, running
against a small demo organization: HTTP and API checks, Playwright browser
journeys, SSL and domain monitors, DNS, a ping, two MCP servers, two
page-integrity checks and three cron-job heartbeats, plus four servers
running the agent. The red row is real: the practice login page that one
browser journey signs in to keeps going down overnight, and WatchFor opens
and closes an incident each time.

![Status timeline in Grafana over one night: one row per monitor, green while up and red while a confirmed incident is open. The login check flaps all night, a cron job reports one failed run around 01:20, and the blue band at 03:00 is a planned maintenance window](/blog/prometheus-grafana-uptime-monitoring-timeline.png)

From top to bottom it has:

- **Status:** fleet health, monitors by status, open incidents, active
  maintenance and how old the data is, then a **Needs attention** table that
  is empty on a good day, and a **status timeline** per monitor across the
  range you pick. The timeline is the panel people end up putting on the wall.
- **Uptime & SLO:** pick a target from the drop-down (99% to 99.99%) and see
  how many monitors meet it, which ones breach it, the error budget left over
  30 days, uptime per monitor for 24 hours, 7 and 30 days, and the monitors
  with the most downtime.
- **Performance:** check duration over time and a 24-hour table per monitor.
- **Certificates, domains & heartbeats:** expiry soonest first, amber under
  30 days and red under 7, and the time since each heartbeat's last check-in.
- **Hosts:** for servers running the WatchFor agent, state, inventory and an
  availability timeline.

![TLS certificates, domain registrations and heartbeats panels: days left per certificate and domain, soonest first, and the last ping of each cron job](/blog/prometheus-grafana-uptime-monitoring-expiry.png)

Every panel explains itself on hover, and the filters at the top
(organization, type, tag, monitor) narrow everything at once. Download
[watchfor-overview.json](/grafana/watchfor-overview.json) and import it under
**Dashboards → New → Import**. It works the same whether Prometheus scrapes
WatchFor directly or through the exporter.

## When the exporter is the better fit

Scraping the endpoint directly is the right default: one less moving part.
The open-source
[WatchFor Prometheus exporter](https://github.com/watchfor-io/prometheus-exporter)
exists for the cases where it isn't:

- **Several WatchFor organizations behind one target.** An agency or a
  platform team with one organization per client or per environment gets one
  scrape job, with an `organization` label added to every series.
- **Keys kept out of the Prometheus config,** for example in a Kubernetes
  Secret. The exporter re-reads key files, so rotating a key needs no
  restart.
- **A target inside the cluster,** next to everything else your Prometheus
  Operator already discovers.
- **Its own health:** `watchfor_exporter_up` and
  `watchfor_exporter_last_success_timestamp_seconds` per organization, so a
  revoked key or a failing poll is an alert, not a silent gap in a graph.

It's a single Go binary in a distroless image that reads the metrics
endpoint and nothing else, never the rest of the API. The repository ships a
`docker compose` file that starts the exporter, Prometheus and Grafana with
the dashboard already provisioned. That compose file is exactly what produced
the screenshots in this post:

```yaml
services:
  watchfor-exporter:
    image: ghcr.io/watchfor-io/prometheus-exporter:latest
    environment:
      WATCHFOR_API_KEY: ${WATCHFOR_API_KEY}
      WATCHFOR_ORGANIZATION: acme
    ports: ["10056:10056"]
    read_only: true
    cap_drop: [ALL]
```

For several organizations, a short YAML file lists each one with its own key
file. The README has that, the Kubernetes manifests and example alert rules.

## What about blackbox_exporter?

If you run Prometheus, you've probably met the
[blackbox exporter](https://github.com/prometheus/blackbox_exporter). It
probes endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC and reports
`probe_success`, and it's excellent at what it does. The difference is where
the probe runs and what happens after it:

| | blackbox_exporter | WatchFor |
| --- | --- | --- |
| Where checks run | Wherever you run the exporter | 20 locations in 7 regions |
| A failure becomes an outage | When your alert rule says so (`for:`) | Once it's confirmed from more than one location |
| Browser journeys (log in, check out) | No | Playwright checks |
| Certificate expiry | `probe_ssl_earliest_cert_expiry` | Yes, plus domain registration expiry |
| Uptime that matches a report | You compute it from probe results | Incident-based, maintenance excluded |
| What you operate | The exporter, its config, one copy per vantage point | One scrape job |

They are not really competitors. Keep blackbox_exporter for the things only
your network can see: internal services, private endpoints, the database
port. Use WatchFor for the view from where your customers are. Both end up
in the same Prometheus, on the same screen.

## Limits, honestly

- **It's a minute behind, by design.** The response refreshes once a
  minute. For paging, WatchFor's own alerts are faster, because they fire
  when the incident opens rather than on the next scrape.
- **It's the Pro plan and above.** On other plans the endpoint answers `403`
  with an upgrade message; the rest of the API is unaffected.
- **Uptime is incident-based.** A monitor with a dozen failed single checks
  and no confirmed incident has 100% uptime. That is deliberate, and it's the
  same rule everywhere in WatchFor, but it can surprise you if you're used to
  counting `probe_success` samples.
- **The `organization` label only exists through the exporter.** Scraping
  several organizations directly? Add a static `organization` label per
  target in `prometheus.yml`.

The [Prometheus docs](/docs/api/prometheus) list every metric and label,
the filters you can put on a scrape (`tag`, `type`, `monitor_id`,
`include_hosts=false`) and the full alert-rule examples.

---

Canonical page: https://watchfor.io/blog/prometheus-grafana-uptime-monitoring · All articles: https://watchfor.io/blog/all.md · Site guide: https://watchfor.io/llms.txt