---
title: Scheduling 25 million checks with one Go binary — and no next_run_at column
description: Most schedulers store the next run time in a database and fight write amplification forever. Ours stores no per-run scheduling state at all: every job's schedule is a pure function, recomputable anywhere. Here's the design — a 256-shard time wheel, an append-only change log — and the honest numbers from load-testing it to 25 million active jobs.
canonical: https://watchfor.io/blog/scheduling-25-million-checks-one-go-binary
---

# Scheduling 25 million checks with one Go binary — and no next_run_at column

2026-08-06 · WatchFor Team · Engineering · engineering, golang, architecture, monitoring

Most schedulers store the next run time in a database and fight write amplification forever. Ours stores no per-run scheduling state at all: every job's schedule is a pure function, recomputable anywhere. Here's the design — a 256-shard time wheel, an append-only change log — and the honest numbers from load-testing it to 25 million active jobs.

Every monitoring product has the same engine-room problem: something has to decide, every second, which checks are due *right now* — and keep deciding correctly through deploys, restarts and clock weirdness.

The obvious build is a database table with a `next_run_at` column and a polling loop. It works great until it doesn't. This post is about why we built ours differently: a single Go binary that holds every job in RAM, computes schedules instead of storing them, and was load-tested to **25 million active scheduled jobs on one node**.

## The trap: storing the schedule

The textbook scheduler:

```sql
SELECT * FROM jobs WHERE next_run_at <= now() LIMIT 1000;
-- dispatch...
UPDATE jobs SET next_run_at = now() + interval WHERE id IN (...);
```

At monitoring scale this has three compounding problems:

1. **Write amplification.** A 1-minute check rewrites its row 1,440 times a day. A million of them is ~17,000 row updates *per second* of pure bookkeeping.
2. **The poll/precision trade-off.** Poll every second and the database melts; poll every 30 seconds and a "1-minute check" is really a 60–90 second check.
3. **State you can lose.** `next_run_at` is state. State drifts, skews, and turns restarts into archaeology.

So we keep no per-run scheduling state anywhere. The schedule is **computed**.

![WatchFor scheduler architecture: Postgres control plane feeds an append-only change log tailed by one Go binary — a 3600-slot time wheel across 256 shards — dispatching through a message bus to probe servers, analytics and alerting, with an override feedback loop](/blog/scheduler-architecture-diagram.svg)

## Schedules as pure functions

For an interval job, the next fire time is fully determined by the job's ID and interval — no history required:

```go
// Simplified — the real version also handles cron
// expressions and per-job overrides.
func NextFire(jobID int64, interval, now int64) int64 {
    // Deterministic per-job jitter: hash the ID, mod the
    // interval. Job 42 always gets the same offset.
    jitter := int64(fnvHash(jobID)) % interval

    // Snap to a global grid anchored at a fixed epoch.
    k := (now - anchor - jitter) / interval
    next := anchor + (k+1)*interval + jitter
    if next <= now {
        next += interval
    }
    return next
}
```

Two properties fall out that we now consider non-negotiable:

**Restarts are boring.** On startup the process loads job *definitions* (id, interval, target — the things that are real state, and live in Postgres) and recomputes every schedule from scratch. 25 million jobs cold-start in about 25 seconds. There's no "where were we?" recovery logic, because there's no stored position to recover.

**Load spreads itself.** The hash jitter deterministically smears a million 1-minute checks across the whole minute — identically on every run and every restart. No thundering herd, no jitter state to persist.

Postgres stays in the picture as a **control plane**: job definitions flow to the scheduler through an append-only change log it tails with a cursor, picked up within a few seconds. Nothing in the hot path ever writes to the database. (Cron expressions are evaluated against the scheduler's clock — UTC in production — with the same deterministic jitter applied.)

## The time wheel

Computing "when does job X fire next" is half the problem. The inverse — "which jobs fire *this* second?" — has to be answered without scanning 25 million jobs. That's a classic time wheel:

```go
const WheelSize = 3600 // one-hour horizon, one-second resolution

type Shard struct {
    jobs      map[int64]*Job
    wheel     [WheelSize][]int64 // slot -> job IDs due that second
    farFuture map[int64]int64    // fires beyond the horizon
    lastTick  int64
}
```

A job due at Unix time `t` sits in slot `t % 3600`. Each second, the tick handler empties exactly one slot — the complete answer to "what's due now", zero searching. Dispatch recomputes the job's next fire (that pure function again) and drops it into a future slot. Jobs more than an hour out (daily checks, long crons) wait in the `farFuture` map and get promoted as their time approaches.

One detail worth being precise about: the tick loop processes **every second since the last tick**, not just "now". If the *running* process stalls — a GC pause, CPU steal — the next tick replays the missed slots in order; checks fire late by the stall, but they fire. A full process restart is different by design: the scheduler doesn't pretend to replay downtime. Checks that would have fired while it was down are skipped, and everything resumes on the normal grid — for a monitoring workload, a fresh check now beats a stale check about the past.

## 256 shards, one lock each

A single wheel behind a single mutex would serialize everything, so there are **256 independent shards** — own jobs map, own wheel, own lock:

```go
shardID := jobID % 256
```

Every second a ticker fans out one goroutine per shard; a shard's tick only ever touches that shard's lock, and change-log updates hash straight to their shard. Why 256? Honest engineering rather than numerology: comfortably more shards than cores so ticks parallelize anywhere, few enough that per-shard overhead is negligible, and a power of two. We tried nothing else because nothing else was needed.

## The watchdog, or: trusting yourself is a bug

An in-memory wheel has one failure mode that keeps you up at night: a job that's *in the map* but whose wheel entry got lost. Enabled, healthy-looking, and it will never fire again — silently.

We know it's not theoretical, because it happened. During an early bulk import, exactly 1 job out of 101 landed in the jobs map without a wheel entry and simply… never ran. No error, no crash — a monitor that looked configured while checking nothing, and a full evening staring at code that was "obviously correct".

The structural fix was in the import path, but the lasting lesson was bigger: **any component that can fail silently needs a second component whose only job is distrust.** Every couple of seconds a watchdog sweeps one shard (a full rotation across all 256 in under 9 minutes) for enabled jobs whose fire time drifted into the past with no wheel entry to explain it. It re-arms them and increments a metric — and the metric is the point. Zero is the only acceptable steady-state; any rearm is a bug we want to hear about, healed before a customer sees a gap in their uptime chart.

## The numbers

Load test on a generic desktop-class workstation — deliberately nothing exotic:

| Metric | Result |
|---|---|
| Active scheduled jobs resident in memory | 25,000,000 |
| Cold start (load definitions + schedule all) | ~25 s |
| Dispatch throughput (due job → serialized envelope handed to the publisher, in-process) | 150,000+ / s |
| Resident memory | ~6 GB (~250 bytes/job) |

To be clear about what this is: a **capacity test of the scheduling engine, not our production traffic** — production runs orders of magnitude below it. The headroom is the point: scheduling won't be the thing that breaks first for a very long time, and it costs one small process to keep that true.

## Trade-offs we accepted

Nothing above is free. The bill, itemized:

- **RAM scales linearly** with jobs (~250 bytes each), and cold start is a linear pass over all of them. Fine at 6 GB / 25 s; a design constraint you should know you're signing.
- **One active instance by design.** This is not an active-active HA story. What makes failover simple is determinism: dispatches carry run IDs derived from `(job, execution time)`, published with message deduplication on the bus — so if instances briefly overlap during a handover, the duplicates collapse downstream instead of double-probing anyone. Recovery is "start a replacement anywhere; ~25 s later it has independently reached the same conclusions about what fires when."
- **Changes are eventually applied.** A new or edited monitor takes effect within seconds (change-log tailing), not microseconds. For monitoring, that's the right trade.
- **At-least-once dispatch, deduplicated by ID** — not a distributed exactly-once protocol. We prefer idempotence over consensus.

## What we deliberately didn't build

No distributed scheduler, no coordination service, no queue cluster with its own on-call rotation. Distributed systems are a tax you pay when you must; a time wheel in 6 GB of RAM meant we didn't have to.

## What happens after dispatch

Scheduling is only the first domino: due checks fan out to probe servers around the world, results flow back through a message bus into analytics storage, and an alerting engine decides whether to wake you — including a feedback loop where an active incident temporarily *speeds up* that monitor's checks so you see recovery within seconds. That pipeline (dedup, flap detection, notification grouping) has enough war stories for a post of its own: [how we confirm an outage before paging anyone](/blog/confirm-outage-before-paging).

---

Canonical page: https://watchfor.io/blog/scheduling-25-million-checks-one-go-binary · All articles: https://watchfor.io/blog/all.md · Site guide: https://watchfor.io/llms.txt