---
title: Scheduling 25 million checks with one Go binary — and no next_run_at column
description: Most schedulers store the next run time in a database and fight write amplification forever. Ours stores no per-run scheduling state at all: every job's schedule is a pure function, recomputable anywhere. Here's the design — a 256-shard time wheel, an append-only change log — and the honest numbers from load-testing it to 25 million active jobs.
canonical: https://watchfor.io/blog/scheduling-25-million-checks-one-go-binary
---

[All posts](/blog) [Engineering](/blog/category/engineering) Aug 06, 2026 · 6 min read · WatchFor Team

# Scheduling 25 million checks with one Go binary — and no next_run_at column

Most schedulers store the next run time in a database and fight write amplification forever. Ours stores no per-run scheduling state at all: every job's schedule is a pure function, recomputable anywhere. Here's the design — a 256-shard time wheel, an append-only change log — and the honest numbers from load-testing it to 25 million active jobs.

Every monitoring product has the same engine-room problem: something has to decide, every second, which checks are due right now — and keep deciding correctly through deploys, restarts and clock weirdness.

The obvious build is a database table with a next_run_at column and a polling loop. It works great until it doesn't. This post is about why we built ours differently: a single Go binary that holds every job in RAM, computes schedules instead of storing them, and was load-tested to 25 million active scheduled jobs on one node .

## The trap: storing the schedule

The textbook scheduler:

SELECT * FROM jobs WHERE next_run_at <= now () LIMIT 1000 ;
-- dispatch...
UPDATE jobs SET next_run_at = now () + interval WHERE id IN (...);
At monitoring scale this has three compounding problems:

- Write amplification. A 1-minute check rewrites its row 1,440 times a day. A million of them is ~17,000 row updates per second of pure bookkeeping.

- The poll/precision trade-off. Poll every second and the database melts; poll every 30 seconds and a "1-minute check" is really a 60–90 second check.

- State you can lose. next_run_at is state. State drifts, skews, and turns restarts into archaeology.

So we keep no per-run scheduling state anywhere. The schedule is computed .

## Schedules as pure functions

For an interval job, the next fire time is fully determined by the job's ID and interval — no history required:

// Simplified — the real version also handles cron
// expressions and per-job overrides.
func NextFire ( jobID int64 , interval , now int64 ) int64 {
// Deterministic per-job jitter: hash the ID, mod the
// interval. Job 42 always gets the same offset.
jitter := int64 ( fnvHash ( jobID )) % interval

// Snap to a global grid anchored at a fixed epoch.
k := ( now - anchor - jitter ) / interval
next := anchor + ( k + 1 ) * interval + jitter
if next <= now {
next += interval
}
return next
}
Two properties fall out that we now consider non-negotiable:

Restarts are boring. On startup the process loads job definitions (id, interval, target — the things that are real state, and live in Postgres) and recomputes every schedule from scratch. 25 million jobs cold-start in about 25 seconds. There's no "where were we?" recovery logic, because there's no stored position to recover.

Load spreads itself. The hash jitter deterministically smears a million 1-minute checks across the whole minute — identically on every run and every restart. No thundering herd, no jitter state to persist.

Postgres stays in the picture as a control plane : job definitions flow to the scheduler through an append-only change log it tails with a cursor, picked up within a few seconds. Nothing in the hot path ever writes to the database. (Cron expressions are evaluated against the scheduler's clock — UTC in production — with the same deterministic jitter applied.)

## The time wheel

Computing "when does job X fire next" is half the problem. The inverse — "which jobs fire this second?" — has to be answered without scanning 25 million jobs. That's a classic time wheel:

const WheelSize = 3600 // one-hour horizon, one-second resolution

type Shard struct {
jobs map [ int64 ] * Job
wheel [ WheelSize ][] int64 // slot -> job IDs due that second
farFuture map [ int64 ] int64 // fires beyond the horizon
lastTick int64
}
A job due at Unix time t sits in slot t % 3600 . Each second, the tick handler empties exactly one slot — the complete answer to "what's due now", zero searching. Dispatch recomputes the job's next fire (that pure function again) and drops it into a future slot. Jobs more than an hour out (daily checks, long crons) wait in the farFuture map and get promoted as their time approaches.

One detail worth being precise about: the tick loop processes every second since the last tick , not just "now". If the running process stalls — a GC pause, CPU steal — the next tick replays the missed slots in order; checks fire late by the stall, but they fire. A full process restart is different by design: the scheduler doesn't pretend to replay downtime. Checks that would have fired while it was down are skipped, and everything resumes on the normal grid — for a monitoring workload, a fresh check now beats a stale check about the past.

## 256 shards, one lock each

A single wheel behind a single mutex would serialize everything, so there are 256 independent shards — own jobs map, own wheel, own lock:

shardID := jobID % 256
Every second a ticker fans out one goroutine per shard; a shard's tick only ever touches that shard's lock, and change-log updates hash straight to their shard. Why 256? Honest engineering rather than numerology: comfortably more shards than cores so ticks parallelize anywhere, few enough that per-shard overhead is negligible, and a power of two. We tried nothing else because nothing else was needed.

## The watchdog, or: trusting yourself is a bug

An in-memory wheel has one failure mode that keeps you up at night: a job that's in the map but whose wheel entry got lost. Enabled, healthy-looking, and it will never fire again — silently.

We know it's not theoretical, because it happened. During an early bulk import, exactly 1 job out of 101 landed in the jobs map without a wheel entry and simply… never ran. No error, no crash — a monitor that looked configured while checking nothing, and a full evening staring at code that was "obviously correct".

The structural fix was in the import path, but the lasting lesson was bigger: any component that can fail silently needs a second component whose only job is distrust. Every couple of seconds a watchdog sweeps one shard (a full rotation across all 256 in under 9 minutes) for enabled jobs whose fire time drifted into the past with no wheel entry to explain it. It re-arms them and increments a metric — and the metric is the point. Zero is the only acceptable steady-state; any rearm is a bug we want to hear about, healed before a customer sees a gap in their uptime chart.

## The numbers

Load test on a generic desktop-class workstation — deliberately nothing exotic:

Metric Result

Active scheduled jobs resident in memory 25,000,000

Cold start (load definitions + schedule all) ~25 s

Dispatch throughput (due job → serialized envelope handed to the publisher, in-process) 150,000+ / s

Resident memory ~6 GB (~250 bytes/job)

To be clear about what this is: a capacity test of the scheduling engine, not our production traffic — production runs orders of magnitude below it. The headroom is the point: scheduling won't be the thing that breaks first for a very long time, and it costs one small process to keep that true.

## Trade-offs we accepted

Nothing above is free. The bill, itemized:

- RAM scales linearly with jobs (~250 bytes each), and cold start is a linear pass over all of them. Fine at 6 GB / 25 s; a design constraint you should know you're signing.

- One active instance by design. This is not an active-active HA story. What makes failover simple is determinism: dispatches carry run IDs derived from (job, execution time) , published with message deduplication on the bus — so if instances briefly overlap during a handover, the duplicates collapse downstream instead of double-probing anyone. Recovery is "start a replacement anywhere; ~25 s later it has independently reached the same conclusions about what fires when."

- Changes are eventually applied. A new or edited monitor takes effect within seconds (change-log tailing), not microseconds. For monitoring, that's the right trade.

- At-least-once dispatch, deduplicated by ID — not a distributed exactly-once protocol. We prefer idempotence over consensus.

## What we deliberately didn't build

No distributed scheduler, no coordination service, no queue cluster with its own on-call rotation. Distributed systems are a tax you pay when you must; a time wheel in 6 GB of RAM meant we didn't have to.

## What happens after dispatch

Scheduling is only the first domino: due checks fan out to probe servers around the world, results flow back through a message bus into analytics storage, and an alerting engine decides whether to wake you — including a feedback loop where an active incident temporarily speeds up that monitor's checks so you see recovery within seconds. That pipeline (dedup, flap detection, notification grouping) has enough war stories for a post of its own — it's next.

[#engineering](/blog/tag/engineering)[#golang](/blog/tag/golang)[#architecture](/blog/tag/architecture)[#monitoring](/blog/tag/monitoring)

## Start monitoring your services today

WatchFor checks HTTP, DNS, SSL, ping, email and 20+ more — from around the world, with alerts to Slack, Discord, email and beyond.

[Learn more](/docs/monitors)[Start free](/auth/sign-up)

Share this article

---

Canonical page: https://watchfor.io/blog/scheduling-25-million-checks-one-go-binary · Site guide: https://watchfor.io/llms.txt