Scheduling 25 million checks with one Go binary — and no next_run_at column

Every monitoring product has the same engine-room problem: something has to decide, every second, which checks are due right now — and keep deciding correctly through deploys, restarts and clock weirdness.

The obvious build is a database table with a next_run_at column and a polling loop. It works great until it doesn't. This post is about why we built ours differently: a single Go binary that holds every job in RAM, computes schedules instead of storing them, and was load-tested to 25 million active scheduled jobs on one node.

The trap: storing the schedule

The textbook scheduler:

SELECT * FROM jobs WHERE next_run_at <= now() LIMIT 1000;
-- dispatch...
UPDATE jobs SET next_run_at = now() + interval WHERE id IN (...);

At monitoring scale this has three compounding problems:

1. Write amplification. A 1-minute check rewrites its row 1,440 times a day. A million of them is ~17,000 row updates per second of pure bookkeeping.
2. The poll/precision trade-off. Poll every second and the database melts; poll every 30 seconds and a "1-minute check" is really a 60–90 second check.
3. State you can lose. next_run_at is state. State drifts, skews, and turns restarts into archaeology.

So we keep no per-run scheduling state anywhere. The schedule is computed.

Schedules as pure functions

For an interval job, the next fire time is fully determined by the job's ID and interval — no history required:

// Simplified — the real version also handles cron
// expressions and per-job overrides.
func NextFire(jobID int64, interval, now int64) int64 {
    // Deterministic per-job jitter: hash the ID, mod the
    // interval. Job 42 always gets the same offset.
    jitter := int64(fnvHash(jobID)) % interval

    // Snap to a global grid anchored at a fixed epoch.
    k := (now - anchor - jitter) / interval
    next := anchor + (k+1)*interval + jitter
    if next <= now {
        next += interval
    }
    return next
}

Two properties fall out that we now consider non-negotiable:

Restarts are boring. On startup the process loads job definitions (id, interval, target — the things that are real state, and live in Postgres) and recomputes every schedule from scratch. 25 million jobs cold-start in about 25 seconds. There's no "where were we?" recovery logic, because there's no stored position to recover.

Load spreads itself. The hash jitter deterministically smears a million 1-minute checks across the whole minute — identically on every run and every restart. No thundering herd, no jitter state to persist.

Postgres stays in the picture as a control plane: job definitions flow to the scheduler through an append-only change log it tails with a cursor, picked up within a few seconds. Nothing in the hot path ever writes to the database. (Cron expressions are evaluated against the scheduler's clock — UTC in production — with the same deterministic jitter applied.)

The time wheel

Computing "when does job X fire next" is half the problem. The inverse — "which jobs fire this second?" — has to be answered without scanning 25 million jobs. That's a classic time wheel:

const WheelSize = 3600 // one-hour horizon, one-second resolution

type Shard struct {
    jobs      map[int64]*Job
    wheel     [WheelSize][]int64 // slot -> job IDs due that second
    farFuture map[int64]int64    // fires beyond the horizon
    lastTick  int64
}

A job due at Unix time t sits in slot t % 3600. Each second, the tick handler empties exactly one slot — the complete answer to "what's due now", zero searching. Dispatch recomputes the job's next fire (that pure function again) and drops it into a future slot. Jobs more than an hour out (daily checks, long crons) wait in the farFuture map and get promoted as their time approaches.

One detail worth being precise about: the tick loop processes every second since the last tick, not just "now". If the running process stalls — a GC pause, CPU steal — the next tick replays the missed slots in order; checks fire late by the stall, but they fire. A full process restart is different by design: the scheduler doesn't pretend to replay downtime. Checks that would have fired while it was down are skipped, and everything resumes on the normal grid — for a monitoring workload, a fresh check now beats a stale check about the past.

256 shards, one lock each

A single wheel behind a single mutex would serialize everything, so there are 256 independent shards — own jobs map, own wheel, own lock:

shardID := jobID % 256

Every second a ticker fans out one goroutine per shard; a shard's tick only ever touches that shard's lock, and change-log updates hash straight to their shard. Why 256? Honest engineering rather than numerology: comfortably more shards than cores so ticks parallelize anywhere, few enough that per-shard overhead is negligible, and a power of two. We tried nothing else because nothing else was needed.

The watchdog, or: trusting yourself is a bug

An in-memory wheel has one failure mode that keeps you up at night: a job that's in the map but whose wheel entry got lost. Enabled, healthy-looking, and it will never fire again — silently.

We know it's not theoretical, because it happened. During an early bulk import, exactly 1 job out of 101 landed in the jobs map without a wheel entry and simply… never ran. No error, no crash — a monitor that looked configured while checking nothing, and a full evening staring at code that was "obviously correct".

The structural fix was in the import path, but the lasting lesson was bigger: any component that can fail silently needs a second component whose only job is distrust. Every couple of seconds a watchdog sweeps one shard (a full rotation across all 256 in under 9 minutes) for enabled jobs whose fire time drifted into the past with no wheel entry to explain it. It re-arms them and increments a metric — and the metric is the point. Zero is the only acceptable steady-state; any rearm is a bug we want to hear about, healed before a customer sees a gap in their uptime chart.

The numbers

Load test on a generic desktop-class workstation — deliberately nothing exotic:

Metric	Result
Active scheduled jobs resident in memory	25,000,000
Cold start (load definitions + schedule all)	~25 s
Dispatch throughput (due job → serialized envelope handed to the publisher, in-process)	150,000+ / s
Resident memory	~6 GB (~250 bytes/job)

To be clear about what this is: a capacity test of the scheduling engine, not our production traffic — production runs orders of magnitude below it. The headroom is the point: scheduling won't be the thing that breaks first for a very long time, and it costs one small process to keep that true.

Trade-offs we accepted

Nothing above is free. The bill, itemized:

• RAM scales linearly with jobs (~250 bytes each), and cold start is a linear pass over all of them. Fine at 6 GB / 25 s; a design constraint you should know you're signing.
• One active instance by design. This is not an active-active HA story. What makes failover simple is determinism: dispatches carry run IDs derived from (job, execution time), published with message deduplication on the bus — so if instances briefly overlap during a handover, the duplicates collapse downstream instead of double-probing anyone. Recovery is "start a replacement anywhere; ~25 s later it has independently reached the same conclusions about what fires when."
• Changes are eventually applied. A new or edited monitor takes effect within seconds (change-log tailing), not microseconds. For monitoring, that's the right trade.
• At-least-once dispatch, deduplicated by ID — not a distributed exactly-once protocol. We prefer idempotence over consensus.

What we deliberately didn't build

No distributed scheduler, no coordination service, no queue cluster with its own on-call rotation. Distributed systems are a tax you pay when you must; a time wheel in 6 GB of RAM meant we didn't have to.

What happens after dispatch

Scheduling is only the first domino: due checks fan out to probe servers around the world, results flow back through a message bus into analytics storage, and an alerting engine decides whether to wake you — including a feedback loop where an active incident temporarily speeds up that monitor's checks so you see recovery within seconds. That pipeline (dedup, flap detection, notification grouping) has enough war stories for a post of its own — it's next.