All posts
Monitoring5 min readWatchFor Team

Six scheduled jobs that will eventually burn you — and the exact check-in for each

Backups, certificate renewal, queue workers, ETL, cleanup jobs, billing runs — every team runs them, every team has been burned by one going silent. Recipes with the exact ping patterns: exit codes, run duration, output capture, and the grace period that fits each job.

Six scheduled jobs that will eventually burn you — and the exact check-in for each

There's a specific flavor of incident that never pages anyone: the scheduled job that stops running. No error — the crontab entry was lost in a server migration, the worker OOM-killed and never restarted, the timer's WantedBy was misspelled. The absence of a thing produces no signal. You find out when you need the thing: the restore that has no backup, the certificate that expired, the queue three days deep.

The fix is heartbeat monitoring — the job pings a unique URL when it runs; the monitor alerts when the ping doesn't arrive. That's the concept. This post is the practice: six real jobs, and the exact check-in pattern each one needs, because a backup and a queue worker fail differently and deserve different instrumentation.

All recipes use WatchFor's ping conventions — a plain GET is success, /<exit-code> reports the process result, /start + ?rid= measures duration, and a POST body attaches output — but the patterns port to any heartbeat service.

1. The nightly backup — the one that must never silently rot

The failure modes: the cron entry vanishes (silence), or the dump starts failing (error), or it "succeeds" in 4 seconds because the disk was full and it wrote nothing (bad duration). Instrument all three:

RID=$(date +%s)
curl -fsS "https://ingest.watchfor.io/p/<token>/start?rid=$RID"
pg_dump mydb | gzip > /backups/mydb-$(date +%F).sql.gz
curl -fsS --data-binary "$(ls -lh /backups | tail -3)" \
  "https://ingest.watchfor.io/p/<token>/$??rid=$RID"

Three things at once: the $? reports the real exit code (a failed dump is a "run failed" alert, distinct from "missed schedule"), the rid pairs start/finish into a measured duration you can alert on — a 40-minute backup finishing in 4 seconds is a red flag even with exit code 0 — and the POST body attaches the last log lines, so the 7am alert already shows what happened.

Schedule: cron expression matching the real crontab (0 3 * * *), timezone-aware, grace 30–60 min (backups have variance). Alert on: missed schedule, non-zero exit, duration above and suspiciously below normal.

2. Certificate renewal — the quiet job with a loud failure

Certbot renewals run twice daily and succeed for months — until a DNS change breaks the ACME challenge, and the renewal failure only becomes visible as an expiry 30 days later. Hook the deploy step:

certbot renew --deploy-hook \
  'curl -fsS https://ingest.watchfor.io/p/<token>'

The deploy hook fires only when a certificate is actually renewed — so set the heartbeat's expected interval to your renewal cadence (for 90-day certificates renewed at 60, expect a ping at least every ~30 days with a generous grace). Pair it with an SSL monitor on the endpoints themselves: the heartbeat tells you renewal stopped working weeks before the certificate monitor's expiry countdown gets urgent — one alerts on the cause, the other guards the effect.

3. The queue worker — a daemon disguised as a job

Workers aren't scheduled; they're supposed to always be running — which makes their death perfectly silent (the queue just grows). Give the worker loop a periodic pulse:

# inside the worker's main loop, every ~60s
requests.get(f"https://ingest.watchfor.io/p/{token}", timeout=5)

Schedule: interval mode, every 1–5 minutes, short grace. The alert fires within minutes of the process dying — Friday 19:00, not Monday 09:00. For fleets, one heartbeat per worker group is usually right (any worker pings; silence means all died — the case that matters). Complete the picture with a queue-depth extraction on your API — the worker heartbeat catches dead consumers, the depth threshold catches slow ones.

4. The ETL / data pipeline — where duration is the health metric

Pipelines rarely just die; they degrade — the nightly sync that took 20 minutes takes 3 hours, blowing past the window where reports are built from its output. Duration tracking is the point:

RID=$RANDOM
curl -fsS "https://ingest.watchfor.io/p/<token>/start?rid=$RID"
python run_pipeline.py; RC=$?
curl -fsS "https://ingest.watchfor.io/p/<token>/$RC?rid=$RID"

Run durations chart over time, so you watch the trend crawl upward across weeks — and set the duration alert at your real deadline ("must finish by 06:00 → alert if a 02:00 start runs over 3h"). The chart also settles the "was it always this slow?" argument with data.

5. The cleanup job — low stakes, until it isn't

Log rotation, temp-file purges, session sweeps: jobs nobody thinks about, whose absence surfaces as a full disk or a bloated table weeks later — usually as a mysterious outage of something else entirely. These need the cheapest possible instrumentation, or nobody will add it:

0 4 * * * /opt/scripts/cleanup.sh && curl -fsS https://ingest.watchfor.io/p/<token>

The && means "ping only on success" — failure or absence both surface as a missed schedule. One line in the crontab, grace of a few hours (nobody needs a 4am page about log rotation — but Tuesday's daytime alert beats next month's full disk).

6. The billing / invoice run — the one with an audience

Monthly jobs are the hardest to babysit by memory ("did the invoice run happen on the 1st?") and the most embarrassing to miss. Cron schedule 0 6 1 * *, timezone-aware — and this is where failure semantics matter most:

invoice_run && curl -fsS "https://ingest.watchfor.io/p/<token>" \
  || curl -fsS --data-binary "exit=$? see /var/log/billing.log" \
       "https://ingest.watchfor.io/p/<token>/fail"

An explicit /fail with attached context pages immediately with the reason — no waiting for the schedule window to expire. A missed monthly window discovered by the grace timeout is already a day late; a failure reported the moment it happens is a same-morning fix.

The pattern

Six jobs, one instrumentation ladder — climb only as high as the job deserves:

  1. Ping on success (&& + curl) — catches silence and failure. Ten seconds to add.
  2. + exit codes (/$?) — separates "didn't run" from "ran and failed", which are different pages.
  3. + duration (/start + rid) — for jobs where slow is broken.
  4. + output capture (POST body) — the alert arrives carrying its own diagnosis.

A single missed window opens an incident immediately (a daily job that missed once has already lost a day), recovery closes it on the next successful check-in, and every ping's output is on record. Each heartbeat is a regular monitor on every WatchFor plan — including Free — and the ping URLs are Healthchecks-compatible, so existing scripts migrate by swapping the domain. Start with recipe 1 tonight; it's the one whose absence you'll otherwise discover during a restore.

#monitoring#automation#reliability#devops

Share this article