--- title: Six scheduled jobs that will eventually burn you — and the exact check-in for each description: Backups, certificate renewal, queue workers, ETL, cleanup jobs, billing runs — every team runs them, every team has been burned by one going silent. Recipes with the exact ping patterns: exit codes, run duration, output capture, and the grace period that fits each job. canonical: https://watchfor.io/blog/cron-monitoring-recipes --- # Six scheduled jobs that will eventually burn you — and the exact check-in for each 2026-08-23 · WatchFor Team · Monitoring · monitoring, automation, reliability, devops Backups, certificate renewal, queue workers, ETL, cleanup jobs, billing runs — every team runs them, every team has been burned by one going silent. Recipes with the exact ping patterns: exit codes, run duration, output capture, and the grace period that fits each job. There's a specific flavor of incident that never pages anyone: the scheduled job that stops running. No error — the crontab entry was lost in a server migration, the worker OOM-killed and never restarted, the timer's `WantedBy` was misspelled. The absence of a thing produces no signal. You find out when you *need* the thing: the restore that has no backup, the certificate that expired, the queue three days deep. The fix is [heartbeat monitoring](/blog/heartbeat-cron-monitoring) — the job pings a unique URL when it runs; the monitor alerts when the ping *doesn't* arrive. That's the concept. This post is the practice: six real jobs, and the exact check-in pattern each one needs, because a backup and a queue worker fail differently and deserve different instrumentation. All recipes use WatchFor's ping conventions — a plain `GET` is success, `/` reports the process result, `/start` + `?rid=` measures duration, and a `POST` body attaches output — but the patterns port to any heartbeat service. ## 1. The nightly backup — the one that must never silently rot The failure modes: the cron entry vanishes (silence), *or* the dump starts failing (error), *or* it "succeeds" in 4 seconds because the disk was full and it wrote nothing (bad duration). Instrument all three: ```bash RID=$(date +%s) curl -fsS "https://ingest.watchfor.io/p//start?rid=$RID" pg_dump mydb | gzip > /backups/mydb-$(date +%F).sql.gz curl -fsS --data-binary "$(ls -lh /backups | tail -3)" \ "https://ingest.watchfor.io/p//$??rid=$RID" ``` Three things at once: the `$?` reports the **real exit code** (a failed dump is a *"run failed"* alert, distinct from *"missed schedule"*), the `rid` pairs start/finish into a **measured duration** you can alert on — a 40-minute backup finishing in 4 seconds is a red flag even with exit code 0 — and the POST body attaches the last log lines, so the 7am alert already shows *what* happened. **Schedule:** cron expression matching the real crontab (`0 3 * * *`), timezone-aware, grace 30–60 min (backups have variance). **Alert on:** missed schedule, non-zero exit, duration above *and* suspiciously below normal. ## 2. Certificate renewal — the quiet job with a loud failure Certbot renewals run twice daily and succeed for months — until a DNS change breaks the ACME challenge, and the *renewal* failure only becomes visible as an *expiry* 30 days later. Hook the deploy step: ```bash certbot renew --deploy-hook \ 'curl -fsS https://ingest.watchfor.io/p/' ``` The deploy hook fires only when a certificate is actually renewed — so set the heartbeat's expected interval to your renewal cadence (for 90-day certificates renewed at 60, expect a ping at least every ~30 days with a generous grace). Pair it with an [SSL monitor](/ssl-monitoring) on the endpoints themselves: the heartbeat tells you renewal *stopped working* weeks before the certificate monitor's expiry countdown gets urgent — one alerts on the cause, the other guards the effect. ## 3. The queue worker — a daemon disguised as a job Workers aren't scheduled; they're *supposed to always be running* — which makes their death perfectly silent (the queue just grows). Give the worker loop a periodic pulse: ```python # inside the worker's main loop, every ~60s requests.get(f"https://ingest.watchfor.io/p/{token}", timeout=5) ``` **Schedule:** interval mode, every 1–5 minutes, short grace. The alert fires within minutes of the process dying — Friday 19:00, not Monday 09:00. For fleets, one heartbeat per worker *group* is usually right (any worker pings; silence means *all* died — the case that matters). Complete the picture with a queue-depth extraction on your API — the worker heartbeat catches dead consumers, the depth threshold catches *slow* ones. ## 4. The ETL / data pipeline — where duration is the health metric Pipelines rarely just die; they *degrade* — the nightly sync that took 20 minutes takes 3 hours, blowing past the window where reports are built from its output. Duration tracking is the point: ```bash RID=$RANDOM curl -fsS "https://ingest.watchfor.io/p//start?rid=$RID" python run_pipeline.py; RC=$? curl -fsS "https://ingest.watchfor.io/p//$RC?rid=$RID" ``` Run durations chart over time, so you watch the trend crawl upward across weeks — and set the duration alert at your real deadline ("must finish by 06:00 → alert if a 02:00 start runs over 3h"). The chart also settles the "was it always this slow?" argument with data. ## 5. The cleanup job — low stakes, until it isn't Log rotation, temp-file purges, session sweeps: jobs nobody thinks about, whose absence surfaces as a full disk or a bloated table *weeks* later — usually as a mysterious outage of something else entirely. These need the cheapest possible instrumentation, or nobody will add it: ```bash 0 4 * * * /opt/scripts/cleanup.sh && curl -fsS https://ingest.watchfor.io/p/ ``` The `&&` means "ping only on success" — failure *or* absence both surface as a missed schedule. One line in the crontab, grace of a few hours (nobody needs a 4am page about log rotation — but Tuesday's daytime alert beats next month's full disk). ## 6. The billing / invoice run — the one with an audience Monthly jobs are the hardest to babysit by memory ("did the invoice run happen on the 1st?") and the most embarrassing to miss. Cron schedule `0 6 1 * *`, timezone-aware — and this is where **failure semantics** matter most: ```bash invoice_run && curl -fsS "https://ingest.watchfor.io/p/" \ || curl -fsS --data-binary "exit=$? see /var/log/billing.log" \ "https://ingest.watchfor.io/p//fail" ``` An explicit `/fail` with attached context pages immediately with the reason — no waiting for the schedule window to expire. A missed *monthly* window discovered by the grace timeout is already a day late; a failure reported the moment it happens is a same-morning fix. ## The pattern Six jobs, one instrumentation ladder — climb only as high as the job deserves: 1. **Ping on success** (`&&` + curl) — catches silence and failure. Ten seconds to add. 2. **\+ exit codes** (`/$?`) — separates *"didn't run"* from *"ran and failed"*, which are different pages. 3. **\+ duration** (`/start` + `rid`) — for jobs where slow *is* broken. 4. **\+ output capture** (POST body) — the alert arrives carrying its own diagnosis. A single missed window opens an incident immediately (a daily job that missed once has already lost a day), recovery closes it on the next successful check-in, and every ping's output is on record. Each heartbeat is a regular monitor on every WatchFor plan — including Free — and the ping URLs are Healthchecks-compatible, so existing scripts migrate by swapping the domain. Start with recipe 1 tonight; it's the one whose absence you'll otherwise discover during a restore. --- Canonical page: https://watchfor.io/blog/cron-monitoring-recipes · All articles: https://watchfor.io/blog/all.md · Site guide: https://watchfor.io/llms.txt