--- title: Webhook reliability: designing deliveries that survive, and monitoring both ends description: Webhooks fail silently by design — the sender moves on, the receiver never knows what it missed. Here's the engineering playbook for both sides: retries and idempotency, signatures and ordering, dead letters — and the monitoring that catches a dead webhook pipeline before your data drifts. canonical: https://watchfor.io/blog/webhook-reliability-guide --- # Webhook reliability: designing deliveries that survive, and monitoring both ends 2026-08-23 · WatchFor Team · Monitoring · webhooks, api, integrations, reliability Webhooks fail silently by design — the sender moves on, the receiver never knows what it missed. Here's the engineering playbook for both sides: retries and idempotency, signatures and ordering, dead letters — and the monitoring that catches a dead webhook pipeline before your data drifts. The dangerous thing about webhooks isn't that they fail — everything fails. It's *how* they fail: *silently, by design*. When an API call fails, the caller gets an error and can react. When a webhook delivery fails, the sender logs it somewhere you'll never look and moves on. The receiver doesn't get an error — it gets **nothing**, which looks exactly like "no events happened." That asymmetry is why webhook incidents are discovered weeks late, as data drift: orders that exist in Stripe but not your database, tickets your integration never created, a billing state two subscriptions behind reality. (New to webhooks? Start with our [Webhooks 101](/blog/webhooks-101) — this post is the production-hardening sequel.) ## Designing the sender If you *send* webhooks — to customers or between your own services — four decisions determine whether receivers can trust you: **Retries with backoff, for hours not seconds.** Receivers restart, deploy, and have bad minutes. A single delivery attempt is a coin flip; the standard is exponential backoff over an extended window (Stripe retries for days). Treat any non-2xx — and any timeout — as retryable, with one exception: some senders stop retrying on `410 Gone`, letting receivers permanently unsubscribe an endpoint. **Idempotency keys, because retries create duplicates.** At-least-once delivery means *sometimes twice*. Every event needs a stable unique ID so the receiver can deduplicate. Never make receivers guess from payload contents. **Signatures, because an open endpoint is an open door.** A webhook receiver is an unauthenticated URL that mutates state — sign every payload (HMAC over body + timestamp, like Stripe's `t=...,v1=...` scheme) so receivers can verify origin and reject replays. Include the timestamp *inside* the signed material, or an attacker can replay old signed payloads forever. **Don't promise ordering — timestamp instead.** Retries guarantee events *will* arrive out of order eventually. Senders that pretend otherwise create receivers that break subtly. Put the event's effective timestamp and an entity version in the payload; receivers apply the newest state and discard stale arrivals. This is also how WatchFor's own [webhook alert channel](/alerting-integrations) behaves as a sender: structured JSON with stable incident IDs and event types, so your automation can dedupe and reorder safely. ## Designing the receiver The receiving side has one golden rule with three corollaries: **Return 200 immediately; process later.** The sender's timeout is short and its retry behavior is out of your control. Validate the signature, drop the payload in a queue, respond 2xx — *then* process. Doing real work inline means slow handlers → sender timeouts → retries → duplicate work → more slowness: the webhook death spiral. - **Dedupe on the event ID** before processing (a simple unique constraint does it). - **Tolerate disorder:** apply events by their embedded timestamp/version, not arrival order. - **Dead-letter what you can't process** — a payload that fails validation goes to a review queue, never into a silent `catch {}`. And for everything you missed while your endpoint was down anyway: **reconcile**. A nightly job that pulls the sender's API and compares against local state is the safety net that turns "we lost six hours of webhooks" from a data-integrity incident into a non-event. ## Monitoring: the part everyone skips Here's the trap: a dead webhook pipeline produces **no errors on either side**. The sender retries quietly and gives up; the receiver hums along processing nothing. Every signal looks healthy. You need checks that measure the pipeline itself — and they're different for each direction. ### Monitoring your receiver (you consume webhooks) Your receiver endpoint is a production API that nobody calls *visibly* — so call it visibly: - **A write-path check**: an [API monitor](/api-monitoring) that POSTs a realistic (signed, test-flagged) payload to the endpoint and asserts the response — status `2xx`, body acknowledging acceptance. This exercises the real path: TLS, routing, auth middleware, signature validation, queue write. It would have caught the classic silent killers: the endpoint returning `301` after an https migration (many senders don't follow redirects), a new WAF rule eating POSTs, a deploy that broke signature verification. - **A freshness check**: expose an internal endpoint reporting seconds since the last *real* event was processed, and alert when it exceeds the expected gap — the direct measurement of "is the pipeline actually flowing?" - **The dependency check**: your queue and workers. A [heartbeat monitor](/heartbeat-monitoring) on the consumer catches the worker that crashed Friday night while the queue quietly grew. ### Monitoring your sender (you emit webhooks) - **Deliver to a canary**: point a test subscription at an endpoint you monitor, and alert when expected deliveries stop arriving — you'll know your dispatcher died before your customers do. - **Watch the failure rate per destination**: a customer endpoint failing for days isn't their problem alone; it's your support ticket in incubation. Surface it to them proactively. ## The checklist | Side | Must-have | Catches | | --- | --- | --- | | Sender | Backoff retries over hours | Receiver deploys and blips | | Sender | Event IDs + signatures + timestamps | Duplicates, forgeries, replays | | Receiver | Ack fast, process async | Timeout → retry death spiral | | Receiver | Dedupe + version-based apply | Duplicates and disorder | | Both | Reconciliation job | Anything missed while down | | Both | Active checks on the pipeline | The silent-death failure mode | Webhooks earn their reputation for flakiness one skipped item at a time — but every item is mundane engineering, not research. Design for retries, verify what arrives, and *actively check* the path in both directions. The teams that do all three stop thinking about webhooks entirely — which is the whole point of the pattern. --- Canonical page: https://watchfor.io/blog/webhook-reliability-guide · All articles: https://watchfor.io/blog/all.md · Site guide: https://watchfor.io/llms.txt