The dangerous thing about webhooks isn't that they fail — everything fails. It's how they fail: silently, by design. When an API call fails, the caller gets an error and can react. When a webhook delivery fails, the sender logs it somewhere you'll never look and moves on. The receiver doesn't get an error — it gets nothing, which looks exactly like "no events happened."
That asymmetry is why webhook incidents are discovered weeks late, as data drift: orders that exist in Stripe but not your database, tickets your integration never created, a billing state two subscriptions behind reality. (New to webhooks? Start with our Webhooks 101 — this post is the production-hardening sequel.)
Designing the sender
If you send webhooks — to customers or between your own services — four decisions determine whether receivers can trust you:
Retries with backoff, for hours not seconds. Receivers restart, deploy, and have bad minutes. A single delivery attempt is a coin flip; the standard is exponential backoff over an extended window (Stripe retries for days). Treat any non-2xx — and any timeout — as retryable, with one exception: some senders stop retrying on 410 Gone, letting receivers permanently unsubscribe an endpoint.
Idempotency keys, because retries create duplicates. At-least-once delivery means sometimes twice. Every event needs a stable unique ID so the receiver can deduplicate. Never make receivers guess from payload contents.
Signatures, because an open endpoint is an open door. A webhook receiver is an unauthenticated URL that mutates state — sign every payload (HMAC over body + timestamp, like Stripe's t=...,v1=... scheme) so receivers can verify origin and reject replays. Include the timestamp inside the signed material, or an attacker can replay old signed payloads forever.
Don't promise ordering — timestamp instead. Retries guarantee events will arrive out of order eventually. Senders that pretend otherwise create receivers that break subtly. Put the event's effective timestamp and an entity version in the payload; receivers apply the newest state and discard stale arrivals.
This is also how WatchFor's own webhook alert channel behaves as a sender: structured JSON with stable incident IDs and event types, so your automation can dedupe and reorder safely.
Designing the receiver
The receiving side has one golden rule with three corollaries:
Return 200 immediately; process later. The sender's timeout is short and its retry behavior is out of your control. Validate the signature, drop the payload in a queue, respond 2xx — then process. Doing real work inline means slow handlers → sender timeouts → retries → duplicate work → more slowness: the webhook death spiral.
- Dedupe on the event ID before processing (a simple unique constraint does it).
- Tolerate disorder: apply events by their embedded timestamp/version, not arrival order.
- Dead-letter what you can't process — a payload that fails validation goes to a review queue, never into a silent
catch {}.
And for everything you missed while your endpoint was down anyway: reconcile. A nightly job that pulls the sender's API and compares against local state is the safety net that turns "we lost six hours of webhooks" from a data-integrity incident into a non-event.
Monitoring: the part everyone skips
Here's the trap: a dead webhook pipeline produces no errors on either side. The sender retries quietly and gives up; the receiver hums along processing nothing. Every signal looks healthy. You need checks that measure the pipeline itself — and they're different for each direction.
Monitoring your receiver (you consume webhooks)
Your receiver endpoint is a production API that nobody calls visibly — so call it visibly:
- A write-path check: an API monitor that POSTs a realistic (signed, test-flagged) payload to the endpoint and asserts the response — status
2xx, body acknowledging acceptance. This exercises the real path: TLS, routing, auth middleware, signature validation, queue write. It would have caught the classic silent killers: the endpoint returning301after an https migration (many senders don't follow redirects), a new WAF rule eating POSTs, a deploy that broke signature verification. - A freshness check: expose an internal endpoint reporting seconds since the last real event was processed, and alert when it exceeds the expected gap — the direct measurement of "is the pipeline actually flowing?"
- The dependency check: your queue and workers. A heartbeat monitor on the consumer catches the worker that crashed Friday night while the queue quietly grew.
Monitoring your sender (you emit webhooks)
- Deliver to a canary: point a test subscription at an endpoint you monitor, and alert when expected deliveries stop arriving — you'll know your dispatcher died before your customers do.
- Watch the failure rate per destination: a customer endpoint failing for days isn't their problem alone; it's your support ticket in incubation. Surface it to them proactively.
The checklist
| Side | Must-have | Catches |
|---|---|---|
| Sender | Backoff retries over hours | Receiver deploys and blips |
| Sender | Event IDs + signatures + timestamps | Duplicates, forgeries, replays |
| Receiver | Ack fast, process async | Timeout → retry death spiral |
| Receiver | Dedupe + version-based apply | Duplicates and disorder |
| Both | Reconciliation job | Anything missed while down |
| Both | Active checks on the pipeline | The silent-death failure mode |
Webhooks earn their reputation for flakiness one skipped item at a time — but every item is mundane engineering, not research. Design for retries, verify what arrives, and actively check the path in both directions. The teams that do all three stop thinking about webhooks entirely — which is the whole point of the pattern.