Sending a webhook is easy. One fetch() call, a 200 response, done. What most tutorials skip is the moment your customer's endpoint goes down at 2 a.m., your retry loop hammers it 50 times in 10 seconds, and you've now poisoned their recovery queue while silently dropping half your events. That is the real webhook problem.
This article is about building a webhook delivery system that treats delivery as a first-class contract — not a best-effort side effect.
Why Webhook Reliability Is Harder Than It Looks
Webhooks invert the normal request cycle. Your server is the caller; your customer's server is the callee. You have zero control over their infrastructure. Their server can be:
- Temporarily overloaded (transient 503)
- Behind a misconfigured firewall
- Running a deploy that causes a 30-second gap in availability
- Intentionally rate-limiting inbound requests
If you fire-and-forget, you will drop events. If you retry naively, you risk duplicate processing on their end. The right answer is a pipeline with three properties: persistence, back-off, and idempotency.
The Core Architecture
A production webhook pipeline has four components:
- Event store — Persists the event before anything else happens
- Delivery queue — Decouples event creation from HTTP dispatch
- Worker process — Attempts delivery with retry logic
- Dead-letter queue (DLQ) — Catches events that exhaust all retries
The key insight is that the event must be written to durable storage before you acknowledge the action that triggered it. If a payment is confirmed and your app crashes before the webhook is queued, that event is gone forever.
[Triggering Action]
│
▼
[Write event to DB] ──→ [Enqueue job]
│
[Worker picks up job]
│
┌──────────┴──────────┐
▼ ▼
[HTTP POST succeeds] [HTTP POST fails]
│ │
[Mark delivered] [Schedule retry with back-off]
│
[Max retries exceeded?]
│
[Move to DLQ]
Step 1: Persist Before You Enqueue
Every outbound webhook event should have a row in a webhook_events table with at minimum:
id(UUID)endpoint_id(which customer endpoint to call)payload(JSON)status(pending / delivered / failed / dead)attempt_countnext_attempt_atcreated_at
Writing to this table and enqueuing the job should happen in the same database transaction. If the enqueue fails, the row remains and a background sweep can re-queue it. This is your safety net.
Step 2: Exponential Back-off With Jitter
Naive retries (try every 60 seconds) will cause thundering herd problems if multiple customers' endpoints recover simultaneously. Use exponential back-off with jitter:
function nextRetryDelay(attempt) {
const base = 30; // seconds
const cap = 3600; // max 1 hour
const exponential = Math.min(cap, base * Math.pow(2, attempt));
const jitter = Math.random() * 0.3 * exponential;
return Math.floor(exponential + jitter);
}
// attempt 0 → ~30s, attempt 1 → ~60s, attempt 2 → ~120s ... capped at 1h
Jitter spreads load across time so your worker pool doesn't spike when a downstream service comes back online after an outage. Cap retries at 5–7 attempts over roughly 24 hours — beyond that, the customer almost certainly needs a human alert, not another automated knock on the door.
Step 3: Idempotency Keys Are Non-Negotiable
Even with careful retry logic, you will deliver the same event twice. Networks time out after the remote server has already processed the request. Your customer's endpoint must be able to handle duplicates safely — but you should also make it easy for them.
Include an X-Webhook-ID header on every request using the event's UUID. Advise customers to store processed IDs and skip duplicates. This is the same pattern Stripe, GitHub, and Svix use in production.
On your end, never re-generate a payload or a timestamp on retry. The payload delivered on attempt 3 must be byte-for-byte identical to attempt 1. Anything else breaks idempotency guarantees.
Step 4: The Dead-Letter Queue
Events that exhaust all retries should not disappear. Move them to a DLQ — this can be as simple as a status = 'dead' flag and a dead_lettered_at timestamp in your existing table.
From the DLQ you should support:
- Manual replay — Let the customer (or your support team) trigger a re-delivery once they've fixed their endpoint
- Bulk replay — Re-queue all dead events for a given endpoint after a customer confirms readiness
- Alerting — Notify the customer when events start dying; proactive communication is better than a support ticket
A simple admin endpoint that moves a dead event back to pending and re-enqueues it is all you need to start.
Tooling for a Solo SaaS Team
You do not need Kafka to build this reliably. Practical choices:
- Queue: BullMQ (Redis-backed, excellent retry/backoff support), or pg-boss if you're already on PostgreSQL and want to avoid an extra dependency
- Workers: A Node.js or Python process running on the same infrastructure as your API
- Storage: Your existing relational DB — no new services required
- Monitoring: Log every attempt with outcome and latency; alert on DLQ growth
If you'd rather not build this from scratch, Svix and Hookdeck are managed webhook delivery platforms that handle the entire pipeline. They're worth considering once webhook volume becomes operationally significant.
Signing Payloads
One detail that often gets bolted on late: sign every webhook payload with an HMAC-SHA256 signature using a per-endpoint secret. Include it as an X-Webhook-Signature header. This lets customers verify the payload came from you and wasn't tampered with in transit. It's a one-line addition and a table-stakes feature for any SaaS shipping webhooks to developers.
Why This Matters for Your Project
If your SaaS product emits events that other systems depend on — order completions, payment confirmations, user lifecycle triggers — a dropped webhook is a broken integration, and a broken integration is a churned customer. Building delivery guarantees directly into your backend from day one is cheaper than retrofitting reliability after your first major incident. The architecture described here scales from dozens to millions of events per day without fundamental redesign, which means the decisions you make early compound in your favour as you grow.
