A webhook demo takes ten minutes: point the sender at an endpoint, print the payload, done. A webhook in production is a distributed system with an unreliable network in the middle and no shared transaction — and it fails in nine specific, recurring ways. These are the nine, each paired with the pattern that survives it. None of the patterns are exotic. All of them are routinely skipped.
The delivery failures
1. Duplicate delivery
DUPLICATE_DELIVERY
The sender times out waiting for your 200 — the handler was slow, the network hiccuped — and retries. Your system receives the same event twice, charges the customer twice, sends the email twice, decrements inventory twice. At-least-once delivery is the contract of every serious webhook sender. Exactly-once is a lie that nobody's docs actually promise.
Surviving pattern: idempotency keys. Every sane provider ships an event id. Store it in a table with a unique constraint and process only on first insert. The database constraint is the lock — do not build the check in application memory, where two workers race and both win.
2. Out-of-order delivery
OUT_OF_ORDER_EVENTS
An updated event arrives before the created event it depends on, because retries reshuffled the timeline. Consumers that apply events in arrival order end up holding state that never existed at the sender — a cancelled subscription flipped back to active by a stale update landing last.
Surviving pattern: order by version, not arrival. Apply an event only if its sequence number or updated-at timestamp beats what you already stored. Better still, treat the event as a doorbell: fetch current state from the sender's API and store that. The event tells you something changed. The API tells you what is true.
3. Missed delivery during downtime
MISSED_DELIVERY
Your endpoint was down for a deploy, or DNS misbehaved, and the sender's retry window — often only a few hours — expired. The event is gone. Nothing errors. Your database is simply missing a customer, permanently, and you find out from an angry email.
Surviving pattern: reconciliation polling. A scheduled sweep — hourly or nightly — lists recent objects from the sender's API and diffs them against local state. Webhooks are the fast path; reconciliation is the guarantee. If the provider offers no list API, the integration has no guarantee. Price it accordingly.
The security failures
4. Signature verification skipped
UNVERIFIED_SENDER
The endpoint accepts any POST that parses. Anyone who finds the URL — server logs, browser history, a public GitHub commit — can inject fake payment-succeeded events, and your system marks invoices paid. Endpoint URLs are not secrets. They leak by design.
Surviving pattern: HMAC verification, fail closed. Verify the provider's signature against the raw request body — before parsing, with a constant-time compare — and return 401 on any failure. Raw body matters: verify-after-parse breaks on the first whitespace difference, and that breakage is how verification ends up commented out.
5. Replay attacks
REPLAY_ATTACK
A valid signed request is captured and re-sent later. The signature still verifies — it was real once. Without a freshness check, an attacker replays a legitimate refund event until something downstream pays out.
Surviving pattern: timestamp tolerance. Serious providers sign a timestamp along with the payload. Reject anything older than a small window — Stripe's default tolerance is five minutes. Combined with the dedupe table from failure one, a replayed id is discarded anyway. Two layers. Keep both.
The load failures
6. The thundering retry herd
RETRY_STORM
Your service goes down for twenty minutes. Every failed delivery enters the sender's retry schedule, and recovery greets you with the entire backlog arriving at once — on top of live traffic. The recovering service falls over again. Retries reschedule. The outage extends itself.
Surviving pattern: ack fast, work async. The HTTP handler does three things: verify the signature, persist the raw event, return 200. Milliseconds. Processing happens off a queue, at your pace, under your concurrency limits. The sender's retry pressure lands on the cheap part of the system.
7. Poison messages
POISON_MESSAGE
One malformed event crashes its processor. The queue redelivers it. It crashes again — forever. Everything behind it waits, or every retry burns a worker. One bad payload becomes a full pipeline stall.
Surviving pattern: max attempts, then dead-letter queue. After three to five failures, the event moves to a DLQ with its error attached, and the pipeline flows on. The DLQ alerts on depth and supports replay after the fix ships. A DLQ nobody can replay from is not a queue. It is a landfill.
The quiet failures
8. Payload schema drift
SCHEMA_DRIFT
The provider adds a field, renames one in an API-version bump, or a partner "improves" their payload without a changelog. Your consumer half-works: it parses what it recognizes and silently drops what it does not. Data goes subtly wrong for weeks before anyone connects the symptom to the cause.
Surviving pattern: validate against a pinned schema, fail closed on surprise. Unknown event types and shape mismatches route to the DLQ and page a human — they do not get best-effort processed. Pin the provider's API version and upgrade deliberately, not accidentally.
9. The silent 200
SILENT_200
A catch-all error handler swallows the processing exception, and the endpoint returns 200 anyway. The sender marks the event delivered and never sends it again. No retry, no alert, no log line anyone reads. This is the worst failure on the list, because every safety mechanism upstream has been told that everything is fine.
Surviving pattern: only ack persistence, never processing. Under the ack-fast architecture, the 200 means exactly one thing: "event stored". Processing failures happen queue-side, where retries, the DLQ, and alerting own them. If you must process synchronously, a processing error must return 5xx so the sender retries. A 200 is a receipt. Do not sign for packages you dropped.
The consumer that survives
The nine patterns compress into one small shape:
// Receive: verify, persist, ack. Nothing else.
export async function POST(req: Request) {
const raw = await req.text()
const sig = req.headers.get('x-signature')
if (!verifyHmac(raw, sig, { toleranceSeconds: 300 })) {
return new Response('invalid signature', { status: 401 }) // fail closed
}
const event = parseEvent(raw) // pinned schema; throws on drift
try {
await db.webhookEvents.insert({ id: event.id, raw, status: 'received' })
} catch (err) {
if (isUniqueViolation(err)) return ok() // duplicate: ack, do nothing
throw err
}
await queue.enqueue(event.id) // async processing; 3 attempts, then DLQ
return ok() // 200 means "stored" — never "processed"
}
Plus two things that live outside the handler: a reconciliation job that diffs provider state on a schedule, and a DLQ depth alert that a human actually receives.
Constraint
These patterns assume the sender retries with backoff and signs its payloads — true of Stripe, GitHub, Shopify, Twilio, and most serious platforms. A partner-built sender that fires once, unsigned, hands you failure modes three and four with no upstream help. In that case reconciliation polling stops being the safety net and becomes the primary channel.
Norseson's webhook relay build packages this exact shape — verification, dedupe, queue, DLQ, replay, reconciliation — as reusable infrastructure, because every integration eventually needs all nine answers. If your current integrations answer none of them, they are working by coincidence. API and automation builds replace the coincidence with a contract.