Webhook Relay

The Problem

Webhooks are the load-bearing integration pattern nobody budgets for. The contract from most providers is at-least-once delivery with retries on timeout — which means duplicates are not an edge case, they are the specification. Add unverified payloads, one malformed message jamming a queue, and consumers that die without a sound, and "we'll just handle the webhook" becomes a recurring production incident.

The four failure modes this relay handles:

• WEBHOOK_RETRY_DUPLICATION — the provider retries, the handler runs twice, the customer is charged twice.
• SIGNATURE_BYPASS — the endpoint processes any well-formed POST, so anyone who finds the URL can inject events.
• POISON_MESSAGE_BLOCKING — one unprocessable message is retried forever at the head of the queue while everything behind it waits.
• SILENT_CONSUMER_FAILURE — a consumer stops processing and nothing notices until a customer does.

The full taxonomy is in webhook failure modes.

The System

The relay sits between providers and consumers: ingest, verify, persist, acknowledge, then process asynchronously.

Verification fails closed. Every inbound request must carry a valid HMAC signature for a known source. Missing header: rejected. Unknown source: rejected. Invalid signature: rejected with an audit entry, because a burst of signature failures is either a rotated secret or an attack, and both deserve attention. Comparison is timing-safe. There is no debug flag that skips verification — the bypass that exists for staging is the bypass that ships to production.

Dedup at ingestion. Verified events are persisted with an idempotency key — the provider's event ID where one exists, a hash of source plus payload where it does not:

create table inbound_events (
  idempotency_key text primary key,
  source          text        not null,
  payload         jsonb       not null,
  received_at     timestamptz not null default now(),
  processed_at    timestamptz,
  attempts        int         not null default 0
);

-- insert ... on conflict (idempotency_key) do nothing
-- keys swept after 72 hours

A duplicate insert conflicts and is acknowledged without processing. The provider gets its 200 either way — fast, before any business logic runs.

Retries. Processing happens off a Redis queue. Failures retry on exponential backoff with jitter, under a per-source retry budget. Backoff without jitter synchronizes retries into waves; a shared global budget lets one broken source starve the others. Both are cheap to get right at the start and expensive to retrofit.

Dead-letter queue. A message that exhausts its retry budget — or fails deterministically on first attempt — moves to the DLQ instead of returning to the head of the queue. The queue keeps flowing; the poison message waits for a human.

Reconciliation. A scheduled job polls provider APIs and diffs their state against processed events. Missed webhooks — provider outage, relay downtime, delivery that never happened — surface as reconciliation gaps instead of silent divergence. Consumers also emit heartbeats; a consumer that stops confirming work trips an alert rather than failing quietly.

Design Decisions

Fail closed on signatures, not log-and-continue. Accepting unverified events "temporarily" makes the endpoint a public write API to your internal systems. Tradeoff: a provider that rotates secrets without notice causes dropped events until the secret is updated — which is one of the gaps the reconciliation job exists to catch.

Persist-then-process, not process-in-handler. Doing work inside the request handler couples processing time to the provider's timeout, and a timeout triggers a retry, which is how duplicates multiply. Acknowledging after a single insert keeps the handler fast and the retry path boring. Tradeoff: an acknowledged event is now the relay's responsibility. The queue and the DLQ exist because the 200 was already returned.

Manual DLQ replay, not automatic. A message lands in the DLQ because something about it, or the consumer, is broken. Automatically replaying it re-runs the failure — a poison loop with extra steps. An operator inspects the message, fixes the cause, and replays the batch deliberately. Tradeoff: dead-lettered events wait on human response time, so the DLQ alerts on first entry, not on depth.

Reconciliation polling, not trust in delivery. Webhooks are a latency optimization over polling, not a replacement for it. The poll is the source of truth arriving late; the webhook is the same truth arriving early. Tradeoff: polling costs API quota and the diff logic is real work. It is also the only mechanism in the system that catches a webhook that never arrived.