Webhook Relay

Situation

External systems commonly retry events, deliver them late or out of order, and time out before knowing whether processing succeeded. When application logic runs directly inside the inbound request, a provider retry can repeat a side effect while failures remain hard to inspect.

This reference models an ingestion boundary between providers and downstream consumers. It does not document a deployed relay or verified integration workload.

Intended users

• Application operators investigating intake, retry, and dead-letter state.
• Integration engineers defining source verification and consumer contracts.
• Background processing systems receiving persisted events after intake.

These are intended roles and system actors. No named provider, customer, or active consumer is claimed.

Constraints

• Provider delivery can be duplicate, delayed, malformed, out of order, or absent.
• Signature rules and secret rotation differ by source.
• An HTTP timeout does not prove that the receiver did no work.
• One poison message should not block unrelated events indefinitely.
• Retries need a budget and terminal state rather than an infinite loop.
• Dead-letter replay can repeat side effects and therefore needs an operator decision.
• Duplicate detection has a retention window; it cannot provide exactly-once behavior across every boundary.
• Reconciliation depends on provider APIs and can also fail or become stale.

The prior 72-hour idempotency window is retained only as an illustrative design parameter. No event store or retention job in this repository verifies it.

What the reference models

Modeled intake boundary. Verify source and signature, derive an idempotency key, persist the event, acknowledge intake, and process asynchronously.

Illustrative event schema. The narrative uses a unique idempotency key, source, payload, timestamps, and attempt count to show what inspectable intake state could contain. It is not a migration from a running database.

Modeled retry behavior. Consumer failures move through bounded backoff and a terminal dead-letter state. The design keeps manual replay visible rather than automatically cycling a failed message.

Modeled reconciliation. Provider state and consumer heartbeats are considered secondary evidence for missed intake or silent processing failure.

Deferred capability. No public endpoint, signature library configuration, queue, database, Redis worker, provider integration, alert, reconciliation job, or executable replay interface is linked.

Critical failure modes

• WEBHOOK_RETRY_DUPLICATION — the same provider event is accepted more than once and repeats a downstream side effect.
• SIGNATURE_BYPASS — an unknown or invalid source reaches processing without a verified trust decision.
• MALFORMED_PAYLOAD — a payload passes transport checks but cannot satisfy the expected event contract.
• OUT_OF_ORDER_EVENT — a later state arrives before the event it depends on.
• DOWNSTREAM_TIMEOUT — the consumer may have completed work even though the relay did not receive confirmation.
• RETRY_EXHAUSTION — bounded attempts end without successful processing.
• POISON_MESSAGE_BLOCKING — one deterministic failure prevents unrelated messages from moving.
• IDEMPOTENCY_COLLISION — two distinct events resolve to the same deduplication key.
• SILENT_CONSUMER_FAILURE — intake continues while a downstream consumer stops confirming work.

These failure modes shape the model. No runtime artifact proves prevention, throughput, delivery, or recovery behavior.

Decisions and trade-offs

Persist before processing over work inside the request

• Decision: Model the inbound request as verified persistence and acknowledgement, with business processing after intake.
• Reason: Provider timeouts and retries should not directly multiply long-running side effects.
• Trade-off: Once acknowledged, the relay owns a durable responsibility to process or expose failure.
• Does not solve: Database failure before persistence, incorrect acknowledgement, or downstream idempotency.

Bounded retries over infinite retry loops

• Decision: Give each source or event class an explicit retry budget and terminal state.
• Reason: Persistent faults should become visible instead of consuming the queue forever.
• Trade-off: Some recoverable events wait for human review after the budget ends.
• Does not solve: Choosing the correct budget, external outage duration, or the underlying consumer defect.

Manual dead-letter replay over automatic replay

• Decision: Require inspection and confirmation before replaying dead-lettered events.
• Reason: A deterministic failure or non-idempotent side effect should not be repeated silently.
• Trade-off: Recovery time depends on operator availability and the quality of the visible event state.
• Does not solve: Invalid semantics, unavailable downstream systems, or unsafe consumer behavior.

Reconciliation over trust in webhook delivery

• Decision: Treat provider queries and consumer health as a later comparison against event state.
• Reason: An event that never arrives cannot be detected by the intake path alone.
• Trade-off: Polling consumes quota and introduces another integration and freshness boundary.
• Does not solve: Providers without a usable listing API or incorrect reconciliation logic.

Verification performed

The public repository verifies:

• This MDX narrative and Reference build frontmatter exist.
• The static Work route can render the narrative, status qualification, evidence table, metadata, related services, and CTA.
• Repository tests can validate content contracts, canonical metadata, sitemap inclusion, and evidence-safe wording.
• The production website build can verify the public evidence page compiles.

The repository does not contain an executable relay, signature test, queue worker, persistence migration, retry test, public endpoint, provider fixture, load test, production telemetry, customer acceptance record, penetration test, or independent audit.

Outcome

The reference demonstrates how unreliable external events can move through explicit intake, duplicate detection, retry, dead-letter, replay, and reconciliation states. It provides a pattern for discussing a future integration or stabilisation engagement; it does not establish delivery performance or production reliability.

Current status

What is deliberately not handled

• Guaranteed delivery by an external provider.
• Exactly-once processing across network and consumer boundaries.
• Infinite event or idempotency-key retention.
• Unlimited throughput or a published performance target.
• Automatic semantic correction of invalid events.
• Elimination of downstream outages.
• A managed 24/7 integration service or emergency response promise.
• Verified queue, database, alerting, or replay implementation.