Webhook Idempotency for Consumers That Survive Retries and Crashes

Webhook idempotency means a repeated delivery does not repeat the business effect you intended to apply. A receiver needs more than a duplicate check to get there. It has to authenticate the request, record durable work before acknowledging it, and make processing safe to repeat after a crash.

The useful boundary is simple: accept an event durably, then process it. That gives retries somewhere safe to land without making a promise of exactly-once delivery or exactly-once business effects.

Start with durable acceptance

Keep the request handler short. It should read the exact request bytes, verify the sender's signature, identify the delivery, and insert an inbox record in a database transaction. Commit that record before returning a success response. A worker can then apply the business change and mark the inbox item complete.

This is an engineering recommendation, not a universal webhook rule. Providers have different response windows and retry policies. The reason to persist first is that a success response tells the sender it can stop trying. If the process sends that response before saving the event, a crash can leave nothing to retry from.

Consider the failure points:

• Crash before the inbox transaction commits: nothing was durably accepted. Do not return success. The sender may retry, depending on its policy.
• Commit succeeds, then the process crashes before responding: the sender may send the same delivery again. A database uniqueness constraint lets the second request find the existing inbox item rather than create a second business action.
• The handler responds before committing: the sender can stop while the event is still only in memory. A later crash can lose it.
• A worker completes a remote action, then crashes before recording completion: a retry may repeat that action. A local database transaction cannot make an external API call atomic with the database write.

If the application must update its own database and publish a message, the AWS transactional outbox pattern stores the outbox entry in the same database transaction. A publisher sends it later. That avoids relying on two independent writes succeeding together, but duplicate publication is still possible, so consumers need idempotency too.

Verify the bytes, then consider freshness

Signature verification should use the raw request body. Parsing JSON and serializing it again can change the bytes being checked. Follow the provider's format rather than building one universal webhook verifier.

For example, GitHub's validation guide describes an HMAC SHA-256 signature in X-Hub-Signature-256 and recommends a constant-time comparison. Svix's manual verification guide describes a different signed value: the message ID, timestamp, and unchanged raw body joined by periods. Its timestamp check uses a configured tolerance.

A valid signature and a new request are separate questions. A signature can establish that the bytes match a provider's signing scheme. A timestamp check, when the provider's scheme includes one, can help reject old signed requests. Deduplication still needs a stable event identity. Stripe documents its own raw-body verification and timestamp tolerance; it also notes that retries receive a new signature and timestamp. Those details apply to Stripe's webhook scheme, not every sender.

Webhook idempotency needs a stable event identity

Store a provider-documented delivery or event ID, and scope the uniqueness rule to the provider and account or destination that owns that ID. Let the database arbitrate concurrent inserts. An in-memory set is not enough when multiple server instances can handle requests or when the process restarts.

The right identity depends on the effect you want to protect. Stripe recommends tracking event IDs to identify duplicate deliveries. It also notes that separate Event objects can represent the same underlying object and event type. If your application should apply an effect only once for that object and type, that is a separate business rule from suppressing a retransmitted delivery.

GitHub keeps the X-GitHub-Delivery value when a delivery is redelivered, and Svix keeps the message ID on resends. Use the identifier documented by the provider you receive from. A payload hash is not a universal event identity: equal or changed bytes alone do not tell your application whether two messages represent the same intended action.

Also keep inbound deduplication separate from an outbound API idempotency key. Stripe's API idempotency documentation describes keys for retrying Stripe API requests. That protects a request your service sends to Stripe; it does not replace the event ID check for an inbound webhook.

Choose a processing shape

Inline handler

• Boundary: Verify the request and finish the business work before responding.
• Tradeoff: This is simple for short local work, but remote calls and slow processing consume the sender's response window. Stripe recommends returning 2xx before complex work.

Database inbox and worker

• Boundary: Commit an inbox record, respond, then let a worker process it.
• Tradeoff: The request path stays short and accepted work is recoverable. You must operate the worker and define what happens to records that repeatedly fail.

Queue-backed worker

• Boundary: Put the event on a durable queue before acknowledging the webhook.
• Tradeoff: The queue separates receipt from processing, but coordinating a database write and queue publish can create a dual-write problem. Use an outbox when both writes must represent one local transaction. Amazon SQS Standard queues can deliver a message more than once, so queue consumers still need idempotent handling.

Managed gateway

• Boundary: Let a gateway receive and forward deliveries, while your application still validates the event and protects its own effects.
• Tradeoff: Hookdeck's own documentation describes configurable retries, attempt logs, replay, and at-least-once delivery. Those vendor-documented features do not remove the need for deduplication in your consumer.

Make worker retries safe

When the worker's effect is entirely in your database, update the relevant business rows and mark the inbox item complete in one transaction. A crash before commit leaves both changes unapplied; a crash after commit leaves both applied. A uniqueness rule on the effect can protect against two workers racing on the same event.

That transaction cannot cover a remote payment, email, or API call. A timeout only tells your worker it did not receive a timely answer; it does not prove the remote service did nothing. If the remote API supports idempotency keys, use a stable key for the intended operation and keep it consistent across retries. The AWS guide to idempotent APIs explains why retrying side-effecting calls can repeat effects and why representing the caller's intent consistently matters.

Queues have their own delivery behavior. With SQS, an unacknowledged message can become visible again after its visibility timeout, and another delivery can occur even during that timeout, as the SQS visibility timeout documentation explains. A dead-letter queue can isolate messages that repeatedly fail so they can be inspected and redriven. Treat those as queue-specific controls, not a guarantee that processing happens once. For SQS FIFO queues, send deduplication is limited to the documented five-minute interval; it does not make an external business effect exactly once. AWS documents that FIFO send deduplication window.

Retries can also amplify load. Set timeouts deliberately and use bounded backoff with jitter for work your own service retries. The AWS guidance on timeouts, retries, and backoff explains why timeouts do not prove an operation failed and why synchronized retries can increase pressure on a service.

Treat provider retries and ordering as provider-specific

Stripe

• Response: Stripe recommends returning 2xx before complex work.
• Retries: In live mode, automatic retries can continue for up to three days. Stripe also documents sandbox retries.
• Identity and order: Track event IDs for duplicate deliveries, and do not depend on event ordering.

See Stripe's webhook documentation for the current provider-specific details.

GitHub

• Response: GitHub expects a 2xx response within 10 seconds.
• Retries: GitHub does not automatically redeliver failed deliveries. A delivery can be manually redelivered for the prior three days.
• Identity: The X-GitHub-Delivery value stays the same on redelivery.

Those rules are documented in GitHub's webhook best practices and its redelivery guide.

Svix

• Response: Svix treats a 2xx response within 15 seconds as successful delivery.
• Retries: Its documented schedule starts immediately, then retries after 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, and another 10 hours.
• Identity: Resends retain the message ID.

See the Svix retry schedule. This is Svix's schedule, not a general webhook retry contract.

Ordering needs its own handling. Stripe says event order is not guaranteed. If an old event could overwrite newer application state, use a provider-supported version or timestamp guard where available, or fetch current state when the provider offers that option. Stripe's event API, for example, supports retrieving events created in the last 30 days through its event retrieval endpoint. A periodic reconciliation path can help find gaps that delivery retries alone will not repair.

Before shipping, trace one event through four points: signature verification, inbox commit, HTTP response, and worker completion. Then force a repeat at each boundary. You should be able to explain which record proves the event was accepted, which key prevents the same effect from running twice, and how an operator can inspect or safely replay a failed item.