Design for at-least-once delivery
A webhook is often treated as one request producing one action, but distributed systems cannot reliably promise exactly-once delivery. The sender may deliver an event successfully and lose the response because the connection drops. The receiver may commit its database transaction and then time out before returning a success status. From the sender's perspective, both cases are uncertain, so retrying is the only safe option. The receiver must therefore assume that the same event can arrive more than once.
A robust receiver should not keep the connection open while completing every business operation. Validate the signature and basic schema, persist the original event with its identifier, source, and receipt time, then return success promptly. ERP updates, CRM writes, LINE notifications, or AI processing can continue in background workers. This reduces sender timeouts, but it also gives the response a precise meaning: HTTP success confirms durable acceptance, not completion of every downstream action.
- Accepted:Return 2xx only after the event is stored durably or placed on a reliable queue.
- Temporary failure:Return 5xx when required infrastructure is unavailable and a later attempt may succeed.
- Invalid request:Return an appropriate 4xx for a bad signature, malformed payload, or missing required fields.
- Slow processing:Acknowledge safely first and move long-running work out of the request path.
Retry temporary failures, not every failure
Retries should be a bounded policy, not an infinite loop at a fixed interval. Network timeouts, connection resets, rate limits, and temporary service outages are usually retryable. An invalid signature, missing field, revoked credential, or unknown resource generally requires a data or configuration change. Retrying those permanent failures creates noise, increases load, and can hide a problem that needs an operator.
For retryable failures, use exponential backoff with jitter. An early retry can recover quickly from a brief interruption, while increasingly longer delays give an unhealthy dependency time to recover. Jitter prevents a large group of failed deliveries from returning at exactly the same moment. Define both a maximum attempt count and a maximum retry window. When either limit is reached, move the event to a dead-letter queue or failed-event store instead of silently discarding it. Honor Retry-After when the receiver supplies a reasonable value.
Record the timestamp, response status, latency, and a sanitized error summary for every attempt. Do not put tokens, signing secrets, or unrestricted payloads into logs. Manual replay must use the original event identifier and the same idempotency path as automatic retry, with an audit record of who initiated it and why. A replay button that bypasses these controls can create another order, send another notification, or apply the same inventory change twice.
Protect the business effect with idempotency
The best idempotency key is a stable, globally unique event identifier generated by the source. If the source does not provide one, construct a key from the source system, event type, object identifier, and object version. Hashing the payload is a fallback, but only after canonicalizing field order and formatting and excluding values such as signatures or delivery timestamps that change on every attempt. A timestamp or customer identifier alone is rarely sufficient: multiple legitimate events can share either value.
Create a database uniqueness constraint on the source and event identifier. Insert the event record and create its pending work in the same transaction. Avoid a simple check-then-insert sequence; two concurrent requests can both observe that no record exists and then execute the effect twice. A unique index, conditional write, or atomic upsert closes that race. Track explicit states such as received, processing, succeeded, and failed so a duplicate request can return the established result or safely defer to the existing worker.
The harder problem is an external side effect. A local transaction cannot roll back an email that was sent or an API call already accepted by another system. Give each downstream operation its own stable idempotency key. An inbox pattern prevents duplicate consumption, while an outbox pattern reliably publishes follow-up work after business data commits. If a third party offers no idempotency support, persist the intended operation and its result locally, control attempts through a state machine, and reconcile an ambiguous timeout by querying remote state before issuing the command again.
Plan for ordering, replay, and operations
Idempotency prevents one event from being applied twice; it does not fix reordered events. If an older order-updated event arrives after order-cancelled, blindly processing receipt order may move the record backward. For entities where ordering matters, include a source version, monotonic sequence, or event time and compare it with the current state before updating. When global ordering is impractical, serialize work per business entity or design events with enough state to reconstruct the latest valid result.
Operational tooling should support searches by event identifier, source object, and time range. Engineers need to see the current state, attempt count, last error, and next retry time. Alerts should focus on growing failure backlogs, processing delay, dead-letter volume, and unusual response-code patterns rather than paging on every isolated failure. Replay tools need access control, audit history, batch limits, and a preview mode so recovery does not create a new traffic spike downstream.
Finally, test uncertainty deliberately. Drop the response after committing the event, submit the same event concurrently, make a downstream call time out after it actually succeeds, and reverse the event order. Then verify that only one business result exists and that operators can recover the flow without editing production data by hand. Webhook reliability is not a single retry setting; it is the combined behavior of durable receipt, idempotent processing, controlled side effects, and practical recovery tools.