Webhooks are outbound HTTP calls your service makes to customer endpoints when events occur. A payment completes: you POST to https://merchant.example.com/webhooks/payment. Unlike inbound requests where you control the client, webhook delivery fires into unknown infrastructure. Customer servers go down, time out, return 500s. Building reliable webhook delivery means treating outbound HTTP as a distributed messaging problem with at-least-once semantics.

The Core Problem#

Naive webhook delivery: event fires, you make an HTTP call, move on. Three failure modes kill this:

Network timeout: the customer’s server accepted the connection but took 30 seconds to respond. Your call timed out. Did they receive it?

5xx response: their server processed the request but returned a 500. Should you retry?

Your own crash: your service died after the event but before the HTTP call completed.

All three require retry. Retry requires idempotency on the receiving end: if you deliver the same webhook twice, the customer must not double-charge or double-fulfill. Include a stable event_id in every webhook payload. Customers use it to deduplicate.

Retry with Exponential Backoff#

On failure, retry with exponential backoff and jitter: 5s, 30s, 5m, 30m, 2h, 8h, 24h. Seven retries over 24 hours. After 24 hours of failures, move the event to a dead letter queue and alert.

Jitter prevents thundering herd: if your service restarts after a crash with 10,000 undelivered webhooks, sending all immediately would hammer customer endpoints. Add random jitter (±20% of the backoff interval) to spread load.

graph TD A[Event: payment.succeeded] --> B[Write to webhook outbox table: status=pending] B --> C[Delivery worker: attempt HTTP POST] C --> D{Response?} D --> |2xx| E[Mark delivered: status=delivered] D --> |5xx or timeout| F[Increment attempt count, schedule retry] F --> G{Attempts < 7?} G --> |Yes| H[Exponential backoff: 5s, 30s, 5m, 30m, 2h, 8h, 24h] H --> C G --> |No| I[Move to dead letter queue, alert customer] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Outbox Pattern for Crash Safety#

The third failure mode (your service crashes before delivering) requires the transactional outbox pattern. Write the webhook event to an outbox table in the same database transaction as the business event. A separate delivery worker polls the outbox and makes the HTTP calls. If your service crashes after the transaction commits but before delivery, the worker picks up the pending event on restart.

This decouples event generation from delivery. The database is the source of truth for pending webhooks.

Ordering#

Webhooks for the same customer resource can arrive out of order due to retries: payment.updated delivered before payment.created if the first delivery failed and retried later. Customers must be prepared for out-of-order delivery. Include a sequence number or timestamp in the payload so customers can detect and handle ordering issues client-side.

At Oracle#

Oracle Integration Cloud’s webhook delivery system handled outbound notifications to customer systems for ERP events (invoice created, PO approved). Initial implementation was synchronous: event fired, HTTP call made inline. A customer’s webhook endpoint timing out at 28 seconds blocked the ERP transaction. We moved to async delivery via an outbox table with a dedicated delivery pool of 20 threads. Delivery latency dropped from blocking-transaction to median 200ms with zero impact on ERP response times.

What I’m Learning#

Webhook delivery is an at-least-once messaging problem where you don’t control the consumer. Outbox for crash safety, idempotency keys for duplicate safety, exponential backoff for failing endpoints. The dead letter queue is the pressure valve: it prevents infinite retry loops while preserving events for manual redelivery.

Have you implemented webhook delivery at scale, and what was the most common failure mode from customer endpoints?