In a webhook delivery system, each customer endpoint is an external dependency you don’t control. Some endpoints are fast and reliable. Others time out consistently, return 500s intermittently, or go down for hours during deployments. Without endpoint health tracking, your delivery workers waste time retrying dead endpoints while healthy endpoints get delayed because the retry queue fills up with failed attempts to unavailable ones.

Per-Endpoint Health Scoring#

Track delivery outcomes per endpoint: successful deliveries, failures, consecutive failures, last success time. Compute a health score. An endpoint that succeeded 99% of the time over the last 100 attempts is healthy. One that failed 10 consecutive times is degraded.

Use this score to adjust retry behavior: healthy endpoints retry on normal schedule. Degraded endpoints back off more aggressively — no point hammering a server that’s been down for 3 hours every 5 seconds.

Circuit Breaking Per Endpoint#

Circuit breaking applied to webhook endpoints: after N consecutive failures, open the circuit for that endpoint. While open, don’t attempt delivery. After a cooldown period (e.g., 15 minutes), send one probe request. If it succeeds, close the circuit and resume normal delivery. If it fails, extend the cooldown.

This matters for worker capacity. Without circuit breaking, 50 endpoints that are all down generate 50 concurrent retry attempts consuming workers. With circuit breaking, those 50 endpoints get probed once every 15 minutes while workers stay available for responsive endpoints.

graph TD A[Delivery attempt] --> B{Circuit state for endpoint?} B --> |Closed: healthy| C[Send HTTP POST] C --> D{Response?} D --> |2xx| E[Record success, keep circuit closed] D --> |Failure| F[Increment consecutive failures] F --> G{Failures >= threshold?} G --> |Yes| H[Open circuit, start 15-min cooldown] G --> |No| I[Schedule retry with backoff] B --> |Open: endpoint down| J[Skip: queue event, wait for probe] H --> K[After cooldown: send probe] K --> L{Probe success?} L --> |Yes| M[Close circuit, resume delivery] L --> |No| N[Extend cooldown: 30 min, 1hr...] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style J fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style K fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style L fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style M fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style N fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Dead Letter Queue and Customer Visibility#

Events that exhaust all retries (after 24 hours per delivery guarantees) move to a dead letter queue. Customers must be able to see and replay these. Provide a dashboard: “these 47 events failed to deliver between 14:00 and 16:00 on Aug 4.” Let them trigger redelivery with one click once their endpoint is restored.

Without customer visibility into DLQ events, the customer’s only signal is missing data. They file a support ticket 3 hours later. You spend time diagnosing something they could have self-served.

Auto-Disabling Chronic Failures#

An endpoint that’s been in open circuit state for 7 days straight is probably abandoned or permanently broken. Auto-disable it and email the customer. This prevents the endpoint from consuming health-tracking storage and worker capacity indefinitely.

Set a re-enable flow: customer fixes their endpoint, clicks “re-enable,” you send a test event, if it succeeds you mark the endpoint active again.

At Oracle#

Oracle’s notification delivery system for integration platform events had 3,200 registered customer webhook endpoints. Without circuit breaking, a traffic spike that caused 200 endpoints to fail simultaneously consumed all 50 delivery workers in retry loops. Healthy endpoints stopped receiving events for 40 minutes. After adding per-endpoint circuit breaking with a 10-failure threshold, the same failure scenario isolated to the affected 200 endpoints. Healthy endpoint delivery latency stayed under 500ms throughout.

What I’m Learning#

Per-endpoint circuit breaking is the key isolation mechanism for webhook delivery. Without it, failures at one customer endpoint degrade delivery for all customers. The circuit breaker turns a shared failure into an isolated one. Customer visibility into DLQ events shifts debugging from support tickets to self-service.

How do you expose webhook delivery failures to customers in your system, and how long before they notice on their own?