Webhook delivery stopped

Summary

Between 03:10 and 04:45 UTC, no webhooks were delivered. After a traffic spike, the delivery workers ran out of database connections and stopped picking up new events. Every event was stored while this happened, and all of them were delivered once the workers came back.

Impact

• 95 minutes with no webhook deliveries.
• About 18,400 events waited in the queue. None were lost or delivered twice.
• The API, the dashboard and email notifications were not affected.

Timeline (UTC)

• 03:10 Webhook deliveries stop.
• 03:15 An alert on queue length pages the on-call engineer.
• 03:48 Cause found: every worker is waiting for a database connection.
• 04:20 Workers restart with a higher connection limit and delivery resumes.
• 04:45 The backlog is cleared.

Root cause

Each delivery worker kept its database connection while it waited on slow customer endpoints. When several endpoints slowed down at once, the pool ran dry and no worker could take new work.

What we are changing

• Workers now release their connection before calling a customer endpoint.
• We alert when the connection pool is more than 80% used, not only when the queue grows.
• A load test with slow endpoints runs before each release of the delivery service.