Checkout failures during payment authorisation

Summary

Between 09:12 and 10:07 UTC, about one in three checkout attempts failed. A configuration change at our payment provider made authorisation requests time out, and webhook delivery queued behind the retries for part of that time.

Impact

• 55 minutes of failed or slow payment authorisations.
• About 34% of checkout attempts returned an error. Most customers who retried got through.
• Webhook delivery was delayed by up to 20 minutes. No webhook was lost.

Timeline (UTC)

• 09:12 Checkout error rates start rising.
• 09:16 The on-call engineer is paged and starts investigating.
• 09:31 The provider confirms a configuration change on their side.
• 09:46 The change is rolled back and error rates start falling.
• 10:07 Checkout has been stable for 30 minutes. Delayed webhooks are replayed.

Root cause

The provider shortened its authorisation timeout. Our client kept waiting for the old, longer timeout, so requests piled up and failed together.

What we are changing

• Match our client timeout to the provider's, and fail fast with a clear retry message.
• Alert on authorisation latency, not only on error rate.
• Agree advance notice of configuration changes with the provider.