Slow dashboard page loads

Summary

From 08:30 to 11:00 UTC, some dashboard pages took up to 10 seconds to load. A scheduled reporting job was running heavy queries on the same database the dashboard uses. The dashboard stayed available the whole time.

Impact

• 150 minutes of slow page loads, mostly on the reports and billing pages.
• The slowest 5% of page loads took more than 8 seconds.
• No errors, and no data was affected.

Timeline (UTC)

• 08:30 Page load times start climbing.
• 08:38 Monitoring flags the slowdown and we start investigating.
• 09:22 The reporting job is identified as the cause.
• 10:15 The job is moved to a read-only replica.
• 11:00 Load times have been normal for 45 minutes.

Root cause

The reporting job had grown with our data. Its queries now ran for over two hours and competed with the dashboard for the same database.

What we are changing

• Reporting jobs only run against read-only replicas.
• Long-running queries on the primary database are stopped after 60 seconds.
• We track dashboard load time as a service objective, with an alert.