High error rates and timeouts in the Agent and Rest APIs
Incident Report for Buildkite
Resolved
This incident has been resolved.
Posted Jul 13, 2024 - 22:52 UTC
Update
We are continuing to monitor for any further issues.
Posted Jul 13, 2024 - 22:42 UTC
Update
We are continuing to monitor for any further issues.
Posted Jul 13, 2024 - 22:25 UTC
Monitoring
We have identified the source of high load and put service protections in place to reduce the impact. The Agent and Rest APIs are now performing nominally. We continue to monitor service performance.
Posted Jul 13, 2024 - 22:10 UTC
Update
We are experiencing degraded database performance which is causing high latency and timeouts on the Agent and Rest APIs. We continue to investigate the root cause. We will provide another update in 30 minutes.
Posted Jul 13, 2024 - 21:36 UTC
Update
We are experiencing a high concurrence of Builds which is causing increased latency and timeouts on the Agent and Rest APIs. We continue to investigate the root cause. We will provide another update in 15 minutes.
Posted Jul 13, 2024 - 21:17 UTC
Investigating
We've spotted an issue with our Agent and Rest APIs. We're currently investigating the issue, and will provide an update soon.
Posted Jul 13, 2024 - 20:39 UTC
This incident affected: Agent API and REST API.