Increase latency and error rates for API requests

Incident Report for Buildkite

Resolved

While rolling out a change to test a new version of Rails on a small percentage of traffic, some web services were not scaled sufficiently to meet demand. This resulted in a 15 minute period where services did not have sufficient capacity.

Posted Dec 11, 2024 - 02:05 UTC

Monitoring

We have identified and rectified an issue with scaling of our API web servers and continue to monitor

Posted Dec 11, 2024 - 01:16 UTC

Investigating

We've identified increased latency and error rates on all API requests (REST, GraphQL, Agent API)

Posted Dec 11, 2024 - 01:02 UTC

This incident affected: Agent API and REST API.