Monitoring - We're seeing signs of recovery and will continue to monitor.
Feb 26, 2026 - 19:36 UTC
Identified - Some customers are experiencing increased latency for jobs being assigned to agents. We have identified the cause and are working on mitigations.
Feb 26, 2026 - 19:10 UTC
Resolved -
Response times have returned to normal. This incident is now resolved.
Feb 26, 02:44 UTC
Monitoring -
We've increased the compute available to the secrets service, and have seen response times return to normal levels.
Feb 26, 00:53 UTC
Investigating -
We're observing increased latency on secrets endpoints for a subset of our customers. We're currently investigating and will provide status updates as they become available.
Feb 26, 00:43 UTC
Resolved -
The cause of pipeline upload failures and dispatch latency that was impacting some of our customers has been resolved.
Feb 17, 21:05 UTC
Update -
We're seeing signs of recovery. Our team is continuing to monitor the situation.
Feb 17, 20:47 UTC
Update -
We've identified that this issue is also impacting pipeline uploads for customers that are affected. This is an isolated issue and not all customers are experiencing impact.
Feb 17, 20:30 UTC
Investigating -
Some customers are experiencing increased latency for jobs being assigned to agents. We are investigating and will provide an update in 30 minutes.
Feb 17, 20:09 UTC
Resolved -
Between 2026-02-12 01:33 and 04:49 UTC, an elevated error rate in job dispatch was observed. This affected a subset of jobs. Affected jobs were unable to be dispatched during the disruption window, but were eventually dispatched due to automatic retries. A subset of pipeline step uploads were also affected; the agent would check for successful step upload over a 5-minute period before timing out.
This occurred due to a bug in a configuration change that resulted in some Sidekiq instances not receiving all required database connection configuration. The configuration change was subsequently reverted, with further investigation underway.
Feb 12, 01:33 UTC