We have successfully disabled the job causing load issues. we have also identified and mitigated a seperate database performance problem that we believe contributed to this incident.
Posted Feb 13, 2024 - 00:08 UTC
Update
We have identified a specific background job that results in excessive database load. Currently we have paused all background jobs while we deploy a change to disable just the problematic job.
Posted Feb 12, 2024 - 22:29 UTC
Monitoring
We've implemented a mitigation and have seen performance improve and are continuing our investigation into the cause
Posted Feb 12, 2024 - 21:37 UTC
Investigating
We are experiencing poor performance across the application and are investigating.
Posted Feb 12, 2024 - 21:27 UTC
This incident affected: Web, Agent API, REST API, Job Queue, SCM Integrations and Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).