Periodic latency spikes in artifact upload

Incident Report for Buildkite

Resolved

This incident has been resolved.

Posted Aug 17, 2023 - 23:51 UTC

Monitoring

Our cloud provider put a workaround in place, and we are currently monitoring its results. Latency levels on artifact upload are back to normal levels.

Posted Aug 17, 2023 - 23:01 UTC

Identified

We continued experiencing spikes in latency on artifact uploads, and we've escalated the issue with our cloud provider and working with them on a resolution. Retrying jobs with artifact failures will likely succeed. We'll provide an update in an hour.

Posted Aug 17, 2023 - 22:16 UTC

Investigating

Our database that handles artifact metadata is experiencing spikes in latency approximately every 18 minutes due to a hardware issue, causing some artifact uploads to time out. We are working with our cloud provider to resolve this. Failures with artifacts will likely succeed if retried.

Posted Aug 17, 2023 - 20:16 UTC

This incident affected: Agent API.