From 00:49 to 01:26 on Nov 15, 2024 (UTC), an estimated 1% of artifacts failed to upload due to signature verification errors.
Up to 1% of build artifact uploads, principally those uploaded by Agent version v3.83.0 or later, encountered signature mismatch errors. This Agent version introduced multipart uploads. A backward-incompatible server library upgrade instigated the incident. Seven minutes after the incident was detected, we rolled back the change. The root cause was a recent upgrade of Ruby library dependencies responsible for URL presigning. One library added an additional header for all REST API calls, a change not accounted for in the older version of a related library which we did not upgrade at the same time. This omission led to the URL signature mismatch errors on upload.
In future, we’ve ensured that this group of libraries is upgraded as a whole to prevent dependency mismatches that could introduce unintentional breaking changes. Additionally, we will enhance our test coverage around presigned URLs to ensure that their signatures match the expectations of our upload service, and improve monitoring of the upload completion rate to reduce detection time.