DEVX-162: fix: serialize registry uploads and retry on HTTP 500

2026-08-26 13:12:45 +00:00
parent 5986b5b9ed
commit 2d7b4bdac3
4 changed files with 151 additions and 11 deletions
@@ -0,0 +1,41 @@
+# DEVX-162: Fix registry push race condition: serialize uploads + retry on HTTP 500
+
+## Problem
+The Gitea container registry (v1.27.2) has a known race condition in
+`BlobUploader.Append()` where concurrent blob uploads cause the file
+offset and DB model to get out of sync, producing HTTP 500 "offset
+mismatch between file and model" errors. This causes the build-images
+workflow to fail intermittently when pushing runner images.
+
+The `package_blob_upload` table accumulates stale entries from failed
+uploads that worsen the problem over time.
+
+## Approach
+Two fixes in devx (a third fix — scheduled cleanup — is tracked
+separately as OBL-INFRA-537):
+
+1. Set `DOCKER_MAX_CONCURRENT_UPLOADS=1` in the build-images workflow
+   to serialize blob uploads and avoid the race condition.
+
+2. Add HTTP 500 retry logic to `push_image` in `build_image.py`.
+   When a push fails with HTTP 500 (not "already exists"), retry up
+   to 3 times with exponential backoff (5s, 10s, 20s).
+
+REQ-1: Build-images workflow sets DOCKER_MAX_CONCURRENT_UPLOADS=1
+REQ-2: push_image retries on HTTP 500 with exponential backoff
+REQ-3: All existing tests pass with 100% coverage
+
+## Test Plan
+- Unit tests for retry logic (mock subprocess)
+- Manual: trigger build-images workflow and verify push succeeds
+
+## Deploy Plan
+- Merge to master
+
+## Rollback Plan
+- Revert the merge commit
+
+## Acceptance Criteria
+- [x] REQ-1: Build-images workflow sets DOCKER_MAX_CONCURRENT_UPLOADS=1
+- [x] REQ-2: push_image retries on HTTP 500 with exponential backoff
+- [x] REQ-3: All existing tests pass with 100% coverage