Public Access
emil
43a098cd9b
Set DOCKER_MAX_CONCURRENT_UPLOADS=1 in build-images workflow to avoid the Gitea registry BlobUploader.Append race condition. Add tenacity-based retry (3 attempts, exponential backoff) for HTTP 500 errors in push_image. # Implements: REQ-1, REQ-2
42 lines
1.5 KiB
Markdown
42 lines
1.5 KiB
Markdown
# DEVX-162: Fix registry push race condition: serialize uploads + retry on HTTP 500
|
|
|
|
## Problem
|
|
The Gitea container registry (v1.27.2) has a known race condition in
|
|
`BlobUploader.Append()` where concurrent blob uploads cause the file
|
|
offset and DB model to get out of sync, producing HTTP 500 "offset
|
|
mismatch between file and model" errors. This causes the build-images
|
|
workflow to fail intermittently when pushing runner images.
|
|
|
|
The `package_blob_upload` table accumulates stale entries from failed
|
|
uploads that worsen the problem over time.
|
|
|
|
## Approach
|
|
Two fixes in devx (a third fix — scheduled cleanup — is tracked
|
|
separately as OBL-INFRA-537):
|
|
|
|
1. Set `DOCKER_MAX_CONCURRENT_UPLOADS=1` in the build-images workflow
|
|
to serialize blob uploads and avoid the race condition.
|
|
|
|
2. Add HTTP 500 retry logic to `push_image` in `build_image.py`.
|
|
When a push fails with HTTP 500 (not "already exists"), retry up
|
|
to 3 times with exponential backoff (5s, 10s, 20s).
|
|
|
|
REQ-1: Build-images workflow sets DOCKER_MAX_CONCURRENT_UPLOADS=1
|
|
REQ-2: push_image retries on HTTP 500 with exponential backoff
|
|
REQ-3: All existing tests pass with 100% coverage
|
|
|
|
## Test Plan
|
|
- Unit tests for retry logic (mock subprocess)
|
|
- Manual: trigger build-images workflow and verify push succeeds
|
|
|
|
## Deploy Plan
|
|
- Merge to master
|
|
|
|
## Rollback Plan
|
|
- Revert the merge commit
|
|
|
|
## Acceptance Criteria
|
|
- [x] REQ-1: Build-images workflow sets DOCKER_MAX_CONCURRENT_UPLOADS=1
|
|
- [x] REQ-2: push_image retries on HTTP 500 with exponential backoff
|
|
- [x] REQ-3: All existing tests pass with 100% coverage
|