devx/docs/specs/DEVX-162.md

DEVX-162: Fix registry push race condition: serialize uploads + retry on HTTP 500

Problem

The Gitea container registry (v1.27.2) has a known race condition in BlobUploader.Append() where concurrent blob uploads cause the file offset and DB model to get out of sync, producing HTTP 500 "offset mismatch between file and model" errors. This causes the build-images workflow to fail intermittently when pushing runner images.

The package_blob_upload table accumulates stale entries from failed uploads that worsen the problem over time.

Approach

Two fixes in devx (a third fix — scheduled cleanup — is tracked separately as OBL-INFRA-537):

1. Set DOCKER_MAX_CONCURRENT_UPLOADS=1 in the build-images workflow to serialize blob uploads and avoid the race condition.

2. Add HTTP 500 retry logic to push_image in build_image.py. When a push fails with HTTP 500 (not "already exists"), retry up to 3 times with exponential backoff (5s, 10s, 20s).

REQ-1: Build-images workflow sets DOCKER_MAX_CONCURRENT_UPLOADS=1 REQ-2: push_image retries on HTTP 500 with exponential backoff REQ-3: All existing tests pass with 100% coverage

Test Plan

• Unit tests for retry logic (mock subprocess)
• Manual: trigger build-images workflow and verify push succeeds

Deploy Plan

• Merge to master

Rollback Plan

• Revert the merge commit

Acceptance Criteria

• REQ-1: Build-images workflow sets DOCKER_MAX_CONCURRENT_UPLOADS=1
• REQ-2: push_image retries on HTTP 500 with exponential backoff
• REQ-3: All existing tests pass with 100% coverage