Public Access
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 devx-ci-bot
|
d9489b5387 | ||
 kireto
|
35f7bc92cd | ||
|
gitea-actions-bot
|
55bcd8fa01 | ||
|
devx-ci-bot
|
990f70845c | ||
|
kireto
|
149e8846b8 | ||
|
gitea-actions-bot
|
a7cdebc0dc | ||
|
devx-ci-bot
|
012f0979ce | ||
|
kireto
|
62412755cb | ||
|
gitea-actions-bot
|
25f00335df |
@@ -2,6 +2,24 @@
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [0.51.7] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Increase HTTP 500 retry count to 5 with longer backoff and visible logging
|
||||
|
||||
## [0.51.6] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Use stderr=STDOUT to capture all docker push output in one stream
|
||||
|
||||
## [0.51.5] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Check stdout for HTTP 500 in _run_push (docker sends to stdout)
|
||||
|
||||
## [0.51.4] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
@@ -16,12 +16,12 @@ quality badges.
|
||||
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/src/branch/master/LICENSE)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
|
||||
## Why devx?
|
||||
|
||||
@@ -87,7 +87,7 @@ extra index and list devx in your dependencies:
|
||||
```toml
|
||||
[project]
|
||||
dependencies = [
|
||||
"devx>=0.51.4",
|
||||
"devx>=0.51.7",
|
||||
]
|
||||
|
||||
[tool.pip]
|
||||
@@ -101,8 +101,8 @@ pip install -e .
|
||||
```
|
||||
|
||||
> **Note:** If your project requires a specific devx version, pin it in
|
||||
> `dependencies` (for example, `"devx==0.51.4"`) or use a version constraint
|
||||
> (for example, `"devx>=0.51.4,<0.52"`).
|
||||
> `dependencies` (for example, `"devx==0.51.7"`) or use a version constraint
|
||||
> (for example, `"devx>=0.51.7,<0.52"`).
|
||||
|
||||
### Optional extras
|
||||
|
||||
|
||||
+8
-8
@@ -12,12 +12,12 @@ project to be reusable across all oblachno-oss repositories.
|
||||
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/src/branch/master/LICENSE)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -74,14 +74,14 @@ Add devx to your `pyproject.toml` dependencies and configure the registry:
|
||||
```toml
|
||||
[project]
|
||||
dependencies = [
|
||||
"devx>=0.51.4",
|
||||
"devx>=0.51.7",
|
||||
]
|
||||
|
||||
[tool.pip]
|
||||
extra-index-url = "https://git.oblachno.oblachno.fyi/api/packages/oblachno-oss/pypi/simple"
|
||||
```
|
||||
|
||||
Pin a specific version if needed: `"devx==0.51.4"` or `"devx>=0.51.4,<0.52"`.
|
||||
Pin a specific version if needed: `"devx==0.51.7"` or `"devx>=0.51.7,<0.52"`.
|
||||
|
||||
### Optional extras
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# DEVX-163: Fix _run_push to check stdout for HTTP 500
|
||||
|
||||
## Problem
|
||||
`_run_push` only checked `result.stderr` for HTTP 500, but docker push
|
||||
sends the "received unexpected HTTP status: 500 Internal Server Error"
|
||||
message to **stdout**, not stderr. This means the tenacity retry logic
|
||||
added in DEVX-162 never triggered — the push failed immediately without
|
||||
retrying.
|
||||
|
||||
## Approach
|
||||
Check both `result.stdout` and `result.stderr` for the "500" status code.
|
||||
Also update the "already exists" check in `push_image` to check both
|
||||
streams, since docker may send that message to stdout as well.
|
||||
|
||||
REQ-1: _run_push checks both stdout and stderr for HTTP 500
|
||||
REQ-2: push_image "already exists" check uses combined stdout+stderr
|
||||
REQ-3: All existing tests pass with 100% coverage
|
||||
|
||||
## Test Plan
|
||||
- Unit tests for stdout 500 detection
|
||||
- Unit tests for stderr 500 detection
|
||||
- Manual: trigger build-images workflow and verify retry works
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: _run_push checks both stdout and stderr for HTTP 500
|
||||
- [x] REQ-2: push_image "already exists" check uses combined stdout+stderr
|
||||
- [x] REQ-3: All existing tests pass with 100% coverage
|
||||
@@ -0,0 +1,34 @@
|
||||
# DEVX-164: Increase HTTP 500 retry count and backoff for docker push
|
||||
|
||||
## Problem
|
||||
The HTTP 500 retry logic (DEVX-162, DEVX-163) works correctly — 3 retry
|
||||
attempts are made. But all 3 attempts fail because the Gitea registry's
|
||||
"offset mismatch" race condition needs more than ~15s to recover. The
|
||||
current backoff is 5s-20s with 3 attempts (total ~15s of waiting).
|
||||
|
||||
## Approach
|
||||
Increase retry count from 3 to 5 and backoff from 5-20s to 10-60s,
|
||||
giving the registry up to ~2 minutes to recover. Add visible logging
|
||||
between retry attempts so the CI logs show the retry happening.
|
||||
|
||||
REQ-1: Increase retry count from 3 to 5
|
||||
REQ-2: Increase backoff from 5-20s to 10-60s exponential
|
||||
REQ-3: Add visible logging between retry attempts (click.echo)
|
||||
REQ-4: All tests pass with 100% coverage
|
||||
|
||||
## Test Plan
|
||||
- Unit tests verify retry count and backoff parameters
|
||||
- Unit tests verify logging output on retry
|
||||
- Manual: trigger build-images workflow and verify retries visible in logs
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: Increase retry count from 3 to 5
|
||||
- [x] REQ-2: Increase backoff from 5-20s to 10-60s exponential
|
||||
- [x] REQ-3: Add visible logging between retry attempts (click.echo)
|
||||
- [x] REQ-4: All tests pass with 100% coverage
|
||||
@@ -48,12 +48,12 @@ Add devx to your `pyproject.toml`:
|
||||
```toml
|
||||
[project]
|
||||
dependencies = [
|
||||
"devx>=0.51.4",
|
||||
"devx>=0.51.7",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"devx>=0.51.4",
|
||||
"devx>=0.51.7",
|
||||
]
|
||||
```
|
||||
|
||||
|
||||
@@ -6,4 +6,4 @@ create_dependency_pr, auto_merge, release, publish), developer tooling
|
||||
molecule testing helpers for Ansible projects.
|
||||
"""
|
||||
|
||||
__version__ = "0.51.4"
|
||||
__version__ = "0.51.7"
|
||||
|
||||
@@ -266,15 +266,22 @@ def _run_push(cmd: list[str]) -> subprocess.CompletedProcess[str]:
|
||||
BlobUploader.Append that causes intermittent HTTP 500 "offset
|
||||
mismatch" errors during concurrent blob uploads. Retrying the
|
||||
push gives the registry time to recover.
|
||||
|
||||
Docker sends push progress/errors to both stdout and stderr depending
|
||||
on the error type, so both streams are checked for the 500 status.
|
||||
Uses stderr=STDOUT to merge both streams into stdout, ensuring all
|
||||
output is captured in one place (docker push output behavior varies
|
||||
depending on TTY detection).
|
||||
"""
|
||||
result = subprocess.run( # nosec B603
|
||||
cmd,
|
||||
capture_output=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0 and "500" in result.stderr:
|
||||
raise PushHTTP500Error(result.stderr.strip())
|
||||
if result.returncode != 0 and "500" in (result.stdout or ""):
|
||||
raise PushHTTP500Error(result.stdout.strip())
|
||||
return result
|
||||
|
||||
|
||||
@@ -308,9 +315,17 @@ def push_image(
|
||||
click.echo(f"Pushing {ft}...")
|
||||
|
||||
@retry(
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential(multiplier=5, min=5, max=20),
|
||||
stop=stop_after_attempt(5),
|
||||
wait=wait_exponential(multiplier=10, min=10, max=60),
|
||||
retry=retry_if_exception_type(PushHTTP500Error),
|
||||
before_sleep=lambda retry_state: click.echo(
|
||||
_(
|
||||
" HTTP 500 from registry, retrying in {wait:.0f}s (attempt {attempt}/5)...",
|
||||
wait=retry_state.next_action.sleep if retry_state.next_action else 0,
|
||||
attempt=retry_state.attempt_number + 1,
|
||||
),
|
||||
err=True,
|
||||
),
|
||||
reraise=True,
|
||||
)
|
||||
def _attempt(_cmd: list[str] = cmd) -> subprocess.CompletedProcess[str]:
|
||||
@@ -329,10 +344,10 @@ def push_image(
|
||||
if result.returncode == 0:
|
||||
click.echo(f"Pushed {ft}")
|
||||
continue
|
||||
stderr = result.stderr.strip()
|
||||
combined_output = (result.stdout or "").strip()
|
||||
# Gitea #31964: push fails because tag already exists.
|
||||
# Delete the old manifest and retry once.
|
||||
if username and token and "already exists" in stderr.lower():
|
||||
if username and token and "already exists" in combined_output.lower():
|
||||
click.echo(" Tag exists (Gitea #31964), deleting old manifest and retrying...")
|
||||
delete_remote_manifest(
|
||||
registry,
|
||||
@@ -345,16 +360,17 @@ def push_image(
|
||||
click.echo(f" Retrying push {ft}...")
|
||||
result = subprocess.run( # nosec B603
|
||||
cmd,
|
||||
capture_output=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
click.echo(f"Pushed {ft} (after retry)")
|
||||
continue
|
||||
stderr = result.stderr.strip()
|
||||
combined_output = (result.stdout or "").strip()
|
||||
click.echo(
|
||||
_("Push failed for {tag}: {error}", tag=ft, error=stderr),
|
||||
_("Push failed for {tag}: {error}", tag=ft, error=combined_output),
|
||||
err=True,
|
||||
)
|
||||
all_ok = False
|
||||
|
||||
@@ -5152,5 +5152,13 @@
|
||||
"pl": "PR has 'refactoring' label — size check bypassed.",
|
||||
"ru": "PR has 'refactoring' label — size check bypassed.",
|
||||
"zh": "PR has 'refactoring' label — size check bypassed."
|
||||
},
|
||||
" HTTP 500 from registry, retrying in {wait:.0f}s (attempt {attempt}/5)...": {
|
||||
"en": "HTTP 500 from registry, retrying in {wait:.0f}s (attempt {attempt}/5)...",
|
||||
"bg": " HTTP 500 от регистъра, повторен опит след {wait:.0f}с (опит {attempt}/5)...",
|
||||
"de": " HTTP 500 vom Registry, Wiederholung in {wait:.0f}s (Versuch {attempt}/5)...",
|
||||
"pl": " HTTP 500 z rejestru, ponawianie za {wait:.0f}s (próba {attempt}/5)...",
|
||||
"ru": " HTTP 500 от реестра, повтор через {wait:.0f}с (попытка {attempt}/5)...",
|
||||
"zh": " 注册表返回 HTTP 500，{wait:.0f}秒后重试（第{attempt}/5次尝试）..."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -198,7 +198,7 @@ class TestBuildImage:
|
||||
class TestPushImage:
|
||||
def test_success(self) -> None:
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest", "1.0"])
|
||||
mock_result = MagicMock(returncode=0, stderr="", stdout="")
|
||||
mock_result = MagicMock(returncode=0, stdout="")
|
||||
with patch("devx.tools.build_image.subprocess.run", return_value=mock_result) as mock_run:
|
||||
assert push_image(spec, "git.example.com") is True
|
||||
assert mock_run.call_count == 2
|
||||
@@ -206,8 +206,8 @@ class TestPushImage:
|
||||
def test_partial_failure(self) -> None:
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest", "1.0"])
|
||||
results = [
|
||||
MagicMock(returncode=0, stderr="", stdout=""),
|
||||
MagicMock(returncode=1, stderr="push failed", stdout=""),
|
||||
MagicMock(returncode=0, stdout=""),
|
||||
MagicMock(returncode=1, stdout="push failed"),
|
||||
]
|
||||
with patch("devx.tools.build_image.subprocess.run", side_effect=results):
|
||||
assert push_image(spec, "git.example.com") is False
|
||||
@@ -221,7 +221,7 @@ class TestPushImage:
|
||||
def test_no_delete_on_success_with_creds(self) -> None:
|
||||
"""Push-first: no delete needed when push succeeds."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
mock_result = MagicMock(returncode=0, stderr="", stdout="")
|
||||
mock_result = MagicMock(returncode=0, stdout="")
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", return_value=mock_result),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
@@ -231,7 +231,7 @@ class TestPushImage:
|
||||
|
||||
def test_no_delete_without_creds(self) -> None:
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
mock_result = MagicMock(returncode=0, stderr="", stdout="")
|
||||
mock_result = MagicMock(returncode=0, stdout="")
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", return_value=mock_result),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
@@ -243,8 +243,8 @@ class TestPushImage:
|
||||
"""Gitea #31964: push fails with 'already exists', delete + retry."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
results = [
|
||||
MagicMock(returncode=1, stderr="package version already exists", stdout=""),
|
||||
MagicMock(returncode=0, stderr="", stdout=""),
|
||||
MagicMock(returncode=1, stdout="package version already exists"),
|
||||
MagicMock(returncode=0, stdout=""),
|
||||
]
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", side_effect=results),
|
||||
@@ -263,7 +263,7 @@ class TestPushImage:
|
||||
def test_no_delete_on_non_already_exists_failure(self) -> None:
|
||||
"""Push fails for other reasons (non-500) — old manifest preserved."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
mock_result = MagicMock(returncode=1, stderr="denied: requested access to the resource is denied", stdout="")
|
||||
mock_result = MagicMock(returncode=1, stdout="denied: requested access to the resource is denied")
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", return_value=mock_result),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
@@ -275,8 +275,8 @@ class TestPushImage:
|
||||
"""Gitea #31964 retry also fails — both pushes fail."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
results = [
|
||||
MagicMock(returncode=1, stderr="package version already exists", stdout=""),
|
||||
MagicMock(returncode=1, stderr="push failed again", stdout=""),
|
||||
MagicMock(returncode=1, stdout="package version already exists"),
|
||||
MagicMock(returncode=1, stdout="push failed again"),
|
||||
]
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", side_effect=results),
|
||||
@@ -288,8 +288,8 @@ class TestPushImage:
|
||||
"""HTTP 500 from registry race condition — retry succeeds."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
results = [
|
||||
MagicMock(returncode=1, stderr="received unexpected HTTP status: 500 Internal Server Error", stdout=""),
|
||||
MagicMock(returncode=0, stderr="", stdout=""),
|
||||
MagicMock(returncode=1, stdout="received unexpected HTTP status: 500 Internal Server Error"),
|
||||
MagicMock(returncode=0, stdout=""),
|
||||
]
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", side_effect=results),
|
||||
@@ -302,9 +302,7 @@ class TestPushImage:
|
||||
def test_http_500_retries_all_fail(self) -> None:
|
||||
"""HTTP 500 retries exhausted — push fails, no delete attempted."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
mock_result = MagicMock(
|
||||
returncode=1, stderr="received unexpected HTTP status: 500 Internal Server Error", stdout=""
|
||||
)
|
||||
mock_result = MagicMock(returncode=1, stdout="received unexpected HTTP status: 500 Internal Server Error")
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", return_value=mock_result),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
@@ -314,19 +312,19 @@ class TestPushImage:
|
||||
mock_del.assert_not_called()
|
||||
|
||||
def test_run_push_raises_on_500(self) -> None:
|
||||
"""_run_push raises PushHTTP500Error when stderr contains 500."""
|
||||
"""_run_push raises PushHTTP500Error when stdout contains 500."""
|
||||
from devx.tools.build_image import _run_push
|
||||
|
||||
mock_result = MagicMock(returncode=1, stderr="HTTP 500 Internal Server Error", stdout="")
|
||||
mock_result = MagicMock(returncode=1, stdout="received unexpected HTTP status: 500 Internal Server Error")
|
||||
with patch("devx.tools.build_image.subprocess.run", return_value=mock_result):
|
||||
with pytest.raises(PushHTTP500Error, match="HTTP 500"):
|
||||
with pytest.raises(PushHTTP500Error, match="500"):
|
||||
_run_push(["docker", "push", "img:latest"])
|
||||
|
||||
def test_run_push_no_raise_on_non_500(self) -> None:
|
||||
"""_run_push returns result when stderr has no 500."""
|
||||
"""_run_push returns result when stdout has no 500."""
|
||||
from devx.tools.build_image import _run_push
|
||||
|
||||
mock_result = MagicMock(returncode=1, stderr="denied: access denied", stdout="")
|
||||
mock_result = MagicMock(returncode=1, stdout="denied: access denied")
|
||||
with patch("devx.tools.build_image.subprocess.run", return_value=mock_result):
|
||||
result = _run_push(["docker", "push", "img:latest"])
|
||||
assert result.returncode == 1
|
||||
@@ -335,7 +333,7 @@ class TestPushImage:
|
||||
"""_run_push returns result on success."""
|
||||
from devx.tools.build_image import _run_push
|
||||
|
||||
mock_result = MagicMock(returncode=0, stderr="", stdout="")
|
||||
mock_result = MagicMock(returncode=0, stdout="")
|
||||
with patch("devx.tools.build_image.subprocess.run", return_value=mock_result):
|
||||
result = _run_push(["docker", "push", "img:latest"])
|
||||
assert result.returncode == 0
|
||||
|
||||
Reference in New Issue
Block a user