oblachno-oss/devx
Public Access
3
0
Fork 0
Code Issues 53 Pull Requests Actions Packages Projects Releases 10 Wiki Activity

fix: run molecule destroy on test failure to clean up containers

Browse Source 
When a molecule test fails or is interrupted (cross-runner cancellation
or KeyboardInterrupt), the test containers are left running because the
final 'destroy' step in the molecule test sequence never executes.
These stale containers accumulate on runner hosts, consuming disk and
memory, causing CI test speed variance.

Added molecule destroy cleanup in all three exit paths:
- Test failure (non-zero exit code)
- Cross-runner cancellation (failed_event)
- KeyboardInterrupt

The destroy command uses the same cwd and env as the test run, with a
120s timeout and suppressed errors (best-effort cleanup).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
emil
2026-08-08 13:56:37 +02:00
co-authored by Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
parent 808e7a2e42
commit d8919ae836
1 changed files with 44 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/devx/molecule/molecule_ci_guard.py
+44
View File
@@ -246,18 +246,62 @@ def cli(pairs: tuple[str, ...], roles_root: Path | None) -> None:
with contextlib.suppress(ProcessLookupError):
os.killpg(os.getpgid(process.pid), signal.SIGKILL)
process.wait()
# Clean up containers left behind by the killed test.
click.echo(_("Cleaning up: running molecule destroy for {scenario}", scenario=scenario))
destroy_cmd = ["molecule", "destroy"]
if scenario != "default":
destroy_cmd.extend(["-s", scenario])
with contextlib.suppress(subprocess.SubprocessError, OSError):
subprocess.run( # nosec B603, B607
destroy_cmd,
cwd=str(cwd),
env=env,
check=False,
capture_output=True,
timeout=120,
)
sys.exit(1)
time.sleep(1)
except KeyboardInterrupt:
with contextlib.suppress(ProcessLookupError):
os.killpg(os.getpgid(process.pid), signal.SIGTERM)
process.wait()
# Clean up containers left behind by the interrupted test.
click.echo(_("Cleaning up: running molecule destroy for {scenario}", scenario=scenario))
destroy_cmd = ["molecule", "destroy"]
if scenario != "default":
destroy_cmd.extend(["-s", scenario])
with contextlib.suppress(subprocess.SubprocessError, OSError):
subprocess.run( # nosec B603, B607
destroy_cmd,
cwd=str(cwd),
env=env,
check=False,
capture_output=True,
timeout=120,
)
sys.exit(1)
rc = process.returncode
if rc != 0:
click.echo(_("FAILED: {pair} exited with code {code}", pair=pair, code=rc))
# Run molecule destroy to clean up containers left behind by the
# failed test. Without this, containers stay running and accumulate
# on the runner, consuming disk/memory and degrading CI performance.
click.echo(_("Cleaning up: running molecule destroy for {scenario}", scenario=scenario))
destroy_cmd = ["molecule", "destroy"]
if scenario != "default":
destroy_cmd.extend(["-s", scenario])
with contextlib.suppress(subprocess.SubprocessError, OSError):
subprocess.run( # nosec B603, B607
destroy_cmd,
cwd=str(cwd),
env=env,
check=False,
capture_output=True,
timeout=120,
)
sys.exit(rc)
click.echo(_("PASSED: {pair}", pair=pair))