Files
grm/docs/specs/GRM-166.md
T
kireto 91635b5a5d
Post-merge / detect-and-configure (push) Successful in 1m4s
Post-merge / release-and-maintain (push) Successful in 1m27s
GRM-166: fix: restrict docker-prune to exited containers
2026-09-15 14:19:00 +00:00

2.0 KiB

GRM-166: Fix docker-prune killing running molecule containers

Problem

docker-prune.service force-removes containers older than 1h by creation time (RunningFor). Molecule instance containers (ubuntu-2604 etc.) run on the runner's shared rootless daemon and are not name-excluded (only GITEA-ACTIONS-TASK jobs are). A stale molecule container left by a crashed run is restarted/reused by the next run's create phase — it then shows RunningFor >1h and gets docker rm -f'd mid-converge: No such container: ubuntu-2604. Observed on infra nightly run 5710 across multiple runners (incl. a 32 GB host), causing shard failures.

Approach

REQ-1: Change the container cleanup ExecStart in ansible/roles/gitea_runner/templates/docker-prune.service.j2 to only target containers with status=exited (add --filter "status=exited"). Running containers — including stale molecule instances adopted by an active run — are never force-removed. REQ-2: Update the template comment to document the race and why only stopped containers are removed. REQ-3: Extend the template-content molecule verify assertions to require the status=exited filter in the rendered unit.

Test Plan

  • make molecule (template-content scenario) verifies the rendered unit.
  • make pytest-cov and make lint-all pass.

Deploy Plan

  • Merge to master → package publishes; runner hosts pick up the role on their next grm install/upgrade cycle (or manual re-run of the role on affected runners).
  • Until then, nightly molecule reruns remain exposed to the race — mitigated by the fact that swept daemons have no stale containers left.

Rollback Plan

  • Revert the merge commit. Running stale leftovers would then again be force-removed (the pre-existing risky behaviour).

Acceptance Criteria

  • REQ-1: prune rm step filtered to status=exited
  • REQ-2: comment documents the creation-time vs running race
  • REQ-3: template-content verify asserts the exited filter