Emil Simeonov
3f1f3e3134
A stale molecule instance restarted by a new run still shows RunningFor
older than 1h, so the janitor force-removed it mid-converge
('No such container'). Only stopped containers are eligible now.
2.0 KiB
GRM-166: Fix docker-prune killing running molecule containers
Problem
docker-prune.service force-removes containers older than 1h by
creation time (RunningFor). Molecule instance containers
(ubuntu-2604 etc.) run on the runner's shared rootless daemon and are
not name-excluded (only GITEA-ACTIONS-TASK jobs are). A stale molecule
container left by a crashed run is restarted/reused by the next run's
create phase — it then shows RunningFor >1h and gets docker rm -f'd
mid-converge: No such container: ubuntu-2604. Observed on infra nightly
run 5710 across multiple runners (incl. a 32 GB host), causing
shard failures.
Approach
REQ-1: Change the container cleanup ExecStart in
ansible/roles/gitea_runner/templates/docker-prune.service.j2 to only
target containers with status=exited (add
--filter "status=exited"). Running containers — including stale
molecule instances adopted by an active run — are never force-removed.
REQ-2: Update the template comment to document the race and why only
stopped containers are removed.
REQ-3: Extend the template-content molecule verify assertions to
require the status=exited filter in the rendered unit.
Test Plan
make molecule(template-content scenario) verifies the rendered unit.make pytest-covandmake lint-allpass.
Deploy Plan
- Merge to master → package publishes; runner hosts pick up the role on their next grm install/upgrade cycle (or manual re-run of the role on affected runners).
- Until then, nightly molecule reruns remain exposed to the race — mitigated by the fact that swept daemons have no stale containers left.
Rollback Plan
- Revert the merge commit. Running stale leftovers would then again be force-removed (the pre-existing risky behaviour).
Acceptance Criteria
- REQ-1: prune rm step filtered to
status=exited - REQ-2: comment documents the creation-time vs running race
- REQ-3: template-content verify asserts the exited filter