GRM-166: fix: restrict docker-prune to exited containers

2026-09-15 14:19:00 +00:00
parent bd0910287a
commit 91635b5a5d
3 changed files with 53 additions and 7 deletions
@@ -0,0 +1,43 @@
+# GRM-166: Fix docker-prune killing running molecule containers
+
+## Problem
+`docker-prune.service` force-removes containers older than 1h by
+**creation** time (`RunningFor`). Molecule instance containers
+(`ubuntu-2604` etc.) run on the runner's shared rootless daemon and are
+not name-excluded (only `GITEA-ACTIONS-TASK` jobs are). A stale molecule
+container left by a crashed run is *restarted/reused* by the next run's
+create phase — it then shows `RunningFor` >1h and gets `docker rm -f`'d
+mid-converge: `No such container: ubuntu-2604`. Observed on infra nightly
+run 5710 across multiple runners (incl. a 32 GB host), causing
+shard failures.
+
+## Approach
+REQ-1: Change the container cleanup `ExecStart` in
+  `ansible/roles/gitea_runner/templates/docker-prune.service.j2` to only
+  target containers with `status=exited` (add
+  `--filter "status=exited"`). Running containers — including stale
+  molecule instances adopted by an active run — are never force-removed.
+REQ-2: Update the template comment to document the race and why only
+  stopped containers are removed.
+REQ-3: Extend the `template-content` molecule verify assertions to
+  require the `status=exited` filter in the rendered unit.
+
+## Test Plan
+- `make molecule` (template-content scenario) verifies the rendered unit.
+- `make pytest-cov` and `make lint-all` pass.
+
+## Deploy Plan
+- Merge to master → package publishes; runner hosts pick up the role on
+  their next grm install/upgrade cycle (or manual re-run of the role on
+  affected runners).
+- Until then, nightly molecule reruns remain exposed to the race —
+  mitigated by the fact that swept daemons have no stale containers left.
+
+## Rollback Plan
+- Revert the merge commit. Running stale leftovers would then again be
+  force-removed (the pre-existing risky behaviour).
+
+## Acceptance Criteria
+- [x] REQ-1: prune rm step filtered to `status=exited`
+- [x] REQ-2: comment documents the creation-time vs running race
+- [x] REQ-3: template-content verify asserts the exited filter