grm/docs/specs/GRM-167.md

GRM-167: Fix runner healthcheck force-removing active molecule containers

Problem

runner-healthcheck.sh runs every 2 minutes and force-removes molecule instance containers on two disk paths:

• Warn path (disk >= 70%): docker rm -f any non-GITEA-ACTIONS-TASK container with RunningFor >= 1h — same creation-age race fixed in GRM-166 for the prune service. A stale molecule instance restarted by a new run still reads >1h old and is killed mid-converge.
• Critical path (disk >= 75%): docker rm -f ALL non-GITEA-ACTIONS-TASK containers with no age or status filter — running molecule instances die instantly. Under parallel DinD load the runner disk crosses 75% routinely (infra nightly run 5710: No such container: ubuntu-2604 across three different runners).

Approach

REQ-1: In ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2, restrict both disk-pressure removal paths to containers that are not running: add --filter "status=exited" --filter "status=created" is NOT sufficient for the critical path since a just-created molecule instance is in created state — use status=exited and status=dead only. Running containers are never janitor-killed; the image/volume/system prunes still reclaim the actual disk. REQ-2: Keep GITEA-ACTIONS-TASK exclusion, the 1h age gate on the warn path, and all image/volume/network/builder prunes unchanged. REQ-3: Update comments documenting the running-container race. REQ-4: Extend molecule template-content verify assertions for the rendered healthcheck script (status=exited present in both paths).

Test Plan

• make molecule template-content + default scenarios pass.
• make pytest-cov, make lint-all pass.

Deploy Plan

• Merge to master → package publishes. Runner hosts apply the role on their next grm install/upgrade run; affected runners may need a manual role re-run for immediate relief.
• Vikunja counter reused GRM-167; prior spec preserved in GRM-167-devx-bump-historical.

Rollback Plan

• Revert the merge commit — restores the aggressive cleanup that kills active CI jobs.

Acceptance Criteria

• REQ-1: both disk paths only remove status=exited/status=dead containers
• REQ-2: exclusions, age gate, and non-container prunes unchanged
• REQ-3: comments updated
• REQ-4: template-content verify covers the filters