52 lines
2.4 KiB
Markdown
52 lines
2.4 KiB
Markdown
# GRM-167: Fix runner healthcheck force-removing active molecule containers
|
|
|
|
## Problem
|
|
`runner-healthcheck.sh` runs every 2 minutes and force-removes molecule
|
|
instance containers on two disk paths:
|
|
|
|
- **Warn path (disk >= 70%)**: `docker rm -f` any non-`GITEA-ACTIONS-TASK`
|
|
container with `RunningFor` >= 1h — same creation-age race fixed in
|
|
GRM-166 for the prune service. A stale molecule instance restarted by a
|
|
new run still reads >1h old and is killed mid-converge.
|
|
- **Critical path (disk >= 75%)**: `docker rm -f` ALL
|
|
non-`GITEA-ACTIONS-TASK` containers with no age or status filter —
|
|
running molecule instances die instantly. Under parallel DinD load the
|
|
runner disk crosses 75% routinely (infra nightly run 5710: `No such
|
|
container: ubuntu-2604` across three different runners).
|
|
|
|
## Approach
|
|
REQ-1: In `ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2`,
|
|
restrict both disk-pressure removal paths to containers that are not
|
|
running: add `--filter "status=exited" --filter "status=created"` is
|
|
NOT sufficient for the critical path since a just-created molecule
|
|
instance is in `created` state — use `status=exited` and
|
|
`status=dead` only. Running containers are never janitor-killed; the
|
|
image/volume/system prunes still reclaim the actual disk.
|
|
REQ-2: Keep `GITEA-ACTIONS-TASK` exclusion, the 1h age gate on the warn
|
|
path, and all image/volume/network/builder prunes unchanged.
|
|
REQ-3: Update comments documenting the running-container race.
|
|
REQ-4: Extend molecule `template-content` verify assertions for the
|
|
rendered healthcheck script (`status=exited` present in both paths).
|
|
|
|
## Test Plan
|
|
- `make molecule` template-content + default scenarios pass.
|
|
- `make pytest-cov`, `make lint-all` pass.
|
|
|
|
## Deploy Plan
|
|
- Merge to master → package publishes. Runner hosts apply the role on
|
|
their next grm install/upgrade run; affected runners may need a manual
|
|
role re-run for immediate relief.
|
|
- Vikunja counter reused GRM-167; prior spec preserved in
|
|
[GRM-167-devx-bump-historical](GRM-167-devx-bump-historical.md).
|
|
|
|
## Rollback Plan
|
|
- Revert the merge commit — restores the aggressive cleanup that kills
|
|
active CI jobs.
|
|
|
|
## Acceptance Criteria
|
|
- [x] REQ-1: both disk paths only remove `status=exited`/`status=dead`
|
|
containers
|
|
- [x] REQ-2: exclusions, age gate, and non-container prunes unchanged
|
|
- [x] REQ-3: comments updated
|
|
- [x] REQ-4: template-content verify covers the filters
|