fix: harden stall-detection enumeration and timestamp parsing
CI / validate (pull_request) Successful in 1m9s
CI / molecule-tests (1) (pull_request) Successful in 5m49s
CI / molecule-tests (3) (pull_request) Successful in 6m47s
CI / molecule-tests (2) (pull_request) Successful in 7m10s
CI / molecule-tests (4) (pull_request) Successful in 7m12s
CI / auto-merge (pull_request) Failing after 7m50s

Post-merge review of GRM-168 found three defects: the docker ps
feeding the probe loop lacked a failure guard, so a slow daemon would
abort the healthcheck under pipefail exactly when diagnostics matter;
CreatedAt contains spaces so whitespace-splitting read only captured
the date and broke the 15-minute age gate (GNU date also rejects the
+0000 UTC suffix); and head -200 could truncate the inspect State
block the spec requires. Pipe-separated fields, guarded enumeration,
stripped UTC suffix, and full inspect output fix all three.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Emil Simeonov
2026-09-18 13:01:38 +02:00
co-authored by Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
parent 148c9d3991
commit a61075ebde
2 changed files with 79 additions and 5 deletions
@@ -38,10 +38,15 @@ fi
STALL_MINUTES={{ gitea_runner_stall_minutes }}
DIAG_DIR="{{ gitea_runner_config_dir }}"
now_epoch=$(date +%s)
timeout 15 docker ps --filter "name=GITEA-ACTIONS-TASK" \
--format '{% raw %}{{.ID}} {{.Names}} {{.CreatedAt}}{% endraw %}' 2>/dev/null \
| while read -r cid cname ccreated _rest; do
created_epoch=$(date -d "$ccreated" +%s 2>/dev/null || echo 0)
# Implements: REQ-1 — guard the enumeration: a slow/dead daemon must not
# abort the healthcheck under pipefail; an empty list just skips probing.
# Implements: REQ-2 — pipe-separate fields: CreatedAt contains spaces, so
# whitespace-splitting `read` only captured the date and broke the age gate.
{ timeout 15 docker ps --filter "name=GITEA-ACTIONS-TASK" \
--format '{% raw %}{{.ID}}|{{.Names}}|{{.CreatedAt}}{% endraw %}' 2>/dev/null || true; } \
| while IFS='|' read -r cid cname ccreated _rest; do
# GNU date rejects the redundant " +0000 UTC" suffix — drop it.
created_epoch=$(date -d "${ccreated% UTC}" +%s 2>/dev/null || echo 0)
age_min=$(( (now_epoch - created_epoch) / 60 ))
[[ "$age_min" -lt "$STALL_MINUTES" ]] && continue
marker="$DIAG_DIR/.stall-diag-$cid"
@@ -52,7 +57,7 @@ timeout 15 docker ps --filter "name=GITEA-ACTIONS-TASK" \
echo "=== stall diagnostics for $cname ($cid), age ${age_min}m ==="
echo "--- exec probe: TIMEOUT (>10s) ---"
echo "--- docker inspect ---"
timeout 15 docker inspect "$cid" 2>/dev/null | head -200
timeout 15 docker inspect "$cid" 2>/dev/null
echo "--- docker top ---"
timeout 15 docker top "$cid" 2>/dev/null
echo "--- docker stats --no-stream ---"
+69
View File
@@ -0,0 +1,69 @@
# GRM-170: Fix stall-detection robustness bugs in runner healthcheck
## Problem
Post-merge review of the GRM-168 stall-detection block in
`runner-healthcheck.sh.j2` found three defects:
1. **Missing `|| true` on the container enumeration.** `timeout 15
docker ps … | while …` runs under `set -euo pipefail`. If the daemon
is unresponsive — precisely the condition the section exists to
diagnose — `docker ps` exits nonzero, pipefail propagates it, and the
healthcheck dies mid-run before reaching the runner-service check.
Every other docker call in the script is guarded; this one is not.
2. **CreatedAt split bug.** `docker ps --format '{{.ID}} {{.Names}}
{{.CreatedAt}}'` emits a timestamp containing spaces
(`2026-09-18 10:30:00 +0000 UTC`), but `read -r cid cname ccreated
_rest` only captures `2026-09-18` — the date part. `date -d` then
computes age from midnight: containers created today always appear
≥N hours old, so the 15-minute gate effectively never filters.
3. **`head -200` truncates `docker inspect`.** Inspect output is ~300+
lines and the `State` block (OOMKilled, Pid, times) the spec requires
can be cut off.
## Approach
REQ-1: Wrap the enumeration so a failed `docker ps` yields empty input
instead of aborting the script: `{ timeout 15 docker ps … || true; } |
while …`.
REQ-2: Emit fields separated by `|` (`{{.ID}}|{{.Names}}|{{.CreatedAt}}`)
and parse with `IFS='|' read -r cid cname ccreated _rest` so the full
timestamp reaches `date -d`; also strip the redundant ` UTC` suffix
because GNU date rejects `+0000 UTC` together. The age gate then
compares real minutes.
REQ-3: Remove the `head -200` truncation on `docker inspect` output so
the full State block is captured in the diagnostics bundle.
REQ-4: Diagnostics-only constraint unchanged — no kills, no restarts.
## Test Plan
- Render the template and run `bash -n` on the output.
- Shell-simulate: feed a fake `docker ps` line with spaced CreatedAt and
verify `date -d` computes minutes correctly (manual check).
- `make lint-all` (ansible-lint, actionlint, ruff) passes.
- Molecule gitea_runner scenario converges with the template change.
## Deploy Plan
Merge via auto-merge → release (fix: commit bumps patch) → infra
dependency-bump PR picks up the new role version → runner role applied
on next infra run. This PR also carries the merged-but-unreleased
GRM-168 healthcheck into the release.
## Rollback Plan
Revert the three-line change set; the section degrades to the GRM-168
behavior (still diagnostics-only, just less robust).
## Acceptance Criteria
- [x] `docker ps` enumeration guarded against nonzero exit.
- [x] Full CreatedAt timestamp parsed via `|` separator.
- [x] `docker inspect` captured without truncation.
- [x] Rendered script passes `bash -n`.
- [x] `make lint-all` passes.