grm/docs/specs/GRM-168.md

GRM-168: Capture dockerd diagnostics when a CI job container stalls

Problem

Recurring CI failures (5+ times on 2026-09-17/18, infra runs 5913, 5925, 5943, 5955 notify-sso-bridge): ~20 min into a long-running job, act_runner's API calls into the job container (docker exec, archive fetch of /var/run/act/workflow/*.txt) time out with docker daemon ping during version negotiation failed / context deadline exceeded — killing the job.

Established facts:

• Host rootless dockerd never restarted (all daemons up since Sep 14); the healthcheck's 10 s docker info never timed out — the daemon API stayed responsive at daemon level.
• No OOM, disk, inode, or load pressure on the host.
• The wedge is therefore per-container (shim/exec path), most consistent with attach-stdio backpressure or a containerd-shim event stall — but cannot be confirmed post-mortem because job containers and their dockerd goroutine state are gone by the time anyone looks.

A SIGUSR1 dockerd dump is not useful here: it lands in the user journal, which runner users cannot read (2026-08-08 journal-permission incident documented in this file's header comments).

Approach

Extend runner-healthcheck.sh.j2 with a stall-detection section that runs after the daemon liveness check. On every healthcheck tick (2 min):

REQ-1: For each running GITEA-ACTIONS-TASK-* container older than gitea_runner_stall_minutes (default 15), probe exec responsiveness with timeout 10 docker exec <id> true.

REQ-2: If the probe times out, write a diagnostics bundle to {{ gitea_runner_config_dir }}/stall-diag-<container>-<timestamp>.log containing: probe result, docker inspect output (State, OOMKilled, Pid, finished/started times), docker top output, docker stats --no-stream for the container, and docker events --since 30m output. Each line prefixed with the container name for grepability.

REQ-3: Cooldown per container — write at most one diagnostics bundle per container id (marker file under the same dir), so a 2-minute healthcheck does not spam dumps on a persistent stall.

REQ-4: Do not kill or restart anything — diagnostics only. The job may recover on its own; if it does not, the captured evidence isolates shim-vs-daemon and stream-vs-exec for the follow-up fix.

Files Affected

• ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2 (extend)
• ansible/roles/gitea_runner/defaults/main.yml (add gitea_runner_stall_minutes)
• docs/specs/GRM-168.md (new)

Test Plan

• make lint-all (ansible-lint + shellcheck-adjacent linters) passes.
• Molecule fast-converge on the gitea_runner role scenario that deploys the healthcheck template (template renders without error).
• Manual trace: the new section only touches containers matching GITEA-ACTIONS-TASK-* older than the threshold; a stalled exec probe writes exactly one bundle per container.

Deploy Plan

Merge via auto-merge → GRM release → infra picks up the new version via the automated dependency PR. Runner hosts get the updated healthcheck on the next gitea_runner role apply (nightly or manual run).

Rollback Plan

Revert the template change — the healthcheck returns to the previous probe set. The diagnostics path is additive; removing it risks nothing.

Acceptance Criteria

• Stalled job containers probed via timeout docker exec.
• One diagnostics bundle per stalled container, written to the runner config dir (readable without journal access).
• Per-container cooldown prevents dump spam.
• Nothing is killed/restarted — diagnostics only.
• make lint-all passes.