# GRM-168: Capture dockerd diagnostics when a CI job container stalls

## Problem

Recurring CI failures (5+ times on 2026-09-17/18, infra runs 5913, 5925,
5943, 5955 notify-sso-bridge): ~20 min into a long-running job,
act_runner's API calls into the job container (`docker exec`, archive
fetch of `/var/run/act/workflow/*.txt`) time out with
`docker daemon ping during version negotiation failed /
context deadline exceeded` — killing the job.

Established facts:

- Host rootless dockerd never restarted (all daemons up since Sep 14);
  the healthcheck's 10 s `docker info` never timed out — the daemon API
  stayed responsive at daemon level.
- No OOM, disk, inode, or load pressure on the host.
- The wedge is therefore per-container (shim/exec path), most consistent
  with attach-stdio backpressure or a containerd-shim event stall — but
  cannot be confirmed post-mortem because job containers and their
  dockerd goroutine state are gone by the time anyone looks.

A `SIGUSR1` dockerd dump is not useful here: it lands in the user
journal, which runner users cannot read (2026-08-08 journal-permission
incident documented in this file's header comments).

## Approach

Extend `runner-healthcheck.sh.j2` with a stall-detection section that
runs after the daemon liveness check. On every healthcheck tick (2 min):

REQ-1: For each running `GITEA-ACTIONS-TASK-*` container older than
`gitea_runner_stall_minutes` (default 15), probe exec responsiveness
with `timeout 10 docker exec <id> true`.

REQ-2: If the probe times out, write a diagnostics bundle to
`{{ gitea_runner_config_dir }}/stall-diag-<container>-<timestamp>.log`
containing: probe result, `docker inspect` output (State, OOMKilled,
Pid, finished/started times), `docker top` output, `docker stats
--no-stream` for the container, and `docker events --since 30m` output.
Each line prefixed with the container name for grepability.

REQ-3: Cooldown per container — write at most one diagnostics bundle
per container id (marker file under the same dir), so a 2-minute
healthcheck does not spam dumps on a persistent stall.

REQ-4: Do not kill or restart anything — diagnostics only. The job may
recover on its own; if it does not, the captured evidence isolates
shim-vs-daemon and stream-vs-exec for the follow-up fix.

## Files Affected

- `ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2` (extend)
- `ansible/roles/gitea_runner/defaults/main.yml` (add `gitea_runner_stall_minutes`)
- `docs/specs/GRM-168.md` (new)

## Test Plan

- `make lint-all` (ansible-lint + shellcheck-adjacent linters) passes.
- Molecule fast-converge on the gitea_runner role scenario that deploys
  the healthcheck template (template renders without error).
- Manual trace: the new section only touches containers matching
  `GITEA-ACTIONS-TASK-*` older than the threshold; a stalled exec probe
  writes exactly one bundle per container.

## Deploy Plan

Merge via auto-merge → GRM release → infra picks up the new version via
the automated dependency PR. Runner hosts get the updated healthcheck on
the next `gitea_runner` role apply (nightly or manual run).

## Rollback Plan

Revert the template change — the healthcheck returns to the previous
probe set. The diagnostics path is additive; removing it risks nothing.

## Acceptance Criteria

- [x] Stalled job containers probed via `timeout docker exec`.
- [x] One diagnostics bundle per stalled container, written to the
      runner config dir (readable without journal access).
- [x] Per-container cooldown prevents dump spam.
- [x] Nothing is killed/restarted — diagnostics only.
- [x] `make lint-all` passes.