85 lines
3.6 KiB
Markdown
85 lines
3.6 KiB
Markdown
# GRM-168: Capture dockerd diagnostics when a CI job container stalls
|
|
|
|
## Problem
|
|
|
|
Recurring CI failures (5+ times on 2026-09-17/18, infra runs 5913, 5925,
|
|
5943, 5955 notify-sso-bridge): ~20 min into a long-running job,
|
|
act_runner's API calls into the job container (`docker exec`, archive
|
|
fetch of `/var/run/act/workflow/*.txt`) time out with
|
|
`docker daemon ping during version negotiation failed /
|
|
context deadline exceeded` — killing the job.
|
|
|
|
Established facts:
|
|
|
|
- Host rootless dockerd never restarted (all daemons up since Sep 14);
|
|
the healthcheck's 10 s `docker info` never timed out — the daemon API
|
|
stayed responsive at daemon level.
|
|
- No OOM, disk, inode, or load pressure on the host.
|
|
- The wedge is therefore per-container (shim/exec path), most consistent
|
|
with attach-stdio backpressure or a containerd-shim event stall — but
|
|
cannot be confirmed post-mortem because job containers and their
|
|
dockerd goroutine state are gone by the time anyone looks.
|
|
|
|
A `SIGUSR1` dockerd dump is not useful here: it lands in the user
|
|
journal, which runner users cannot read (2026-08-08 journal-permission
|
|
incident documented in this file's header comments).
|
|
|
|
## Approach
|
|
|
|
Extend `runner-healthcheck.sh.j2` with a stall-detection section that
|
|
runs after the daemon liveness check. On every healthcheck tick (2 min):
|
|
|
|
REQ-1: For each running `GITEA-ACTIONS-TASK-*` container older than
|
|
`gitea_runner_stall_minutes` (default 15), probe exec responsiveness
|
|
with `timeout 10 docker exec <id> true`.
|
|
|
|
REQ-2: If the probe times out, write a diagnostics bundle to
|
|
`{{ gitea_runner_config_dir }}/stall-diag-<container>-<timestamp>.log`
|
|
containing: probe result, `docker inspect` output (State, OOMKilled,
|
|
Pid, finished/started times), `docker top` output, `docker stats
|
|
--no-stream` for the container, and `docker events --since 30m` output.
|
|
Each line prefixed with the container name for grepability.
|
|
|
|
REQ-3: Cooldown per container — write at most one diagnostics bundle
|
|
per container id (marker file under the same dir), so a 2-minute
|
|
healthcheck does not spam dumps on a persistent stall.
|
|
|
|
REQ-4: Do not kill or restart anything — diagnostics only. The job may
|
|
recover on its own; if it does not, the captured evidence isolates
|
|
shim-vs-daemon and stream-vs-exec for the follow-up fix.
|
|
|
|
## Files Affected
|
|
|
|
- `ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2` (extend)
|
|
- `ansible/roles/gitea_runner/defaults/main.yml` (add `gitea_runner_stall_minutes`)
|
|
- `docs/specs/GRM-168.md` (new)
|
|
|
|
## Test Plan
|
|
|
|
- `make lint-all` (ansible-lint + shellcheck-adjacent linters) passes.
|
|
- Molecule fast-converge on the gitea_runner role scenario that deploys
|
|
the healthcheck template (template renders without error).
|
|
- Manual trace: the new section only touches containers matching
|
|
`GITEA-ACTIONS-TASK-*` older than the threshold; a stalled exec probe
|
|
writes exactly one bundle per container.
|
|
|
|
## Deploy Plan
|
|
|
|
Merge via auto-merge → GRM release → infra picks up the new version via
|
|
the automated dependency PR. Runner hosts get the updated healthcheck on
|
|
the next `gitea_runner` role apply (nightly or manual run).
|
|
|
|
## Rollback Plan
|
|
|
|
Revert the template change — the healthcheck returns to the previous
|
|
probe set. The diagnostics path is additive; removing it risks nothing.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [x] Stalled job containers probed via `timeout docker exec`.
|
|
- [x] One diagnostics bundle per stalled container, written to the
|
|
runner config dir (readable without journal access).
|
|
- [x] Per-container cooldown prevents dump spam.
|
|
- [x] Nothing is killed/restarted — diagnostics only.
|
|
- [x] `make lint-all` passes.
|