GRM-168: docs: fix vale quote punctuation in spec
This commit was merged in pull request #275.
This commit is contained in:
+72
-9
@@ -1,21 +1,84 @@
|
||||
# GRM-168: Bump devx to v0.51.9
|
||||
# GRM-168: Capture dockerd diagnostics when a CI job container stalls
|
||||
|
||||
## Problem
|
||||
grm pins devx@v0.51.0 which rejects `deps:` as a conventional commit type,
|
||||
causing post-merge CI failures on dependency bump commits.
|
||||
|
||||
Recurring CI failures (5+ times on 2026-09-17/18, infra runs 5913, 5925,
|
||||
5943, 5955 notify-sso-bridge): ~20 min into a long-running job,
|
||||
act_runner's API calls into the job container (`docker exec`, archive
|
||||
fetch of `/var/run/act/workflow/*.txt`) time out with
|
||||
`docker daemon ping during version negotiation failed /
|
||||
context deadline exceeded` — killing the job.
|
||||
|
||||
Established facts:
|
||||
|
||||
- Host rootless dockerd never restarted (all daemons up since Sep 14);
|
||||
the healthcheck's 10 s `docker info` never timed out — the daemon API
|
||||
stayed responsive at daemon level.
|
||||
- No OOM, disk, inode, or load pressure on the host.
|
||||
- The wedge is therefore per-container (shim/exec path), most consistent
|
||||
with attach-stdio backpressure or a containerd-shim event stall — but
|
||||
cannot be confirmed post-mortem because job containers and their
|
||||
dockerd goroutine state are gone by the time anyone looks.
|
||||
|
||||
A `SIGUSR1` dockerd dump is not useful here: it lands in the user
|
||||
journal, which runner users cannot read (2026-08-08 journal-permission
|
||||
incident documented in this file's header comments).
|
||||
|
||||
## Approach
|
||||
REQ-1: Bump devx from v0.51.0 to v0.51.9 in pyproject.toml
|
||||
|
||||
Extend `runner-healthcheck.sh.j2` with a stall-detection section that
|
||||
runs after the daemon liveness check. On every healthcheck tick (2 min):
|
||||
|
||||
REQ-1: For each running `GITEA-ACTIONS-TASK-*` container older than
|
||||
`gitea_runner_stall_minutes` (default 15), probe exec responsiveness
|
||||
with `timeout 10 docker exec <id> true`.
|
||||
|
||||
REQ-2: If the probe times out, write a diagnostics bundle to
|
||||
`{{ gitea_runner_config_dir }}/stall-diag-<container>-<timestamp>.log`
|
||||
containing: probe result, `docker inspect` output (State, OOMKilled,
|
||||
Pid, finished/started times), `docker top` output, `docker stats
|
||||
--no-stream` for the container, and `docker events --since 30m` output.
|
||||
Each line prefixed with the container name for grepability.
|
||||
|
||||
REQ-3: Cooldown per container — write at most one diagnostics bundle
|
||||
per container id (marker file under the same dir), so a 2-minute
|
||||
healthcheck does not spam dumps on a persistent stall.
|
||||
|
||||
REQ-4: Do not kill or restart anything — diagnostics only. The job may
|
||||
recover on its own; if it does not, the captured evidence isolates
|
||||
shim-vs-daemon and stream-vs-exec for the follow-up fix.
|
||||
|
||||
## Files Affected
|
||||
|
||||
- `ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2` (extend)
|
||||
- `ansible/roles/gitea_runner/defaults/main.yml` (add `gitea_runner_stall_minutes`)
|
||||
- `docs/specs/GRM-168.md` (new)
|
||||
|
||||
## Test Plan
|
||||
- `make lint-all` passes
|
||||
- `make pytest-cov` passes
|
||||
|
||||
- `make lint-all` (ansible-lint + shellcheck-adjacent linters) passes.
|
||||
- Molecule fast-converge on the gitea_runner role scenario that deploys
|
||||
the healthcheck template (template renders without error).
|
||||
- Manual trace: the new section only touches containers matching
|
||||
`GITEA-ACTIONS-TASK-*` older than the threshold; a stalled exec probe
|
||||
writes exactly one bundle per container.
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master
|
||||
|
||||
Merge via auto-merge → GRM release → infra picks up the new version via
|
||||
the automated dependency PR. Runner hosts get the updated healthcheck on
|
||||
the next `gitea_runner` role apply (nightly or manual run).
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
Revert the template change — the healthcheck returns to the previous
|
||||
probe set. The diagnostics path is additive; removing it risks nothing.
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: Bump devx from v0.51.0 to v0.51.9 in pyproject.toml
|
||||
|
||||
- [x] Stalled job containers probed via `timeout docker exec`.
|
||||
- [x] One diagnostics bundle per stalled container, written to the
|
||||
runner config dir (readable without journal access).
|
||||
- [x] Per-container cooldown prevents dump spam.
|
||||
- [x] Nothing is killed/restarted — diagnostics only.
|
||||
- [x] `make lint-all` passes.
|
||||
|
||||
Reference in New Issue
Block a user