CI molecule tests on infra (OBL-INFRA-511) were failing with two Docker daemon instability patterns on the grm-managed runners:
Stuck container kill (infra job 30752): 500 Server Error: cannot kill container: tried to kill container, but did not receive an exit event
Simultaneous daemon unresponsiveness (infra run 3416): 5 of 6 molecule jobs died at the same instant with context deadline exceeded on the Docker socket
## Summary
CI molecule tests on infra (OBL-INFRA-511) were failing with two Docker daemon instability patterns on the grm-managed runners:
1. **Stuck container kill** (infra job 30752): `500 Server Error: cannot kill container: tried to kill container, but did not receive an exit event`
2. **Simultaneous daemon unresponsiveness** (infra run 3416): 5 of 6 molecule jobs died at the same instant with `context deadline exceeded` on the Docker socket
## Changes
- **daemon.json**: Add `live-restore`, `shutdown-timeout` (30s), `max-concurrent-downloads/uploads` (3), `default-ulimits` (nofile 1M)
- **healthcheck**: Add 10s timeout to `docker info`, stuck container detection (removing/stopping state) + force-cleanup, container/network prune
- **prune service**: Add container, network, and builder-cache cleanup
- **healthcheck interval**: Reduced from 5min to 2min
- **molecule template-content**: Updated verify tests for new content
## Testing
- ansible-lint pass
- molecule test -s template-content pass (converge + idempotence + verify)
Closes GRM-156
Generated with [Devin](https://devin.ai)
CI molecule tests were failing with two Docker daemon instability
patterns on the grm-managed runners:
1. Stuck container kill (infra job 30752): "500 Server Error: cannot
kill container: tried to kill container, but did not receive an exit
event" — container processes don't respond to SIGTERM within Docker's
default grace period.
2. Simultaneous daemon unresponsiveness (infra run 3416): 5 of 6
molecule jobs died at the same instant with "context deadline
exceeded" on the Docker socket — all rootless daemons became
unresponsive.
Root causes:
- daemon.json lacked resilience settings (no live-restore, no
shutdown-timeout, no concurrency limits, no ulimits)
- Healthcheck ran every 5min and only checked `docker info` — too
coarse, and `docker info` can hang indefinitely on a stuck daemon
- No stuck container detection or force-cleanup
- Prune service only ran daily and didn't clean stopped containers
or networks
Fix:
- Add live-restore, shutdown-timeout (30s), max-concurrent-downloads/
uploads (3), and default-ulimits (nofile 1M) to daemon.json
- Add timeout (10s) to healthcheck's `docker info` check
- Add stuck container detection (removing/stopping state) + force-cleanup
- Add container/network prune to healthcheck disk-space cleanup
- Add container/network/builder-cache prune to daily prune service
- Reduce healthcheck interval from 5min to 2min
- Update molecule template-content verify tests for new content
Closes GRM-156
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
emo
changed title from fix: harden rootless Docker daemon resilience on CI runners (GRM-156) to GRM-156: Harden rootless Docker daemon resilience on CI runners2026-08-05 13:36:59 +00:00
Documentation: WARNING — Ansible role changed but no docs updated
Tests: OK
Commit conventions: OK
No issues found by automated checks.
Auto-merge: If all CI checks pass, this PR will be merged automatically.
## Automated PR Review
- Architecture compliance: OK
- Best practices: OK
- Security: OK
- i18n: OK
- Resource management: OK
- Documentation: WARNING — Ansible role changed but no docs updated
- Tests: OK
- Commit conventions: OK
No issues found by automated checks.
---
**Auto-merge:** If all CI checks pass, this PR will be merged automatically.
emil
approved these changes 2026-08-05 13:50:31 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
emo
emil
Summary
CI molecule tests on infra (OBL-INFRA-511) were failing with two Docker daemon instability patterns on the grm-managed runners:
500 Server Error: cannot kill container: tried to kill container, but did not receive an exit eventcontext deadline exceededon the Docker socketChanges
live-restore,shutdown-timeout(30s),max-concurrent-downloads/uploads(3),default-ulimits(nofile 1M)docker info, stuck container detection (removing/stopping state) + force-cleanup, container/network pruneTesting
Closes GRM-156
Generated with Devin
fix: harden rootless Docker daemon resilience on CI runners (GRM-156)to GRM-156: Harden rootless Docker daemon resilience on CI runnersAutomated PR Review
No issues found by automated checks.
Auto-merge: If all CI checks pass, this PR will be merged automatically.
Auto-approved: all CI checks passed (validate, molecule-tests).