The default systemd user service configuration (10-oomd-user-service-defaults.conf)
sets ManagedOOMMemoryPressure=kill with ManagedOOMMemoryPressureLimit=50%.
Under parallel DinD load (6 molecule tests), memory pressure exceeds 50%
for 30 seconds, causing systemd-oomd to SIGKILL molecule containers
(rc=137). This was the root cause of recurring "container is not running"
and "Failed to create temporary directory" errors in CI.
Fix: Create a per-user systemd override that sets:
- ManagedOOMMemoryPressure=auto (don't kill on pressure)
- ManagedOOMMemoryPressureLimit=100% (never trigger)
- OOMScoreAdjust=-500 (very unlikely to be OOM-killed)
The runner users run Docker-in-Docker workloads that legitimately use
large amounts of memory. systemd-oomd's aggressive memory pressure
killing is inappropriate for CI runner workloads.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 23:09:46 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Logs image count and docker system df output at each healthcheck run
to diagnose "No such image" failures. When the gentle prune removes
images, the log shows the image count dropping to 0, making the root
cause immediately visible instead of requiring forensics.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 19:11:11 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The gentle prune (70% disk threshold) was running
docker image prune -af --filter "until=1h" every 2 minutes.
This removed the CI runner image (ci-full:latest, ~2.3GB)
between jobs, causing "No such image" errors when act_runner
tried to create job containers with forcePull=false.
The host disk normally sits at 72-74%, above the 70% warn
threshold, so the gentle prune was running constantly. With
the CI image unused between jobs, it was pruned after 1 hour.
Fix: remove docker image prune from the gentle path. Only
prune containers, volumes, and networks at 70%. Images are
only pruned at the 75% critical threshold when disk is
genuinely full.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 18:50:41 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The 1min interval + 70% critical threshold caused more problems than
it solved. The host disk normally sits at ~74% — just under the
original 75% critical threshold. Lowering to 70% triggered aggressive
docker system prune -af --volumes every single minute, which:
1. Wiped all cached images, forcing all 6 parallel slots to re-pull
2-3GB each simultaneously — increasing disk pressure, not reducing it
2. Caused network prune to remove networks mid-job → "network not found"
errors in Gitea Actions
Reverting to 2min/70%warn/75%critical. Images stay cached, only gentle
until=1h pruning runs at 70%, and full prune only triggers at 75%
(above the normal 74% baseline).
The network prune --filter "until=1h" fix from the previous commit is
kept — that was a genuine bug fix.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 11:54:41 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The critical disk prune path was doing `docker network prune -f`
without a filter, which removes ALL unused networks. This races with
Gitea Actions jobs that are mid-startup: the runner creates a network,
then the healthcheck prunes it before the container attaches, causing
"network not found" errors.
Use `--filter "until=1h"` (same as the warn path) to only remove
networks older than 1 hour.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 10:34:45 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
userland-proxy: false requires bridge-nf-call-iptables which is not
available in rootless Docker. This caused dockerd to fail with:
"cannot restrict inter-container communication or run without the
userland proxy: stat /proc/sys/net/bridge/bridge-nf-call-iptables:
no such file or directory"
Keep log-opts (max-size=10m, max-file="3") which works correctly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 09:43:35 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Docker's daemon.json parser expects max-file as a string, not an
integer. The unquoted Jinja template rendered "max-file": 3 which
caused dockerd to fail with "cannot unmarshal number into Go struct
field LogConfig.log-opts of type string".
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 09:37:36 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Runners are consistently running out of disk during parallel molecule
tests. The 2min healthcheck interval is too coarse — a single molecule
scenario can fill 10+ GB in 2 minutes. Reduce to 1min and lower disk
thresholds (60% warn, 70% critical) so pruning kicks in earlier.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 09:04:24 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add userland-proxy=false and log-opts (10m, 3 files) to the rootless
Docker daemon.json. The userland proxy adds overhead under parallel
container creation and can become a bottleneck during CI molecule
tests. Log rotation prevents disk exhaustion from container logs under
heavy DinD load, which caused the daemon to become unresponsive
(context deadline exceeded on /_ping).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 01:19:13 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Gitea Actions skips dependent jobs of skipped jobs without evaluating
if: conditions. When molecule-tests is skipped (ansible-changed=false),
auto-merge was also skipped because it had molecule-tests in needs.
Remove molecule-tests from needs and rely on validate success only.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
At 76%, molecule containers fail with "container is not running" because
overlay2 runs out of space under parallel DinD load. The gentle prune
(until=1h) at 75% was insufficient — runners stayed at 76% because
nothing was older than 1h on busy runners.
Lower the critical threshold to 75% so the full prune (no until filter)
triggers immediately when disk reaches the level where tests fail. Also
lower the gentle prune threshold to 70% to start cleanup earlier.
This matches the infra CI disk gate which skips tests at 75%+.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 00:43:31 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The healthcheck previously used a single 75% threshold with an
until=1h filter. On busy runners with constant CI jobs, nothing
is older than 1h, so the prune barely frees anything — disk stays
at ~76% and the CI disk gate (80%) skips all molecule tests.
Two-tier approach:
- 75-79%: gentle prune (until=1h) — same as before
- 80%+: full prune (no until filter) — removes ALL stopped
containers and unused images regardless of age, plus
builder cache and networks
This ensures that when disk pressure is critical, the healthcheck
actually reclaims space instead of skipping everything that's
recently created.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 00:43:31 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Molecule doesn't support --cwd flag. Use pushd/popd to change
directory before running molecule test/destroy.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-10 00:43:20 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The cross-runner cancellation guard was killing healthy molecule slots
when unhealthy slots failed, defeating fail-fast: false. Replace with
direct molecule test execution.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
emil
kireto
emo
gitea-admin