The prune service and healthcheck were force-removing ALL containers
(except GITEA-ACTIONS-TASK), including molecule test containers that
CI jobs were actively using. This caused "No such container" errors
during molecule prepare/converge phases.
Filter by RunningFor field — only remove containers showing
"hour/day/week/month/year ago", excluding "minutes/seconds ago".
This prevents killing molecule containers from running CI jobs
while still cleaning up stale containers from crashed jobs.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-09 01:07:38 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
docker network prune -f removes ALL unused networks, including ones
that molecule tests are actively creating (e.g. 'traefik' network
created during molecule create phase before containers are attached).
This caused "network traefik not found" errors in CI molecule tests.
Add --filter "until=1h" to only prune networks older than 1 hour,
matching the container prune filter.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-09 01:07:38 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The healthcheck's `journalctl --user` command fails with "No journal
files were opened due to insufficient permissions" for runner users
that lack journal access. This caused the healthcheck to always report
"OK: runner healthy" even when all runners were unregistered — the
auto-recovery never triggered.
Replace journal-based detection with a Gitea API query: read the
runner's ID from the .runner file and verify it exists in
GET /api/v1/admin/actions/runners. This works regardless of journal
permissions.
Also add scripts/cleanup_stale_runners.py for bulk cleanup of stale
runner registrations (runners that haven't been online for a
configurable threshold).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-09 01:07:38 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The prune service and healthcheck force-remove ALL containers via
"docker ps -aq | xargs -r docker rm -f", but this includes the CI
job container itself (named GITEA-ACTIONS-TASK-*). Removing it causes
"RWLayer of container is unexpectedly nil" errors and kills the
active CI job.
Exclude containers whose name starts with GITEA-ACTIONS-TASK from
the force-remove step. Use "docker ps -a --format" with name filtering
instead of "docker ps -aq".
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-09 01:07:38 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The docker-prune systemd service and healthcheck script used
"docker container prune -f" which only removes stopped containers.
Running containers from failed/interrupted molecule tests accumulated
indefinitely on runner hosts, consuming disk and memory, causing CI
test speed variance (12s → 39s for identical test suites).
Changes:
- docker-prune.service.j2: stop+rm ALL containers before system prune
- runner-healthcheck.sh.j2: force-remove all containers when disk
threshold is hit, use "system prune -af --volumes" for thorough cleanup
- template-content/verify.yml: update assertions to match new templates
The molecule_ci_guard.py in devx was also updated to run "molecule
destroy" on test failure/interruption, preventing containers from
being left running in the first place.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
gitea-admin
kireto
emo
emil