Add pgrep diagnostics before/after pkill to identify lingering dockerd
processes. If pkill fails and pgrep still finds dockerd, kill by PID
directly via os.kill. When inner dockerd can't be killed (still alive),
use /dev/shm as data root since the overlay is still full.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The inner dockerd started by the CI image doesn't respond to SIGTERM.
Use pkill -9 to force-kill it, then verify it's actually dead by
running docker info. If the old daemon is still alive (can't be killed),
use /dev/shm/docker.sock as an alternate socket path to avoid conflicts.
Also moved the data root cleanup after the kill verification, so we
don't delete files while the old daemon might still be writing to them.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/dev/shm is a 16G tmpfs — too small for the 505MB molecule-test-base
image. Use /tmp/docker-data on the container's overlay instead, after
killing the inner dockerd and cleaning up its data root to free ~2.4GB.
Also use the standard DOCKER_SOCK path (/var/run/docker.sock) for the
local dockerd, since the inner dockerd has been killed and the socket
is free.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When no socket has sufficient space, don't fall back to the low-space
inner dockerd (which will fail on image pulls). Instead, kill the inner
dockerd, clean up its data root to free space, and start a local
dockerd on /dev/shm with vfs storage driver.
Also adds pkill of the inner dockerd and cleanup of its data root
(overlay2, image, volumes, containers) before starting the local
dockerd, to free up the 2.4GB used by the inner dockerd's data.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The start_docker.py script was checking /var/run/docker.sock first,
which inside CI containers is an inner dockerd (v29.5.3) with data
root on the container's 38G overlay (often 100% full). When
_get_docker_free_bytes() returned 0 (data root path not accessible
from inside container), the script assumed it was the host Docker
with plenty of space and returned True immediately — without trying
the host's rootless Docker socket at /run/host-docker.sock.
Fix: try /run/host-docker.sock FIRST (before /var/run/docker.sock).
The host socket is mounted by the gitea runner config and has access
to the host's full filesystem (455G). Only trust free_bytes == 0
(= data root not accessible from container) for /run/host-docker.sock,
since the host's root dir is genuinely outside the container. For
other sockets (inner dockerd), free_bytes == 0 means the path doesn't
exist inside the container — don't trust it.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The CI container can't access the host Docker's data root
(/home/grm-ci-runner-X/.local/share/docker) to check disk space.
When free_bytes=0, the root dir is on the host filesystem (455G).
Always use host Docker in that case instead of starting a local
dockerd on the 16G /dev/shm tmpfs (which fills up with images).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The new dockerd can't create bridge network or ip6tables chains
(permission denied). Disable both — molecule tests use host network.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The new dockerd fails with 'iptables: Permission denied' because it
can't create NAT chains. Disable iptables since molecule tests don't
need network isolation.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The existing dockerd holds /var/run/docker.sock and can't be killed
from inside the container (different PID namespace). Use a new socket
path /dev/shm/docker.sock and data root /dev/shm/docker (16G tmpfs).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The CI container's pre-existing dockerd (v29.5.3) uses the default
data root on the 38G overlay (100% full). Kill it before starting
the new dockerd with --data-root /dev/shm/docker (16G tmpfs).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The CI container's overlay (38G) is often 100% full, causing the
inner DinD daemon to fail pulling images (no space left on device).
/dev/shm is a 16G tmpfs with plenty of space for molecule test images.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The gitea_runner config now mounts the rootless Docker socket at
/run/host-docker.sock. start_docker.py checks this path first,
giving CI containers access to the host's full filesystem (455 GB)
instead of the container's limited overlay (38 GB).
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When a CI container has an inner dockerd (DinD) writing to the
container's overlay (e.g. 38 GB), image pulls fail with ENOSPC.
start_docker.py now checks the daemon's free disk space and tries
rootless sockets (which have access to the host's full filesystem)
when the default socket has less than 20 GB free.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
emil
kireto
emo