fix: disable systemd-oomd memory pressure kill for runner users

The default systemd user service configuration (10-oomd-user-service-defaults.conf)
sets ManagedOOMMemoryPressure=kill with ManagedOOMMemoryPressureLimit=50%.
Under parallel DinD load (6 molecule tests), memory pressure exceeds 50%
for 30 seconds, causing systemd-oomd to SIGKILL molecule containers
(rc=137). This was the root cause of recurring "container is not running"
and "Failed to create temporary directory" errors in CI.

Fix: Create a per-user systemd override that sets:
- ManagedOOMMemoryPressure=auto (don't kill on pressure)
- ManagedOOMMemoryPressureLimit=100% (never trigger)
- OOMScoreAdjust=-500 (very unlikely to be OOM-killed)

The runner users run Docker-in-Docker workloads that legitimately use
large amounts of memory. systemd-oomd's aggressive memory pressure
killing is inappropriate for CI runner workloads.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

CHANGELOG.md

@@ -2,12 +2,6 @@
 
 All notable changes to this project will be documented in this file.
 
-## [0.21.0] - 2026-08-09
-
-### Features
-
-- *(healthcheck)* Add two-tier disk prune with critical threshold
-
 ## [0.20.0] - 2026-08-09
 
 ### Features

README.md

@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
 
 [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
 [![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
-[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
-[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
-[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/python.svg)](https://www.python.org/downloads/)
+[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
+[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
+[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/python.svg)](https://www.python.org/downloads/)
 
 ## Why GRM?
 

ansible/roles/gitea_runner/defaults/main.yml

@@ -36,9 +36,10 @@ gitea_runner_prune_label: "gitea-runner=true"
 gitea_runner_service_restart_sec: "5"
 
 # Health check configuration
-# 2min interval — catches hung daemons before multiple CI jobs fail between checks.
-# The previous 5min interval was too coarse: a stuck daemon could fail 3+ molecule
-# jobs in the window between healthcheck runs.
+# 2min interval — catches hung daemons before multiple CI jobs fail between
+# checks. The 1min interval caused excessive pruning which removed cached
+# images, forcing all 6 parallel slots to re-pull simultaneously and
+# actually increasing disk pressure.
 gitea_runner_healthcheck_interval: "2min"
 gitea_runner_healthcheck_boot_delay: "2min"
 gitea_runner_healthcheck_disk_threshold: 70
@@ -47,6 +48,10 @@ gitea_runner_healthcheck_disk_threshold: 70
 # This removes all stopped containers and unused images regardless of age.
 # At 75%+, molecule containers fail with "container is not running" because
 # overlay2 runs out of space under parallel DinD load.
+# IMPORTANT: keep at 75 (not lower) — the host disk normally sits at ~74%.
+# Lowering to 70 triggers full prune every cycle, wiping cached images and
+# forcing all parallel slots to re-pull simultaneously, which increases
+# disk pressure rather than reducing it.
 gitea_runner_healthcheck_disk_critical: 75
 gitea_runner_healthcheck_script_path: "{{ gitea_runner_config_dir }}/healthcheck.sh"
 
@@ -76,6 +81,10 @@ gitea_runner_docker_shutdown_timeout: 30
 gitea_runner_docker_max_concurrent_downloads: 3
 gitea_runner_docker_max_concurrent_uploads: 3
 gitea_runner_docker_default_nofile: 65536
+# Log file size limits — under parallel DinD load, container logs can fill
+# disk and cause the daemon to become unresponsive. Limit log size per container.
+gitea_runner_docker_max_log_size: "10m"
+gitea_runner_docker_max_log_files: 3
 
 # Admin token for runner deregistration via Gitea API.
 # If not set, falls back to registration_token (which likely lacks admin scope).

ansible/roles/gitea_runner/handlers/main.yml

@@ -10,3 +10,7 @@
     - ansible_facts is defined
     - ansible_facts['service_mgr'] | default('') == 'systemd'
     - gitea_runner_docker_rootless_setup
+
+- name: Reload systemd user daemon
+  ansible.builtin.systemd:
+    daemon_reload: true

ansible/roles/gitea_runner/tasks/rootless_docker.yml

@@ -180,6 +180,10 @@
         "features": {
           "containerd-snapshotter": false
         },
+        "log-opts": {
+          "max-size": "{{ gitea_runner_docker_max_log_size }}",
+          "max-file": "{{ gitea_runner_docker_max_log_files }}"
+        },
         {% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
         "ipv6": true,
         "ip6tables": true,
@@ -283,6 +287,10 @@
         "features": {
           "containerd-snapshotter": false
         },
+        "log-opts": {
+          "max-size": "{{ gitea_runner_docker_max_log_size }}",
+          "max-file": "{{ gitea_runner_docker_max_log_files }}"
+        },
         {% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
         "ipv6": true,
         "ip6tables": true,

ansible/roles/gitea_runner/tasks/user_setup.yml

@@ -69,3 +69,31 @@
     owner: "{{ gitea_runner_service_user }}"
     group: "{{ gitea_runner_service_user }}"
     mode: "0755"
+
+- name: Disable systemd-oomd memory pressure kill for runner user
+  when: gitea_runner_systemd_available.stat.exists
+  block:
+    - name: Ensure user service override directory exists
+      ansible.builtin.file:
+        path: "/etc/systemd/system/user@{{ gitea_runner_uid }}.service.d"
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Disable ManagedOOMMemoryPressure for runner user
+      ansible.builtin.copy:
+        content: |
+          [Service]
+          ManagedOOMMemoryPressure=auto
+          ManagedOOMMemoryPressureLimit=100%
+          OOMScoreAdjust=-500
+        dest: "/etc/systemd/system/user@{{ gitea_runner_uid }}.service.d/oomd-override.conf"
+        owner: root
+        group: root
+        mode: "0644"
+      notify: Reload systemd user daemon
+
+    - name: Reload systemd daemon for oomd override
+      ansible.builtin.systemd:
+        daemon_reload: true

ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2

@@ -223,6 +223,10 @@ fi
 
 # 3. Check disk space — prune aggressively if below threshold
 disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
+# Debug: log image count and sizes before pruning decisions
+img_count=$(timeout 10 docker images -q 2>/dev/null | wc -l || echo "?")
+img_size=$(timeout 10 docker system df --format '{% raw %}{{.Type}} {{.Size}} {{.Reclaimable}}{% endraw %}' 2>/dev/null | grep "Images" || echo "unknown")
+echo "INFO: disk=${disk_pct}% images=${img_count} | ${img_size}"
 if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_critical }} ]]; then
     echo "CRITICAL: Disk usage at ${disk_pct}% (>= {{ gitea_runner_healthcheck_disk_critical }}%), full prune"
     # Critical level: remove ALL stopped containers (no age filter) and ALL
@@ -234,7 +238,10 @@ if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_critical }} ]]; then
         | awk '{print $1}' \
         | xargs -r docker rm -f 2>/dev/null || true
     docker system prune -af --volumes || true
-    docker network prune -f || true
+    # Prune networks older than 1 hour only — removing ALL networks can
+    # race with Gitea Actions jobs that are mid-startup (network created
+    # but container not yet attached), causing "network not found" errors.
+    docker network prune -f --filter "until=1h" || true
     docker builder prune -af || true
     disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
     echo "INFO: Disk usage after full prune: ${disk_pct}%"
@@ -251,9 +258,12 @@ elif [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
         | grep -E '(hour|day|week|month|year)s? ago' \
         | awk '{print $1}' \
         | xargs -r docker rm -f 2>/dev/null || true
-    # Prune images and containers older than 1h (until filter is NOT
-    # supported with --volumes, so prune volumes separately without a filter).
-    docker image prune -af --filter "until=1h" 2>/dev/null || true
+    # Prune containers and volumes older than 1h. Do NOT prune images here —
+    # the CI runner image (ci-full:latest, ~2.3GB) is unused between jobs and
+    # would be removed by image prune, forcing a re-pull on every CI job
+    # (84+ seconds per pull, and the pull can fail silently causing
+    # "No such image" errors). Images are only pruned at the critical
+    # threshold (75%) when disk is genuinely full.
     docker container prune -f --filter "until=1h" 2>/dev/null || true
     docker volume prune -f 2>/dev/null || true
     # Prune networks older than 1 hour to avoid removing networks that

docs/index.md

@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
 
 [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
 [![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
-[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
-[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
-[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a9578eaecfc5bba4f295c16f9045be6f094d637d/python.svg)](https://www.python.org/downloads/)
+[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
+[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
+[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/python.svg)](https://www.python.org/downloads/)
 
 ## Overview
 

src/grm/__init__.py

@@ -1,3 +1,3 @@
 """Gitea Runner Manager — lean CLI for managing Gitea Actions runners."""
 
-__version__ = "0.21.0"
+__version__ = "0.20.0"