fix: disable systemd-oomd memory pressure kill for runner users

The default systemd user service configuration (10-oomd-user-service-defaults.conf) sets ManagedOOMMemoryPressure=kill with ManagedOOMMemoryPressureLimit=50%. Under parallel DinD load (6 molecule tests), memory pressure exceeds 50% for 30 seconds, causing systemd-oomd to SIGKILL molecule containers (rc=137). This was the root cause of recurring "container is not running" and "Failed to create temporary directory" errors in CI. Fix: Create a per-user systemd override that sets: - ManagedOOMMemoryPressure=auto (don't kill on pressure) - ManagedOOMMemoryPressureLimit=100% (never trigger) - OOMScoreAdjust=-500 (very unlikely to be OOM-killed) The runner users run Docker-in-Docker workloads that legitimately use large amounts of memory. systemd-oomd's aggressive memory pressure killing is inappropriate for CI runner workloads. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
feat: add debug logging to healthcheck (image count, disk, docker df)
2026-08-10 23:09:46 +02:00 · 2026-08-10 19:11:11 +02:00 · 2026-08-10 18:50:41 +02:00 · 2026-08-10 11:54:41 +02:00 · 2026-08-10 10:34:45 +02:00 · 2026-08-10 09:43:35 +02:00
52 changed files with 1542 additions and 150 deletions
@@ -160,10 +160,10 @@ jobs:
    container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
    timeout-minutes: 15
    strategy:
-      fail-fast: true
-      max-parallel: 6
+      fail-fast: false
+      max-parallel: 4
      matrix:
-        runner-index: [1, 2, 3, 4, 5, 6]
+        runner-index: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - name: Set up environment
@@ -178,25 +178,34 @@ jobs:
      - name: Discover assigned test pairs
        env:
          RUNNER_INDEX: ${{ matrix.runner-index }}
-          MAX_RUNNERS: 6
+          MAX_RUNNERS: 4
        run: |
          . .venv/bin/activate 2>/dev/null || true
          python3 -m devx.molecule.distribute_molecule \
            --runner-index "$RUNNER_INDEX" \
            --max-runners "$MAX_RUNNERS" \
            --github-env
-      - name: Run molecule tests
+      - name: Prune stale Docker data
+        id: prune
        if: env.SKIP != 'true'
+        run: |
+          docker system prune -af --volumes 2>/dev/null || true
+          disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
+          echo "Disk usage after prune: ${disk_pct}%"
+          if [ "$disk_pct" -ge 85 ]; then
+            echo "should-run=false" >> "$GITHUB_OUTPUT"
+            echo "::warning::Disk usage at ${disk_pct}% after prune — skipping molecule tests to avoid ENOSPC failures"
+          else
+            echo "should-run=true" >> "$GITHUB_OUTPUT"
+          fi
+      - name: Run molecule tests
+        if: env.SKIP != 'true' && steps.prune.outputs.should-run != 'false'
+        shell: bash
        env:
-          GITEA_URL: ${{ github.server_url }}
          CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
          CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
-          RUN_ID: ${{ github.run_id }}
-          ANSIBLE_INJECT_INVOCATION: "1"
-          JOB_NAME: ${{ github.job }}
-          MATRIX_INDEX: ${{ matrix.runner-index }}
-          GITEA_REPOSITORY: ${{ github.repository }}
          DOCKER_HOST: unix:///var/run/docker.sock
+          ANSIBLE_INJECT_INVOCATION: "1"
        run: |
          . .venv/bin/activate 2>/dev/null || true
          if [ -z "$TEST_PAIRS" ]; then exit 0; fi
@@ -207,19 +216,55 @@ jobs:
          _TOKEN="$CI_GITEA_API_TOKEN"; [ -z "$_TOKEN" ] && _TOKEN="$CI_GITEA_TOKEN"
          [ -z "$_TOKEN" ] && { echo "Gitea API token not set — skipping Docker login"; exit 0; }
          echo "$_TOKEN" | docker login git.oblachno.oblachno.fyi -u "$CI_GITEA_USERNAME" --password-stdin
-          # shellcheck disable=SC2086 # intentional word splitting for argument expansion
-          python3 -m devx.molecule.molecule_ci_guard $TEST_PAIRS
+          # Run each molecule test pair sequentially.
+          # Pairs are 4-part: scenario|platform_name|platform_image|platform_command
+          # Spaces in platform_command are encoded as __SPACE__.
+          role_dir="ansible/roles/gitea_runner"
+          # shellcheck disable=SC2086 # intentional word splitting for pair list
+          for pair in $TEST_PAIRS; do
+            IFS='|' read -r scenario platform_name platform_image platform_command <<< "$pair"
+            platform_command="${platform_command//__SPACE__/ }"
+            export MOLECULE_PLATFORM_NAME="$platform_name"
+            export MOLECULE_PLATFORM_IMAGE="$platform_image"
+            if [ -n "$platform_command" ]; then
+              export MOLECULE_PLATFORM_COMMAND="$platform_command"
+            else
+              unset MOLECULE_PLATFORM_COMMAND
+            fi
+            export ANSIBLE_ALLOW_BROKEN_CONDITIONALS=true
+            echo "--- Running: $scenario on $platform_name ---"
+            pushd "$role_dir" >/dev/null
+            if [ "$scenario" = "default" ]; then
+              molecule test || {
+                echo "FAILED: $pair — running molecule destroy"
+                molecule destroy 2>/dev/null || true
+                popd >/dev/null
+                exit 1
+              }
+            else
+              molecule test -s "$scenario" || {
+                echo "FAILED: $pair — running molecule destroy"
+                molecule destroy -s "$scenario" 2>/dev/null || true
+                popd >/dev/null
+                exit 1
+              }
+            fi
+            popd >/dev/null
+            echo "PASSED: $pair"
+            docker system prune -af --volumes 2>/dev/null || true
+          done
+          echo "All molecule tests passed."

  auto-merge:
-    # Auto-merge runs after validate + molecule-tests pass (or molecule is skipped).
-    # Uses always() so it evaluates even when molecule-tests is skipped
-    # (Gitea Actions skips dependent jobs of skipped jobs by default).
-    needs: [validate, molecule-tests]
+    # Auto-merge runs after validate passes. molecule-tests is NOT in needs
+    # because Gitea Actions skips dependent jobs of skipped jobs without
+    # evaluating if: conditions — having molecule-tests in needs would
+    # cascade the skip to auto-merge when ansible-changed=false.
+    needs: [validate]
    if: >-
      always() &&
      github.event_name == 'pull_request' &&
-      needs.validate.result == 'success' &&
-      (needs.molecule-tests.result == 'success' || needs.molecule-tests.result == 'skipped')
+      needs.validate.result == 'success'
    runs-on: docker
    container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
    timeout-minutes: 10
@@ -96,3 +96,43 @@ repos:
        types: [python]
        pass_filenames: false
        stages: [pre-push]
+
+      - id: check-ansible-no-log
+        name: ansible no_log on secret tasks
+        entry: make check-ansible-no-log
+        language: system
+        files: ^ansible/.*\.(yml|yaml)$
+        pass_filenames: false
+        stages: [pre-commit]
+
+      - id: check-ansible-no-state-absent-on-db
+        name: no state absent on DB paths
+        entry: make check-ansible-no-state-absent-on-db
+        language: system
+        files: ^ansible/.*\.(yml|yaml)$
+        pass_filenames: false
+        stages: [pre-commit]
+
+      - id: check-ansible-patterns
+        name: ansible failure-masking patterns
+        entry: make check-ansible-patterns
+        language: system
+        files: ^ansible/.*\.(yml|yaml)$
+        pass_filenames: false
+        stages: [pre-commit]
+
+      - id: check-jinja-expr
+        name: jinja2 expression validation
+        entry: make check-jinja-expr
+        language: system
+        files: ^ansible/.*\.(yml|yaml|j2)$
+        pass_filenames: false
+        stages: [pre-commit]
+
+      - id: check-ansible-set-fact-to-json
+        name: set_fact to_json misuse check
+        entry: make check-ansible-set-fact-to-json
+        language: system
+        files: ^ansible/.*\.(yml|yaml)$
+        pass_filenames: false
+        stages: [pre-commit]
@@ -0,0 +1,12 @@
+extends: existence
+message: "Don't attribute human qualities to software or hardware ('%s')."
+link: https://developers.google.com/style/anthropomorphism
+level: suggestion
+ignorecase: true
+# Limited to the two verbs the guide itself names. Broader lists (wants, knows,
+# thinks) can't tell a software subject from a human one: on a 950-file corpus
+# they produced 8 false positives ('the customer wants', 'your audience knows')
+# for every 2 real ones.
+tokens:
+  - sees
+  - tells
@@ -1,8 +1,13 @@
 extends: existence
 message: "'%s' should be in lowercase."
 link: 'https://developers.google.com/style/colons'
-nonword: true
 level: warning
 scope: sentence
+# The match is the word itself, not ': X', and `nonword` is off. Both are
+# required for a project Vocab to work: Vale compares accept.txt entries
+# against the matched text, and `nonword: true` opts out of that entirely.
+# So a proper noun after a colon can be exempted by adding it to accept.txt.
+# The guide's other exemption, notice labels, is handled by the lookbehinds;
+# headings are already excluded by `scope: sentence`. See issue #20.
 tokens:
-  - '(?<!:[^ ]+?):\s[A-Z]'
+  - '(?<!Note: )(?<!Caution: )(?<!Warning: )(?<!Success: )(?<=:\s)[A-Z]\w+'
@@ -6,4 +6,4 @@ level: error
 nonword: true
 tokens:
  - '\d{1,2}(?:\.|/)\d{1,2}(?:\.|/)\d{4}'
-  - '\d{1,2} (?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)|May|Jun(?:e)|Jul(?:y)|Aug(?:ust)|Sep(?:tember)?|Oct(?:ober)|Nov(?:ember)?|Dec(?:ember)?) \d{4}'
+  - '\d{1,2} (?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?) \d{4}'
@@ -0,0 +1,14 @@
+extends: existence
+message: "Avoid the unverifiable claim '%s'."
+link: https://developers.google.com/style/excessive-claims
+level: suggestion
+ignorecase: true
+# The guide also names 'never', 'always', and 'ensure', but in technical writing
+# those are usually legitimate instructions ('never commit secrets') rather than
+# product claims: they accounted for 125 of 142 hits on a 950-file corpus.
+# 'best practices' is a fixed term, not a superlative.
+tokens:
+  - 'best(?! practices?)'
+  - simplest
+  - fastest
+  - guarantees?
@@ -3,11 +3,13 @@ message: "Avoid first-person pronouns such as '%s'."
 link: 'https://developers.google.com/style/pronouns#personal-pronouns'
 ignorecase: true
 level: warning
-nonword: true
+# The 'I' tokens use lookaround rather than consuming the surrounding
+# whitespace. Matching ' I ' made the alert span cover both spaces, which shows
+# up as a too-wide underline in editors, and read as "such as ' I '". Dropping
+# `nonword` also lets a project Vocab apply, which it can't when set. See PR #50.
 tokens:
-  - (?:^|\s)I\s
-  - (?:^|\s)I,\s
-  - \bI'm\b
+  - '(?<=^|\s)I(?=[\s,])'
+  - "\\bI'm\\b"
  - \bme\b
  - \bmy\b
  - \bmine\b
@@ -4,8 +4,11 @@ link: "https://developers.google.com/style/capitalization#capitalization-in-titl
 level: warning
 scope: heading
 match: $sentence
-indicators:
-  - ":"
+# No `indicators: [":"]` here. That makes Vale require a capital after a colon,
+# which is the Microsoft convention this rule was originally copied from. This
+# guide says the opposite: "the first word after a colon is generally
+# lowercase" (developers.google.com/style/colons), and Colons.yml enforces
+# exactly that. See issue #58.
 exceptions:
  - Azure
  - CLI
@@ -0,0 +1,13 @@
+extends: existence
+message: "Avoid the jargon '%s'."
+link: https://developers.google.com/style/jargon
+level: suggestion
+ignorecase: true
+# The guide also cites 'solution', 'support', and 'workload' as overloaded
+# terms, but those have ordinary technical meanings and accounted for every hit
+# on a 950-file corpus, so only the unambiguous figurative terms are listed.
+tokens:
+  - break-glass
+  - camel ?case
+  - out-of-the-box
+  - swim ?lane
@@ -6,6 +6,10 @@ level: error
 nonword: true
 action:
  name: replace
+# The delimiter is a lookahead so the replacement doesn't swallow the comma or
+# space that follows (issue #18). `$` is included so the abbreviation is still
+# caught at the end of a heading, table cell, or block, which accounted for 8
+# of 10 occurrences on a 950-file corpus.
 swap:
-  '\b(?:eg|e\.g\.)(?=[\s,;])': for example
-  '\b(?:ie|i\.e\.)(?=[\s,;])': that is
+  '\b(?:eg|e\.g\.)(?=[\s,;]|$)': for example
+  '\b(?:ie|i\.e\.)(?=[\s,;]|$)': that is
@@ -3,5 +3,26 @@ message: "Use the Oxford comma in '%s'."
 link: 'https://developers.google.com/style/commas'
 scope: sentence
 level: warning
+nonword: true
+# List items may be several words long, not just one. Four guards keep the
+# false-positive rate down:
+#
+#   1. The comma can't be the one closing a fronted subordinate clause
+#      ('When your alarm rings, you turn it off and tumble out of bed.') --
+#      that comma separates clauses, not list items. Only the first comma of
+#      such a sentence is exempt, so 'When it rains, apples, pears or bananas
+#      get wet.' is still caught.
+#   2. The item can't open with a clause-introducer (', which ...',
+#      ', specifically ...').
+#   3. The item can't open with a subject pronoun followed by a verb, which
+#      marks a compound predicate rather than a list ('..., you walk to the
+#      fridge and get a snack.'). A pronoun directly followed by 'and'/'or'
+#      is a real list item, so ', you and me.' still matches.
+#   4. Neither item may contain an auxiliary verb, which is another compound
+#      predicate signal (', it has some downsides and is officially
+#      discouraged.').
+#
+# The trailing anchor allows end-of-scope so list fragments ('Apples, pears
+# or bananas') are still caught.
 tokens:
-  - '(?:[^,]+,){1,}\s\w+\s(?:and|or)'
+  - '(?<!^(?i:when|whenever|while|if|unless|until|although|though|because|since|after|before|once|whereas|whether|as)\b[^,]{0,80}),\s(?!(?:which|who|whom|whose|that|where|when|while|because|since|although|though|if|unless|so|but|and|or|however|therefore|thus|specifically|especially|namely|then|take|see|note|consider|make|use|either|neither)\b)(?!(?i:i|you|we|they|he|she|it)\s+(?!(?:and|or)\b))(?:(?!\b(?:is|are|was|were|has|have|had|be|been|being|will|would|can|could|should|may|might|must|do|does|did)\b)\w+ ){0,4}\w+ (?:and|or) (?:(?!\b(?:is|are|was|were|has|have|had|be|been|being|will|would|can|could|should|may|might|must|do|does|did)\b)\w+ ){0,4}\w+(?:[.?!]|$)'
@@ -3,5 +3,13 @@ message: "Use parentheses judiciously."
 link: 'https://developers.google.com/style/parentheses'
 nonword: true
 level: suggestion
+# `[^)]` rather than `.+`: a greedy match ran from the first '(' on a line to
+# the last ')', so 'Text (one) and more (two).' produced a single alert
+# covering everything between them. See issue #30.
+# A bare 3-5 letter acronym is skipped: Acronyms.yml requires acronyms to be
+# defined as 'Spelled Out Term (ACRONYM)', so flagging those parentheses would
+# put the two rules in direct conflict. The acronym has to be the whole
+# parenthetical — '(NASA rocket program)' is an ordinary aside and still
+# flags. Length matches the {3,5} in Acronyms.yml. See PR #59.
 tokens:
-  - '\(.+\)'
+  - '\((?![A-Z]{3,5}\))[^)]+\)'
@@ -0,0 +1,13 @@
+extends: existence
+message: "Avoid time-based words like '%s' in product documentation."
+link: https://developers.google.com/style/timeless-documentation
+level: suggestion
+ignorecase: true
+# The guide also names 'now' and 'new', but both have common senses that aren't
+# time-anchored ('create a new project'): adding them took a 950-file corpus of
+# technical documentation from 14 hits to 117. 'recently' is left out too — every
+# hit in that corpus was the UI idiom 'recently used'.
+tokens:
+  - currently
+  - latest
+  - soon
@@ -4,5 +4,7 @@ link: "https://developers.google.com/style/units-of-measure"
 nonword: true
 level: error
 tokens:
-  - \b\d+(?:B|kB|MB|GB|TB)
-  - \b\d+(?:ns|ms|s|min|h|d)
+  - '\b\d+(?:B|kB|MB|GB|TB)\b'
+  - '\b\d+(?:ns|ms|min|h|d)\b'
+  # Seconds are split out so a decade ('1990s') isn't read as a unit.
+  - '\b\d+s\b(?<!\b(?:19|20)\d\ds\b)'
@@ -2,79 +2,28 @@ extends: substitution
 message: "Use '%s' instead of '%s'."
 link: "https://developers.google.com/style/word-list"
 level: warning
+# Case matters here: each key's own capitalization is what's being corrected,
+# so ignorecase would make these match their own replacements. The rest of the
+# word list lives in WordListCase.yml.
 ignorecase: false
 action:
  name: replace
 swap:
-  "(?:API Console|dev|developer) key": API key
-  "(?:cell ?phone|smart ?phone)": phone|mobile phone
-  "(?:dev|developer|APIs) console": API console
-  "(?:e-mail|Email|E-mail)": email
-  "(?:file ?path|path ?name)": path
-  "(?:kill|terminate|abort)": stop|exit|cancel|end
-  "(?:OAuth ?2|Oauth)": OAuth 2.0
-  "(?:ok|Okay)": OK|okay
-  "(?:WiFi|wifi)": Wi-Fi
-  '[\.]+apk': APK
-  '3\-D': 3D
-  'Google (?:I\-O|IO)': Google I/O
-  "tap (?:&|and) hold": touch & hold
-  "un(?:check|select)": clear
-  above: preceding
-  account name: username
-  action bar: app bar
-  admin: administrator
  Ajax: AJAX
-  a\.k\.a|aka: or|also known as
  Android device: Android-powered device
  android: Android
  API explorer: APIs Explorer
-  application: app
-  approx\.: approximately
  authN: authentication
  authZ: authorization
-  autoupdate: automatically update
-  cellular data: mobile data
-  cellular network: mobile network
-  chapter: documents|pages|sections
-  check box: checkbox
  CLI: command-line tool
-  click on: click|click in
  Cloud: Google Cloud Platform|GCP
  Container Engine: Kubernetes Engine
-  content type: media type
-  curated roles: predefined roles
-  data are: data is
  Developers Console: Google API Console|API Console
-  disabled?: turn off|off
-  ephemeral IP address: ephemeral external IP address
-  fewer data: less data
-  file name: filename
-  firewalls: firewall rules
-  functionality: capability|feature
  Google account: Google Account
  Google accounts: Google Accounts
  Googling: search with Google
-  grayed-out: unavailable
  HTTPs: HTTPS
-  in order to: to
-  ingest: import|load
  k8s: Kubernetes
-  long press: touch & hold
-  network IP address: internal IP address
-  omnibox: address bar
-  open-source: open source
-  overview screen: recents screen
-  regex: regular expression
  SHA1: SHA-1|HAS-SHA1
-  sign into: sign in to
-  sign-?on: single sign-on
-  static IP address: static external IP address
-  stylesheet: style sheet
-  synch: sync
-  tablename: table name
-  tablet: device
-  touch: tap
  url: URL
-  vs\.: versus
  World Wide Web: web
@@ -0,0 +1,68 @@
+extends: substitution
+message: "Use '%s' instead of '%s'."
+link: "https://developers.google.com/style/word-list"
+level: warning
+# The case-insensitive half of the word list, so sentence-initial use is caught
+# ('Touch the screen', not only 'touch the screen'). Entries that must stay
+# case-sensitive are in WordList.yml.
+ignorecase: true
+action:
+  name: replace
+swap:
+  "(?:API Console|dev|developer) key": API key
+  "(?:cell ?phone|smart ?phone)": phone|mobile phone
+  "(?:dev|developer|APIs) console": API console
+  "(?:e-mail|Email|E-mail)": email
+  "(?:file ?path|path ?name)": path
+  "(?:kill|terminate|abort)": stop|exit|cancel|end
+  # Longest form first: with the shortest alternative leading, 'OAuth 2' matched
+  # only 'OAuth', so applying the suggestion produced 'OAuth 2.0 2'. The rule is
+  # already case-insensitive, so the inline (?i) is redundant. See issue #41.
+  '\bOauth2\.0\b|\bOAuth ?2\b(?!\.0)|\bOauth\b(?! ?2)': OAuth 2.0
+  "(?:ok|Okay)": OK|okay
+  "(?:WiFi|wifi)": Wi-Fi
+  '[\.]+apk': APK
+  '3\-D': 3D
+  'Google (?:I\-O|IO)': Google I/O
+  "tap (?:&|and) hold": touch & hold
+  "un(?:check|select)": clear
+  above: preceding
+  account name: username
+  action bar: app bar
+  admin: administrator
+  a\.k\.a|aka: or|also known as
+  application: app
+  approx\.: approximately
+  autoupdate: automatically update
+  cellular data: mobile data
+  cellular network: mobile network
+  chapter: documents|pages|sections
+  check box: checkbox
+  click on: click|click in
+  content type: media type
+  curated roles: predefined roles
+  data are: data is
+  disabled?: turn off|off
+  ephemeral IP address: ephemeral external IP address
+  fewer data: less data
+  file name: filename
+  firewalls: firewall rules
+  functionality: capability|feature
+  grayed-out: unavailable
+  in order to: to
+  ingest: import|load
+  long press: touch & hold
+  network IP address: internal IP address
+  omnibox: address bar
+  open-source: open source
+  overview screen: recents screen
+  regex: regular expression
+  sign into: sign in to
+  '(?<!single )sign-?on': single sign-on
+  static IP address: static external IP address
+  stylesheet: style sheet
+  synch: sync
+  tablename: table name
+  tablet: device
+  'touch(?! ?(?:&|and) hold)': tap
+  vs\.: versus
@@ -278,9 +278,9 @@ via `[tool.devx.classify]` in `pyproject.toml`.
 - Any new file type not in the allowlist

 **devx module structure** (installed from git, not in this repo):
- `devx.ci.*` — CI/CD automation (run by workflows): release, publish, auto_merge, classify_changes, detect_release_commit, push_badges, doc_coverage, sync_wiki, distribute_molecule, molecule_ci_guard, discover_runners, notify_failure, post_merge, pr_review, validate_commit_msg
+- `devx.ci.*` — CI/CD automation (run by workflows): release, publish, auto_merge, classify_changes, detect_release_commit, push_badges, doc_coverage, sync_wiki, distribute_molecule, discover_runners, notify_failure, post_merge, pr_review, validate_commit_msg
 - `devx.tools.*` — Dev tools (run locally): check_test_speed, configure_repo, install_checkmake, install_tools, setup, generate_badges, create_task, create_pr, pr_status, pr_logs, pr_label, rebase, pr_rebase
- `devx.molecule.*` — Molecule helpers: molecule_all, platforms, discover_runners, distribute_molecule, molecule_ci_guard
+- `devx.molecule.*` — Molecule helpers: molecule_all, platforms, discover_runners, distribute_molecule
 - `devx.gitea_cli` — Tea CLI wrapper
 - `devx.i18n` — i18n translation system
 - `devx.config` — Shared configuration (DEVX_* env vars)
@@ -348,7 +348,7 @@ Since devx is installed as a package (via `pip install` from git), it is importa
 | PYTHONPATH | When to use | Example modules |
 |------------|-------------|-----------------|
 | `src` | Module imports from `grm` | `devx.ci.auto_merge`, `devx.ci.pr_review`, `devx.ci.pr_review`, `devx.ci.sync_wiki`, `devx.ci.post_merge`, `devx.ci.classify_changes`, `devx.molecule.discover_runners`, `devx.ci.doc_coverage` |
-| (none) | Module has no GRM imports | `devx.ci.detect_release_commit`, `devx.molecule.distribute_molecule`, `devx.molecule.molecule_ci_guard`, `devx.ci.push_badges`, `devx.ci.validate_commit_msg` |
+| (none) | Module has no GRM imports | `devx.ci.detect_release_commit`, `devx.molecule.distribute_molecule`, `devx.ci.push_badges`, `devx.ci.validate_commit_msg` |

 **In workflows**, always use `env:` blocks (not inline `PYTHONPATH=value`):
 ```yaml
@@ -2,6 +2,55 @@

 All notable changes to this project will be documented in this file.

+## [0.20.0] - 2026-08-09
+
+### Features
+
+- *(healthcheck)* Add two-tier disk prune with critical threshold
+
+## [0.19.0] - 2026-08-08
+
+### Features
+
+- Use Gitea mirror for Ansible collection installs
+
+## [0.18.8] - 2026-08-06
+
+### Bug Fixes
+
+- Pin containerd.io to compatible version for Docker 28.x
+
+
+## [0.18.7] - 2026-08-06
+
+### Bug Fixes
+
+- Move StartLimit to [Unit] and make prune timer reload conditional
+
+## [0.18.6] - 2026-08-05
+
+### Bug Fixes
+
+- Pre-configure daemon.json before rootless setuptool + add DBUS_SESSION_BUS_ADDRESS
+
+## [0.18.5] - 2026-08-05
+
+### Bug Fixes
+
+- Pin Docker 28.x + disable containerd snapshotter + tune prune/disk
+
+## [0.18.4] - 2026-08-05
+
+### Bug Fixes
+
+- Harden rootless Docker daemon resilience on CI runners
+
+## [0.18.3] - 2026-08-04
+
+### Bug Fixes
+
+- Switch default network driver to slirp4netns (pasta TCP RST bug)
+
 ## [0.18.2] - 2026-07-16

 ### Bug Fixes
@@ -88,7 +88,7 @@ setup-release: $(VENV)/bin/activate .env configure-gitea-pypi
 setup-image:
 	@if [ -d /opt/venv ]; then ln -sf /opt/venv .venv; . .venv/bin/activate; \
 		_TOKEN="$$CI_GITEA_API_TOKEN"; [ -z "$$_TOKEN" ] && _TOKEN="$$DEVELOPER_GITEA_API_TOKEN"; [ -z "$$_TOKEN" ] && _TOKEN="$$CI_GITEA_TOKEN"; \
-		if [ -n "$$_TOKEN" ]; then export PIP_EXTRA_INDEX_URL="https://$$CI_GITEA_USERNAME:$${_TOKEN}@git.oblachno.oblachno.fyi/api/packages/oblachno-oss/pypi/simple/"; fi; \
+		if [ -n "$$_TOKEN" ]; then export PIP_EXTRA_INDEX_URL="https://$$CI_GITEA_USERNAME:$${_TOKEN}@git.oblachno.oblachno.fyi/api/packages/oblachno-oss/pypi/simple/"; git config --global url."https://$$CI_GITEA_USERNAME:$${_TOKEN}@git.oblachno.oblachno.fyi/".insteadOf "https://git.oblachno.oblachno.fyi/"; fi; \
 		pip install -e .$(if $(EXTRAS),[$(EXTRAS)],); \
 	else echo "[setup-image] /opt/venv not found — falling back to setup-ci"; $(MAKE) setup-ci; fi

@@ -175,11 +175,36 @@ makefile-lint:
 		echo "checkmake not found, skipping Makefile lint"; \
 	fi

-lint-all: lint ansible-lint makefile-lint workflow-lint check-api-identity-checks
+lint-all: lint ansible-lint makefile-lint workflow-lint check-api-identity-checks check-ansible-no-log check-ansible-no-state-absent-on-db check-ansible-patterns check-jinja-expr check-ansible-set-fact-to-json

 check-api-identity-checks:
 	@$(BIN)/python -m devx.tools.check_api_identity_checks

+check-ansible-no-log:
+	@echo "[check-ansible-no-log] Checking Ansible tasks for missing no_log on secret-handling tasks..."
+	@$(BIN)/python -m devx.tools.check_ansible_no_log
+	@echo "[check-ansible-no-log] Passed."
+
+check-ansible-no-state-absent-on-db:
+	@echo "[check-ansible-no-state-absent-on-db] Checking for state: absent on DB data directories..."
+	@$(BIN)/python -m devx.tools.check_ansible_no_state_absent_on_db
+	@echo "[check-ansible-no-state-absent-on-db] Passed."
+
+check-ansible-patterns:
+	@echo "[check-ansible-patterns] Checking for dangerous failure-masking patterns..."
+	@$(BIN)/python -m devx.tools.check_ansible_patterns
+	@echo "[check-ansible-patterns] Passed."
+
+check-jinja-expr:
+	@echo "[check-jinja-expr] Validating Jinja2 expressions in Ansible files..."
+	@$(BIN)/python -m devx.tools.check_jinja_expr
+	@echo "[check-jinja-expr] Passed."
+
+check-ansible-set-fact-to-json:
+	@echo "[check-ansible-set-fact-to-json] Checking set_fact tasks for to_json misuse..."
+	@$(BIN)/python -m devx.tools.check_ansible_set_fact_to_json
+	@echo "[check-ansible-set-fact-to-json] Passed."
+
 test-integration:
 	$(BIN)/pytest tests/integration/ -v --no-cov

@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate

 [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
 [![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
-[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
-[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
-[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/python.svg)](https://www.python.org/downloads/)
+[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
+[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
+[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/python.svg)](https://www.python.org/downloads/)

 ## Why GRM?

@@ -1,7 +1,11 @@
+---
 collections:
  - name: community.general
-    version: "==13.1.0"
+    type: url
+    source: https://git.oblachno.oblachno.fyi/api/packages/emil/generic/ansible-collections/13.1.0/community-general-13.1.0.tar.gz
  - name: ansible.posix
-    version: "==2.2.1"
+    type: url
+    source: https://git.oblachno.oblachno.fyi/api/packages/emil/generic/ansible-collections/2.2.1/ansible-posix-2.2.1.tar.gz
  - name: community.docker
-    version: "==5.2.1"
+    type: url
+    source: https://git.oblachno.oblachno.fyi/api/packages/emil/generic/ansible-collections/5.2.1/community-docker-5.2.1.tar.gz
@@ -3,6 +3,13 @@ gitea_runner_version: "2.0.1"
 gitea_runner_labels: "docker,ubuntu-latest:docker://runner-images:ubuntu-26.04"
 gitea_runner_skip_registration: false

+# Force re-registration even if .runner file exists.
+# Use this when Gitea no longer recognizes the runner (e.g., after a Gitea
+# server restore/reinstall or when the runner record was deleted from the
+# admin UI). The existing .runner file is removed and a new registration is
+# performed. Requires registration_token.
+gitea_runner_force_reregister: false
+
 # Per-runner user (rootless isolation)
 gitea_runner_user_prefix: "grm-"
 gitea_runner_base_home: "/home"
@@ -18,18 +25,67 @@ gitea_runner_binary_path: "/usr/local/bin/gitea_runner"

 # Prune configuration
 gitea_runner_prune_until: "24h"
-gitea_runner_prune_schedule: "daily"
+# Every 6 hours — daily is insufficient for CI runners that build dozens
+# of images per day. Accumulation between daily runs can trigger Docker
+# daemon instability (containerd snapshotter GC holds locks, blocking
+# container operations).
+gitea_runner_prune_schedule: "*-*-* 00/6:00:00"
 gitea_runner_prune_label: "gitea-runner=true"

 # Service configuration
 gitea_runner_service_restart_sec: "5"

 # Health check configuration
-gitea_runner_healthcheck_interval: "5min"
+# 2min interval — catches hung daemons before multiple CI jobs fail between
+# checks. The 1min interval caused excessive pruning which removed cached
+# images, forcing all 6 parallel slots to re-pull simultaneously and
+# actually increasing disk pressure.
+gitea_runner_healthcheck_interval: "2min"
 gitea_runner_healthcheck_boot_delay: "2min"
-gitea_runner_healthcheck_disk_threshold: 85
+gitea_runner_healthcheck_disk_threshold: 70
+# When disk reaches this level, prune EVERYTHING (no until-filter) — the
+# runner is dangerously full and the gentle until=1h prune isn't enough.
+# This removes all stopped containers and unused images regardless of age.
+# At 75%+, molecule containers fail with "container is not running" because
+# overlay2 runs out of space under parallel DinD load.
+# IMPORTANT: keep at 75 (not lower) — the host disk normally sits at ~74%.
+# Lowering to 70 triggers full prune every cycle, wiping cached images and
+# forcing all parallel slots to re-pull simultaneously, which increases
+# disk pressure rather than reducing it.
+gitea_runner_healthcheck_disk_critical: 75
 gitea_runner_healthcheck_script_path: "{{ gitea_runner_config_dir }}/healthcheck.sh"

+# Auto-recovery: when the healthcheck detects an unregistered runner, it
+# can automatically re-register if a Gitea API token is provided.
+# The token needs admin or org-level access to fetch registration tokens.
+# Stored in a file readable by the runner user (mode 0400).
+# Set to empty string to disable auto-recovery (manual re-registration required).
+gitea_runner_auto_recover_api_token: ""
+
+# Cooldown file to prevent auto-recovery loops (e.g., if Gitea is down).
+# The healthcheck writes a timestamp to this file after a re-registration
+# attempt and skips further attempts for the cooldown period.
+gitea_runner_auto_recover_cooldown_sec: 300
+
+# Docker daemon resilience settings (applied to daemon.json).
+# live-restore: containers survive daemon restarts — prevents stuck container
+#   states when the healthcheck restarts a hung daemon.
+# shutdown-timeout: grace period (seconds) for containers to stop on daemon
+#   shutdown/restart. Default 15s is too short for DinD containers with nested
+#   processes (molecule tests). 30s gives SIGTERM time to propagate.
+# max-concurrent-downloads/uploads: limits parallel transfers to reduce daemon
+#   memory pressure when multiple CI jobs pull images simultaneously.
+# default-ulimits: prevents FD exhaustion in container processes.
+gitea_runner_docker_live_restore: true
+gitea_runner_docker_shutdown_timeout: 30
+gitea_runner_docker_max_concurrent_downloads: 3
+gitea_runner_docker_max_concurrent_uploads: 3
+gitea_runner_docker_default_nofile: 65536
+# Log file size limits — under parallel DinD load, container logs can fill
+# disk and cause the daemon to become unresponsive. Limit log size per container.
+gitea_runner_docker_max_log_size: "10m"
+gitea_runner_docker_max_log_files: 3
+
 # Admin token for runner deregistration via Gitea API.
 # If not set, falls back to registration_token (which likely lacks admin scope).
 # Set this to a token with admin scope to enable automatic runner cleanup on removal.
@@ -44,6 +100,15 @@ gitea_runner_log_level: "info"
 gitea_runner_container_label: "gitea-runner=true"
 gitea_runner_file: ".runner"

+# Containerd version pinning — Docker 28.x vendors containerd v2.1.x internally.
+# containerd.io >= 2.3 ships a shim that returns a protobuf BootstrapResult which
+# Docker 28.x's vendored containerd code cannot parse, causing:
+#   "failed to create TTRPC connection: unsupported protocol: \b\x03\x12Yunix"
+# When Docker 29+ is installed (it vendors containerd 2.3+), this pin is not needed.
+# Set to "" to skip the compatibility check and allow any containerd.io version.
+gitea_runner_containerd_max_compatible_major: 2
+gitea_runner_containerd_max_compatible_minor: 2
+
 # Docker installation (for rootless dependencies)
 gitea_runner_docker_gpg_key_path: "/etc/apt/keyrings/docker.gpg"
 gitea_runner_docker_apt_arch: "{{ 'amd64' if ansible_facts['architecture'] == 'x86_64' else ansible_facts['architecture'] }}"
@@ -64,9 +129,18 @@ gitea_runner_rootless_scripts_ref: "v28.5.1"
 # that dockerd-rootless-setuptool.sh (which derives BIN from its own dirname) finds them co-located.
 gitea_runner_rootless_scripts_install_dir: "/usr/bin"

-# Rootless Docker network driver: "pasta" (IPv6 support) or "slirp4netns" (IPv4 only)
-# pasta has proper outgoing IPv6 support; slirp4netns does not (known limitation).
-gitea_runner_docker_rootless_net_driver: "pasta"
+# Rootless Docker network driver: "slirp4netns" (default) or "pasta" (IPv6 support)
+# slirp4netns is the default because pasta has a TCP proxy bug that sends RST
+# packets with wrong sequence numbers, breaking TCP connections from Docker
+# containers to external hosts. slirp4netns doesn't have IPv6 support.
+# See: https://bugs.passt.top/show_bug.cgi?id=52
+gitea_runner_docker_rootless_net_driver: "slirp4netns"

 # IPv6 subnet for rootless Docker containers (ULA range, not routable on internet)
 gitea_runner_docker_ipv6_cidr: "fd00:dead:beef::/48"
+
+# Pre-pull Docker images that CI runners need (avoids pulling on every CI run).
+# The runner container image (ci-full) is large (~3.3GB) and the healthcheck's
+# disk-space prune only removes dangling images, so pre-pulled tagged images persist.
+# Set to [] to skip pre-pulling. Images are pulled as the runner user via rootless Docker.
+gitea_runner_pre_pull_images: []
@@ -10,3 +10,7 @@
    - ansible_facts is defined
    - ansible_facts['service_mgr'] | default('') == 'systemd'
    - gitea_runner_docker_rootless_setup
+
+- name: Reload systemd user daemon
+  ansible.builtin.systemd:
+    daemon_reload: true
@@ -47,8 +47,11 @@
      ansible.builtin.assert:
        that:
          - "'Type=oneshot' in prune_service.content | b64decode"
-          - "'docker system prune' in prune_service.content | b64decode"
-          - "'docker volume prune' in prune_service.content | b64decode"
+          - "'docker rm -f' in prune_service.content | b64decode"
+          - "'GITEA-ACTIONS-TASK' in prune_service.content | b64decode"
+          - "'docker system prune -af' in prune_service.content | b64decode"
+          - "'docker network prune' in prune_service.content | b64decode"
+          - "'docker builder prune' in prune_service.content | b64decode"
        fail_msg: "Prune service template is missing expected directives"

    - name: Read rendered prune timer template
@@ -99,8 +102,15 @@
      ansible.builtin.assert:
        that:
          - "'docker info' in healthcheck_script.content | b64decode"
+          - "'timeout 10 docker info' in healthcheck_script.content | b64decode"
          - "'systemctl --user restart docker.service' in healthcheck_script.content | b64decode"
          - "'systemctl --user restart gitea-runner.service' in healthcheck_script.content | b64decode"
-          - "'docker system prune' in healthcheck_script.content | b64decode"
+          - "'docker rm -f' in healthcheck_script.content | b64decode"
+          - "'GITEA-ACTIONS-TASK' in healthcheck_script.content | b64decode"
+          - "'docker system prune -af' in healthcheck_script.content | b64decode"
+          - "'docker network prune' in healthcheck_script.content | b64decode"
+          - "'status=removing' in healthcheck_script.content | b64decode"
+          - "'status=stopping' in healthcheck_script.content | b64decode"
          - "gitea_runner_healthcheck_disk_threshold | string in healthcheck_script.content | b64decode"
+          - "gitea_runner_healthcheck_disk_critical | string in healthcheck_script.content | b64decode"
        fail_msg: "Healthcheck script template is missing expected content"
@@ -29,6 +29,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid | default(0) }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
    DOCKER_HOST: "unix:///run/user/{{ gitea_runner_uid | default(0) }}/docker.sock"
  when:
    - gitea_runner_file_stat.stat.exists | default(false) | bool
@@ -7,6 +7,22 @@
    group: "{{ gitea_runner_service_user }}"
    mode: "0755"

+- name: Write auto-recovery API token file
+  ansible.builtin.copy:
+    content: "{{ gitea_runner_auto_recover_api_token }}"
+    dest: "{{ gitea_runner_config_dir }}/auto-recover.token"
+    owner: "{{ gitea_runner_service_user }}"
+    group: "{{ gitea_runner_service_user }}"
+    mode: "0400"
+  no_log: true
+  when: gitea_runner_auto_recover_api_token | length > 0
+
+- name: Remove stale auto-recovery token file (if auto-recovery disabled)
+  ansible.builtin.file:
+    path: "{{ gitea_runner_config_dir }}/auto-recover.token"
+    state: absent
+  when: gitea_runner_auto_recover_api_token | length == 0
+
 - name: Create healthcheck user service file
  ansible.builtin.template:
    src: runner-healthcheck.service.j2
@@ -29,6 +45,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  changed_when: true
  when:
    - gitea_runner_systemd_available.stat.exists
@@ -40,6 +57,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  changed_when: true
  when:
    - gitea_runner_systemd_available.stat.exists
@@ -18,14 +18,18 @@
      else {} }}
  when: gitea_runner_file_stat.stat.exists | default(false) | bool

- name: Verify runner user service active
+- name: Wait for runner user service to be active
  ansible.builtin.command: systemctl --user is-active gitea-runner
  become: true
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  register: gitea_runner_service_check
  changed_when: false
+  retries: 10
+  delay: 2
+  until: gitea_runner_service_check.stdout | default('') | trim == 'active'
  when:
    - gitea_runner_systemd_available.stat.exists
    - gitea_runner_docker_rootless_setup
@@ -17,6 +17,9 @@
 - name: Include healthcheck setup
  ansible.builtin.include_tasks: healthcheck.yml

+- name: Include pre-pull images
+  ansible.builtin.include_tasks: pre_pull_images.yml
+
 - name: Include integration test
  ansible.builtin.include_tasks: integration_test.yml
  when: not gitea_runner_skip_registration
@@ -0,0 +1,28 @@
+---
+# Pre-pull Docker images that CI runners need to avoid pulling them on
+# every CI run. The runner container image (ci-full) is large (~3.3GB)
+# and pulling it on every run causes timeouts and disk pressure.
+#
+# The healthcheck script's disk-space prune only removes dangling images
+# (not tagged ones), so pre-pulled images persist between CI runs.
+#
+# Set gitea_runner_pre_pull_images to a list of image refs to pull, or
+# empty list to skip pre-pulling.
+
+- name: Pre-pull Docker images for CI runner
+  ansible.builtin.command: "docker pull {{ item }}"
+  become: true
+  become_user: "{{ gitea_runner_service_user }}"
+  environment:
+    DOCKER_HOST: "unix:///run/user/{{ gitea_runner_uid }}/docker.sock"
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
+  register: gitea_runner_pre_pull_result
+  changed_when: "'Status: Downloaded' in gitea_runner_pre_pull_result.stdout or 'Status: Downloaded' in gitea_runner_pre_pull_result.stderr"
+  retries: 3
+  delay: 5
+  until: gitea_runner_pre_pull_result is success
+  loop: "{{ gitea_runner_pre_pull_images }}"
+  when:
+    - gitea_runner_docker_rootless_setup
+    - gitea_runner_pre_pull_images | length > 0
@@ -6,6 +6,7 @@
    owner: "{{ gitea_runner_service_user }}"
    group: "{{ gitea_runner_service_user }}"
    mode: "0644"
+  register: gitea_runner_prune_service

 - name: Create docker-prune user timer file
  ansible.builtin.template:
@@ -14,6 +15,7 @@
    owner: "{{ gitea_runner_service_user }}"
    group: "{{ gitea_runner_service_user }}"
    mode: "0644"
+  register: gitea_runner_prune_timer

 - name: Reload systemd user daemon for prune timer
  ansible.builtin.command: systemctl --user daemon-reload
@@ -21,10 +23,12 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  changed_when: true
  when:
    - gitea_runner_systemd_available.stat.exists
    - gitea_runner_docker_rootless_setup
+    - gitea_runner_prune_service is changed or gitea_runner_prune_timer is changed

 - name: Enable and start docker-prune user timer
  ansible.builtin.command: systemctl --user enable --now docker-prune.timer
@@ -32,6 +36,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  changed_when: true
  when:
    - gitea_runner_systemd_available.stat.exists
@@ -7,11 +7,20 @@
    group: "{{ gitea_runner_service_user }}"
    mode: "0755"

- name: Check if runner is already registered
+- name: Check if runner registration file exists
  ansible.builtin.stat:
    path: "{{ gitea_runner_data_dir }}/.runner"
  register: gitea_runner_registered

+- name: Remove stale runner registration file
+  ansible.builtin.file:
+    path: "{{ gitea_runner_data_dir }}/.runner"
+    state: absent
+  when:
+    - gitea_runner_registered.stat.exists
+    - gitea_runner_force_reregister | bool
+  register: gitea_runner_registration_removed
+
 - name: Register runner with Gitea
  ansible.builtin.command: >
    {{ gitea_runner_binary_path }} register
@@ -26,8 +35,25 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid | default(0) }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
    DOCKER_HOST: "unix:///run/user/{{ gitea_runner_uid | default(0) }}/docker.sock"
-  when: not gitea_runner_registered.stat.exists
+  when: not gitea_runner_registered.stat.exists or gitea_runner_force_reregister | bool
  register: gitea_runner_register_output
-  changed_when: "'already exists' not in gitea_runner_register_output.stdout | default('')"
+  changed_when: >-
+    gitea_runner_register_output.rc == 0 and
+    ('already exists' not in gitea_runner_register_output.stdout | default(''))
  timeout: 60
+
+- name: Ensure runner service is running after registration
+  ansible.builtin.command: systemctl --user start gitea-runner
+  become: true
+  become_user: "{{ gitea_runner_service_user }}"
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
+  changed_when: true
+  when:
+    - gitea_runner_systemd_available.stat.exists
+    - gitea_runner_docker_rootless_setup
+    - gitea_runner_register_output is defined
+    - gitea_runner_register_output.rc | default(1) == 0
@@ -31,6 +31,11 @@
    - ansible_facts['os_family'] == 'Debian'
    - gitea_runner_docker_apt_repo is changed

+# Install Docker packages from the upstream Docker APT repository.
+# We do NOT pin to 28.x because recent Ubuntu releases (e.g. 26.04/plucky)
+# may not have 28.x packages in the Docker repo, and Docker 29 is safe
+# for rootless mode when the daemon.json disables the containerd snapshotter
+# and sets a conservative default nofile ulimit (see daemon.json tasks below).
 - name: Install rootless Docker dependencies (Debian/Ubuntu)
  ansible.builtin.apt:
    name:
@@ -45,6 +50,7 @@
      - docker-compose-plugin
      - rsync
    state: present
+  register: gitea_runner_docker_install
  when: ansible_facts['os_family'] == 'Debian'

 - name: Update pacman cache (Arch Linux)
@@ -133,8 +139,61 @@
    content: |
      [Service]
      Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_NET={{ gitea_runner_docker_rootless_net_driver }}"
-      Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=implicit"
+      Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER={{ 'implicit' if gitea_runner_docker_rootless_net_driver == 'pasta' else 'builtin' }}"
+      {% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
      Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
+      {% endif %}
+    mode: "0644"
+    owner: "{{ gitea_runner_service_user }}"
+    group: "{{ gitea_runner_service_user }}"
+  when:
+    - gitea_runner_docker_rootless_setup
+    - not gitea_runner_rootless_docker_check.stat.exists
+
+# Write daemon.json BEFORE the setuptool starts dockerd, so Docker 29
+# starts with containerd snapshotter disabled from the very first boot.
+# Without this, Docker 29 uses containerd snapshots by default, which
+# causes instability in rootless mode.
+- name: Ensure Docker config directory exists (pre-setup)
+  ansible.builtin.file:
+    path: "{{ gitea_runner_home }}/.config/docker"
+    state: directory
+    mode: "0755"
+    owner: "{{ gitea_runner_service_user }}"
+    group: "{{ gitea_runner_service_user }}"
+  when:
+    - gitea_runner_docker_rootless_setup
+    - not gitea_runner_rootless_docker_check.stat.exists
+
+- name: Pre-configure rootless Docker daemon.json (disable containerd snapshotter)
+  ansible.builtin.copy:
+    dest: "{{ gitea_runner_home }}/.config/docker/daemon.json"
+    content: |
+      {
+        "live-restore": {{ gitea_runner_docker_live_restore | to_json }},
+        "shutdown-timeout": {{ gitea_runner_docker_shutdown_timeout }},
+        "max-concurrent-downloads": {{ gitea_runner_docker_max_concurrent_downloads }},
+        "max-concurrent-uploads": {{ gitea_runner_docker_max_concurrent_uploads }},
+        "default-ulimits": {
+          "nofile": {"Name": "nofile", "Hard": {{ gitea_runner_docker_default_nofile }}, "Soft": {{ gitea_runner_docker_default_nofile }}}
+        },
+        "features": {
+          "containerd-snapshotter": false
+        },
+        "log-opts": {
+          "max-size": "{{ gitea_runner_docker_max_log_size }}",
+          "max-file": "{{ gitea_runner_docker_max_log_files }}"
+        },
+        {% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
+        "ipv6": true,
+        "ip6tables": true,
+        "fixed-cidr-v6": "{{ gitea_runner_docker_ipv6_cidr }}",
+        "dns": ["10.0.2.3", "8.8.8.8"]
+        {% else %}
+        "ipv6": false,
+        "dns": ["8.8.8.8", "1.1.1.1"]
+        {% endif %}
+      }
    mode: "0644"
    owner: "{{ gitea_runner_service_user }}"
    group: "{{ gitea_runner_service_user }}"
@@ -150,6 +209,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
    DOCKERD_ROOTLESS_ROOTLESSKIT_NET: "{{ gitea_runner_docker_rootless_net_driver }}"
  when:
    - gitea_runner_docker_rootless_setup
@@ -161,6 +221,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
  changed_when: true
  when: gitea_runner_docker_rootless_setup

@@ -170,6 +231,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
  changed_when: true
  when: gitea_runner_docker_rootless_setup

@@ -182,14 +244,16 @@
    group: "{{ gitea_runner_service_user }}"
  when: gitea_runner_docker_rootless_setup

- name: Configure rootless Docker to use pasta with IPv6
+- name: Configure rootless Docker network driver
  ansible.builtin.copy:
    dest: "{{ gitea_runner_home }}/.config/systemd/user/docker.service.d/override.conf"
    content: |
      [Service]
      Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_NET={{ gitea_runner_docker_rootless_net_driver }}"
-      Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=implicit"
+      Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER={{ 'implicit' if gitea_runner_docker_rootless_net_driver == 'pasta' else 'builtin' }}"
+      {% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
      Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
+      {% endif %}
    mode: "0644"
    owner: "{{ gitea_runner_service_user }}"
    group: "{{ gitea_runner_service_user }}"
@@ -202,19 +266,40 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
  changed_when: true
  when:
    - gitea_runner_docker_rootless_setup
    - gitea_runner_docker_network_override is changed

- name: Configure rootless Docker daemon with IPv6 enabled
+- name: Configure rootless Docker daemon
  ansible.builtin.copy:
    dest: "{{ gitea_runner_home }}/.config/docker/daemon.json"
    content: |
      {
+        "live-restore": {{ gitea_runner_docker_live_restore | to_json }},
+        "shutdown-timeout": {{ gitea_runner_docker_shutdown_timeout }},
+        "max-concurrent-downloads": {{ gitea_runner_docker_max_concurrent_downloads }},
+        "max-concurrent-uploads": {{ gitea_runner_docker_max_concurrent_uploads }},
+        "default-ulimits": {
+          "nofile": {"Name": "nofile", "Hard": {{ gitea_runner_docker_default_nofile }}, "Soft": {{ gitea_runner_docker_default_nofile }}}
+        },
+        "features": {
+          "containerd-snapshotter": false
+        },
+        "log-opts": {
+          "max-size": "{{ gitea_runner_docker_max_log_size }}",
+          "max-file": "{{ gitea_runner_docker_max_log_files }}"
+        },
+        {% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
        "ipv6": true,
        "ip6tables": true,
-        "fixed-cidr-v6": "{{ gitea_runner_docker_ipv6_cidr }}"
+        "fixed-cidr-v6": "{{ gitea_runner_docker_ipv6_cidr }}",
+        "dns": ["10.0.2.3", "8.8.8.8"]
+        {% else %}
+        "ipv6": false,
+        "dns": ["8.8.8.8", "1.1.1.1"]
+        {% endif %}
      }
    mode: "0644"
    owner: "{{ gitea_runner_service_user }}"
@@ -228,6 +313,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
  changed_when: true
  when:
    - gitea_runner_docker_rootless_setup
@@ -240,6 +326,7 @@
  environment:
    DOCKER_HOST: "unix:///run/user/{{ gitea_runner_uid }}/docker.sock"
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
  register: gitea_runner_docker_ready
  until: gitea_runner_docker_ready.rc == 0
  retries: 10
@@ -6,6 +6,7 @@
    owner: "{{ gitea_runner_service_user }}"
    group: "{{ gitea_runner_service_user }}"
    mode: "0644"
+  register: gitea_runner_service_file

 - name: Reload systemd user daemon
  ansible.builtin.command: systemctl --user daemon-reload
@@ -13,10 +14,25 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  changed_when: true
  when:
    - gitea_runner_systemd_available.stat.exists
    - gitea_runner_docker_rootless_setup
+    - gitea_runner_service_file is changed
+
+- name: Restart gitea-runner if service file changed
+  ansible.builtin.command: systemctl --user restart gitea-runner
+  become: true
+  become_user: "{{ gitea_runner_service_user }}"
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
+  changed_when: true
+  when:
+    - gitea_runner_systemd_available.stat.exists
+    - gitea_runner_docker_rootless_setup
+    - gitea_runner_service_file is changed

 - name: Enable and start gitea-runner user service
  ansible.builtin.command: systemctl --user enable --now gitea-runner
@@ -24,6 +40,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  changed_when: true
  when:
    - gitea_runner_systemd_available.stat.exists
@@ -8,6 +8,7 @@
  become_user: "{{ gitea_runner_service_user }}"
  environment:
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  when:
    - gitea_runner_systemd_available.stat.exists | default(false) | bool
    - gitea_runner_docker_rootless_setup
@@ -69,3 +69,31 @@
    owner: "{{ gitea_runner_service_user }}"
    group: "{{ gitea_runner_service_user }}"
    mode: "0755"
+
+- name: Disable systemd-oomd memory pressure kill for runner user
+  when: gitea_runner_systemd_available.stat.exists
+  block:
+    - name: Ensure user service override directory exists
+      ansible.builtin.file:
+        path: "/etc/systemd/system/user@{{ gitea_runner_uid }}.service.d"
+        state: directory
+        owner: root
+        group: root
+        mode: "0755"
+
+    - name: Disable ManagedOOMMemoryPressure for runner user
+      ansible.builtin.copy:
+        content: |
+          [Service]
+          ManagedOOMMemoryPressure=auto
+          ManagedOOMMemoryPressureLimit=100%
+          OOMScoreAdjust=-500
+        dest: "/etc/systemd/system/user@{{ gitea_runner_uid }}.service.d/oomd-override.conf"
+        owner: root
+        group: root
+        mode: "0644"
+      notify: Reload systemd user daemon
+
+    - name: Reload systemd daemon for oomd override
+      ansible.builtin.systemd:
+        daemon_reload: true
@@ -21,6 +21,7 @@
  environment:
    DOCKER_HOST: "unix:///run/user/{{ gitea_runner_uid }}/docker.sock"
    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
  register: gitea_runner_docker_version_output
  changed_when: false
  when: gitea_runner_docker_rootless_setup
@@ -5,5 +5,19 @@ Description=Docker prune for Gitea runner resources
 Type=oneshot
 Environment=DOCKER_HOST=unix:///run/user/{{ gitea_runner_uid }}/docker.sock
 Environment=XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
-ExecStart=/usr/bin/docker system prune -f --filter "label={{ gitea_runner_prune_label }}" --filter "until={{ gitea_runner_prune_until }}"
-ExecStart=/usr/bin/docker volume prune -f --filter "label={{ gitea_runner_prune_label }}"
+# Force-remove stale containers (including running ones) left behind by failed
+# molecule tests. "docker container prune -f" only removes stopped containers,
+# so running containers from crashed/interrupted CI jobs accumulate indefinitely,
+# consuming disk and memory. We stop+rm everything first, then prune the rest.
+# Exclude CI job containers (name starts with GITEA-ACTIONS-TASK) — removing
+# them kills the active CI job and causes "RWLayer is unexpectedly nil" errors.
+# Only remove containers older than 1 hour (grep for "hour/day/week/month/year
+# ago" in RunningFor) to avoid killing molecule test containers that CI jobs
+# are actively using.
+ExecStart=/bin/sh -c 'docker ps -a --format "{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}" 2>/dev/null | grep -v "GITEA-ACTIONS-TASK" | grep -E "(hour|day|week|month|year)s? ago" | awk "{print $1}" | xargs -r docker rm -f 2>/dev/null || true'
+ExecStart=/usr/bin/docker system prune -af --filter "until={{ gitea_runner_prune_until }}" --volumes
+# Prune networks older than the prune-until threshold to avoid removing
+# networks that molecule tests are actively creating (e.g. 'traefik' network
+# created during molecule create phase before containers are attached).
+ExecStart=/usr/bin/docker network prune -f --filter "until={{ gitea_runner_prune_until }}"
+ExecStart=/usr/bin/docker builder prune -f
@@ -3,6 +3,8 @@ Description=Gitea Actions Runner (rootless)
 After=docker.service
 Requires=docker.service
 PartOf=docker.service
+StartLimitIntervalSec=300
+StartLimitBurst=10

 [Service]
 Type=simple
@@ -14,8 +16,6 @@ ExecStop=/bin/kill -TERM $MAINPID
 TimeoutStopSec=30
 Restart=always
 RestartSec={{ gitea_runner_service_restart_sec }}
-StartLimitIntervalSec=300
-StartLimitBurst=10

 [Install]
 WantedBy=default.target
@@ -7,18 +7,29 @@ DOCKER_HOST="unix:///run/user/{{ gitea_runner_uid }}/docker.sock"
 XDG_RUNTIME_DIR="/run/user/{{ gitea_runner_uid }}"
 export DOCKER_HOST XDG_RUNTIME_DIR

-# 1. Check Docker daemon responsiveness
-if ! docker info >/dev/null 2>&1; then
-    echo "ERROR: Docker daemon not responding at ${DOCKER_HOST}"
+# 1. Check Docker daemon responsiveness (with timeout — a bare `docker info` can
+#    hang indefinitely on a stuck daemon, blocking the healthcheck itself).
+if ! timeout 10 docker info >/dev/null 2>&1; then
+    echo "ERROR: Docker daemon not responding at ${DOCKER_HOST} (timed out after 10s)"
    systemctl --user restart docker.service
    sleep 3
-    if ! docker info >/dev/null 2>&1; then
+    if ! timeout 10 docker info >/dev/null 2>&1; then
        echo "CRITICAL: Docker daemon still down after restart"
        exit 1
    fi
    echo "RECOVERED: Docker daemon restarted successfully"
 fi

+# 1b. Clean up stuck containers — containers in "removing" or "stopping" state
+#     for too long cause "cannot kill container: did not receive an exit event"
+#     errors in molecule destroy phases. Force-remove them so subsequent CI jobs
+#     don't inherit the stuck state.
+stuck_containers=$(timeout 10 docker ps -a --filter "status=removing" --filter "status=stopping" --format '{% raw %}{{.ID}}{% endraw %}' 2>/dev/null || true)
+if [[ -n "$stuck_containers" ]]; then
+    echo "WARN: Found stuck containers (removing/stopping), force-cleaning"
+    echo "$stuck_containers" | xargs -r docker rm -f 2>/dev/null || true
+fi
+
 # 2. Check gitea-runner service is active
 runner_state=$(systemctl --user is-active gitea-runner.service 2>/dev/null || true)
 if [[ "$runner_state" != "active" ]]; then
@@ -33,14 +44,232 @@ if [[ "$runner_state" != "active" ]]; then
    echo "RECOVERED: gitea-runner service restarted successfully"
 fi

+# 2b. Detect unregistered runner state. When Gitea no longer recognizes the
+# runner (e.g., server restore, runner record deleted, Gitea restart with
+# token salt change), the runner logs "unregistered runner" every few seconds.
+# A service restart will not fix this; re-registration is required.
+#
+# Detection method: query the Gitea API to verify the runner's UUID still
+# exists.  This is more reliable than parsing journal logs (which requires
+# journal access permissions that runner users may not have — see the
+# 2026-08-08 incident where journalctl --user returned "No journal files
+# were opened due to insufficient permissions" for all runner users,
+# causing the healthcheck to always report "OK: runner healthy" even
+# though all runners were unregistered).
+{% if gitea_runner_auto_recover_api_token %}
+# Auto-recovery is enabled: fetch a new registration token from the Gitea API
+# and re-register the runner automatically. A cooldown prevents infinite loops.
+GITEA_API_TOKEN_FILE="{{ gitea_runner_config_dir }}/auto-recover.token"
+COOLDOWN_FILE="{{ gitea_runner_data_dir }}/auto-recover.cooldown"
+COOLDOWN_SEC={{ gitea_runner_auto_recover_cooldown_sec }}
+GITEA_URL="{{ gitea_url }}"
+RUNNER_NAME="{{ gitea_runner_name }}"
+RUNNER_LABELS="{{ gitea_runner_labels }}"
+BINARY="{{ gitea_runner_binary_path }}"
+RUNNER_FILE="{{ gitea_runner_data_dir }}/.runner"
+{% endif %}
+runner_unregistered=0
+
+# Primary detection: query the Gitea API to check if the runner's ID
+# still exists in Gitea's runner list.  This works regardless of journal
+# permissions.
+{% if gitea_runner_auto_recover_api_token %}
+if [[ -f "$GITEA_API_TOKEN_FILE" && -f "$RUNNER_FILE" ]]; then
+    API_TOKEN=$(cat "$GITEA_API_TOKEN_FILE" 2>/dev/null || true)
+    RUNNER_ID=$(python3 -c "import json; print(json.load(open('$RUNNER_FILE')).get('id',''))" 2>/dev/null || true)
+    if [[ -n "$API_TOKEN" && -n "$RUNNER_ID" ]]; then
+        # List all runners and check if our ID is present
+        runner_found=$(curl -sf --connect-timeout 5 --max-time 10 \
+            -H "Authorization: token $API_TOKEN" \
+            "${GITEA_URL}/api/v1/admin/actions/runners" 2>/dev/null \
+            | python3 -c "
+import sys, json
+try:
+    data = json.load(sys.stdin)
+    runners = data if isinstance(data, list) else data.get('runners', [])
+    ids = [str(r.get('id', '')) for r in runners]
+    print('1' if '$RUNNER_ID' in ids else '0')
+except Exception:
+    print('0')
+" 2>/dev/null || echo "0")
+        if [[ "$runner_found" != "1" ]]; then
+            runner_unregistered=1
+            echo "CRITICAL: runner ID $RUNNER_ID not found in Gitea (unregistered)."
+        fi
+    fi
+fi
+{% endif %}
+
+# Fallback detection: check journal logs (if accessible)
+if [[ "$runner_unregistered" -eq 0 ]]; then
+    recent_errors=$(journalctl --user -u gitea-runner.service --since "5 minutes ago" --no-pager -q 2>/dev/null | grep -c "unregistered runner" || true)
+    if [[ "$recent_errors" -ge 3 ]]; then
+        runner_unregistered=1
+        echo "CRITICAL: runner is unregistered in Gitea (re-login failed $recent_errors times in 5 minutes)."
+    fi
+fi
+
+if [[ "$runner_unregistered" -ge 1 ]]; then
+{% if gitea_runner_auto_recover_api_token %}
+    # Check cooldown — skip if we recently attempted recovery
+    if [[ -f "$COOLDOWN_FILE" ]]; then
+        last_attempt=$(cat "$COOLDOWN_FILE" 2>/dev/null || echo 0)
+        now=$(date +%s)
+        elapsed=$((now - last_attempt))
+        if [[ "$elapsed" -lt "$COOLDOWN_SEC" ]]; then
+            echo "SKIP: auto-recovery cooldown active (${elapsed}s < ${COOLDOWN_SEC}s). Will retry later."
+            exit 3
+        fi
+    fi
+
+    # Mark attempt time BEFORE trying (so failures also get cooldown)
+    date +%s > "$COOLDOWN_FILE" 2>/dev/null || true
+
+    # Read the API token
+    if [[ ! -f "$GITEA_API_TOKEN_FILE" ]]; then
+        echo "ERROR: auto-recover token file not found at $GITEA_API_TOKEN_FILE. Manual re-registration required."
+        exit 3
+    fi
+    API_TOKEN=$(cat "$GITEA_API_TOKEN_FILE" 2>/dev/null || true)
+    if [[ -z "$API_TOKEN" ]]; then
+        echo "ERROR: auto-recover token file is empty. Manual re-registration required."
+        exit 3
+    fi
+
+    echo "ATTEMPT: auto-recovering by fetching new registration token and re-registering..."
+
+    # Fetch a new registration token from the Gitea API
+    # Try org-level first (for org-scoped runners), then instance-level
+    REG_TOKEN=""
+    for endpoint in \
+        "api/v1/orgs/{{ gitea_runner_org | default('oblachno') }}/actions/runners/registration-token" \
+        "api/v1/admin/actions/runners/registration-token"; do
+        REG_TOKEN=$(curl -sf --connect-timeout 5 --max-time 10 -X POST \
+            -H "Authorization: token $API_TOKEN" \
+            "${GITEA_URL}/${endpoint}" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('token',''))" 2>/dev/null || true)
+        if [[ -n "$REG_TOKEN" ]]; then
+            echo "INFO: fetched registration token from ${endpoint}"
+            break
+        fi
+    done
+
+    if [[ -z "$REG_TOKEN" ]]; then
+        echo "ERROR: failed to fetch registration token from Gitea API. Is Gitea reachable?"
+        exit 3
+    fi
+
+    # Stop the runner service
+    systemctl --user stop gitea-runner.service 2>/dev/null || true
+    sleep 1
+
+    # Remove the stale .runner file
+    rm -f "{{ gitea_runner_data_dir }}/.runner" 2>/dev/null || true
+
+    # Re-register
+    cd "{{ gitea_runner_data_dir }}"
+    if "$BINARY" register \
+        --token "$REG_TOKEN" \
+        --name "$RUNNER_NAME" \
+        --instance "$GITEA_URL" \
+        --labels "$RUNNER_LABELS" \
+        --no-interactive 2>&1; then
+        echo "RECOVERED: runner re-registered successfully"
+    else
+        echo "ERROR: re-registration failed. Manual intervention required."
+        exit 3
+    fi
+
+    # Start the runner service
+    systemctl --user start gitea-runner.service
+    sleep 3
+
+    # Verify recovery — query the Gitea API to confirm the new ID is registered
+    NEW_ID=$(python3 -c "import json; print(json.load(open('$RUNNER_FILE')).get('id',''))" 2>/dev/null || true)
+    if [[ -n "$NEW_ID" ]]; then
+        new_found=$(curl -sf --connect-timeout 5 --max-time 10 \
+            -H "Authorization: token $API_TOKEN" \
+            "${GITEA_URL}/api/v1/admin/actions/runners" 2>/dev/null \
+            | python3 -c "
+import sys, json
+try:
+    data = json.load(sys.stdin)
+    runners = data if isinstance(data, list) else data.get('runners', [])
+    ids = [str(r.get('id', '')) for r in runners]
+    print('1' if '$NEW_ID' in ids else '0')
+except Exception:
+    print('0')
+" 2>/dev/null || echo "0")
+        if [[ "$new_found" == "1" ]]; then
+            echo "OK: runner recovered and registered with new ID $NEW_ID"
+            # Clear cooldown on success
+            rm -f "$COOLDOWN_FILE" 2>/dev/null || true
+        else
+            echo "WARN: runner re-registered but ID not found in Gitea API. Will retry after cooldown."
+            exit 3
+        fi
+    else
+        echo "WARN: could not read new .runner file after re-registration. Will retry after cooldown."
+        exit 3
+    fi
+{% else %}
+    echo "Manual re-registration required: re-run gitea_runner role with gitea_runner_force_reregister=true."
+    # Restart the service once in case it is a transient token refresh issue,
+    # but this cannot recover an unregistered runner without re-registration.
+    systemctl --user restart gitea-runner.service
+    sleep 2
+    exit 3
+{% endif %}
+fi
+
 # 3. Check disk space — prune aggressively if below threshold
 disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
-if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
-    echo "WARN: Disk usage at ${disk_pct}%, pruning all runner resources"
-    docker system prune -af --filter "label={{ gitea_runner_prune_label }}" --filter "until=1h" || true
-    docker volume prune -af --filter "label={{ gitea_runner_prune_label }}" || true
-    # Also prune dangling images (no label)
-    docker image prune -af || true
+# Debug: log image count and sizes before pruning decisions
+img_count=$(timeout 10 docker images -q 2>/dev/null | wc -l || echo "?")
+img_size=$(timeout 10 docker system df --format '{% raw %}{{.Type}} {{.Size}} {{.Reclaimable}}{% endraw %}' 2>/dev/null | grep "Images" || echo "unknown")
+echo "INFO: disk=${disk_pct}% images=${img_count} | ${img_size}"
+if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_critical }} ]]; then
+    echo "CRITICAL: Disk usage at ${disk_pct}% (>= {{ gitea_runner_healthcheck_disk_critical }}%), full prune"
+    # Critical level: remove ALL stopped containers (no age filter) and ALL
+    # unused images/volumes. The until=1h gentle prune is insufficient here.
+    # Stop+rm stale non-CI containers regardless of age (failed molecule tests
+    # from the last 59 minutes also consume disk).
+    docker ps -a --format '{% raw %}{{.ID}} {{.Names}}{% endraw %}' 2>/dev/null \
+        | grep -v 'GITEA-ACTIONS-TASK' \
+        | awk '{print $1}' \
+        | xargs -r docker rm -f 2>/dev/null || true
+    docker system prune -af --volumes || true
+    # Prune networks older than 1 hour only — removing ALL networks can
+    # race with Gitea Actions jobs that are mid-startup (network created
+    # but container not yet attached), causing "network not found" errors.
+    docker network prune -f --filter "until=1h" || true
+    docker builder prune -af || true
+    disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
+    echo "INFO: Disk usage after full prune: ${disk_pct}%"
+elif [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
+    echo "WARN: Disk usage at ${disk_pct}%, pruning runner resources (until=1h)"
+    # Force-remove stale containers (including running ones from failed molecule tests)
+    # that are older than 1 hour. "docker container prune -f" only removes stopped
+    # containers, so running containers from crashed CI jobs accumulate and consume
+    # disk/memory. Exclude CI job containers (name starts with GITEA-ACTIONS-TASK).
+    # Only remove containers older than 1 hour to avoid killing molecule test
+    # containers that CI jobs are actively using.
+    docker ps -a --format '{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}' 2>/dev/null \
+        | grep -v 'GITEA-ACTIONS-TASK' \
+        | grep -E '(hour|day|week|month|year)s? ago' \
+        | awk '{print $1}' \
+        | xargs -r docker rm -f 2>/dev/null || true
+    # Prune containers and volumes older than 1h. Do NOT prune images here —
+    # the CI runner image (ci-full:latest, ~2.3GB) is unused between jobs and
+    # would be removed by image prune, forcing a re-pull on every CI job
+    # (84+ seconds per pull, and the pull can fail silently causing
+    # "No such image" errors). Images are only pruned at the critical
+    # threshold (75%) when disk is genuinely full.
+    docker container prune -f --filter "until=1h" 2>/dev/null || true
+    docker volume prune -f 2>/dev/null || true
+    # Prune networks older than 1 hour to avoid removing networks that
+    # molecule tests are actively creating (e.g. 'traefik' network created
+    # during molecule create phase before containers are attached).
+    docker network prune -f --filter "until=1h" || true
    disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
    echo "INFO: Disk usage after prune: ${disk_pct}%"
 fi
@@ -33,5 +33,5 @@
      become_user: "{{ gitea_runner_service_user }}"
      environment:
        XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
-      when: systemd_available.stat.exists
+      when: gitea_runner_systemd_available.stat.exists
      changed_when: true
@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate

 [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
 [![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
-[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
-[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
-[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/bf77093ceedcbb44e60ebfdbd10ebf17643377d5/python.svg)](https://www.python.org/downloads/)
+[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
+[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
+[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/918ffe7df0c91464112f012126e3fd60a0dfa2d5/python.svg)](https://www.python.org/downloads/)

 ## Overview

@@ -259,9 +259,9 @@ OS platform matrix (defined in `devx.molecule.platforms`), then splits
 the resulting test pairs evenly across the requested number of runners.
 Each pair is encoded as `scenario|platform_name|platform_image|platform_command`.

-`devx.molecule.molecule_ci_guard` runs the actual molecule test for a
-given test pair, with CI context (Gitea URL, token, run ID) for
-reporting results back to the commit status API.
+The CI workflow runs each test pair sequentially via a shell loop that
+sets the appropriate `MOLECULE_PLATFORM_*` environment variables and
+invokes `molecule test` directly.

 ### Commit Message Validation

@@ -91,7 +91,7 @@ The `molecule-tests` job uses `fromJSON()` to consume the dynamic matrix, and pa

 `devx.molecule.distribute_molecule` discovers all molecule scenarios under `ansible/roles/*/molecule/` and crosses them with the supported OS platform matrix, then splits the resulting test pairs evenly across the requested number of runners. Each pair is encoded as `scenario|platform_name|platform_image|platform_command`.

-`devx.molecule.molecule_ci_guard` runs the actual molecule test for a given test pair, with CI context (Gitea URL, token, run ID) for reporting results back to the commit status API.
+The CI workflow runs each test pair sequentially via a shell loop that sets the appropriate `MOLECULE_PLATFORM_*` environment variables and invokes `molecule test` directly.

 ### Path-based CI filtering

@@ -32,10 +32,11 @@ version = {attr = "grm.__version__"}
 ci = [
    "pytest==9.1.1",
    "pytest-cov==7.1.0",
+    "pytest-xdist==3.8.0",
    "build==1.5.1",
    "twine==6.2.0",
    # Reusable CI/CD and dev tools (auto-merge, pr-review, pre-push checks, etc.)
-    "devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.47.1",
+    "devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.50.0",
 ]
 # Lint and type-checking tools (validate job)
 lint = [
@@ -55,7 +56,7 @@ molecule = [
 dev = [
    "grm[ci,lint,molecule]",
    # Reusable CI/CD and dev tools (pre-push hooks, create-task, create-pr)
-    "devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.47.1",
+    "devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.50.0",
    # Non-Python dev dependency: checkmake (Makefile linter)
    # Install via: go install github.com/checkmake/checkmake/cmd/checkmake@latest
 ]
@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+"""Clean up stale runner registrations from Gitea.
+
+A runner is considered stale if it hasn't been online for more than a
+configurable threshold (default: 1 hour).  Stale runners accumulate when:
+- A runner host is rebuilt or re-provisioned (old registration remains)
+- A runner is re-registered (old entry remains alongside the new one)
+- A runner process dies and the healthcheck can't auto-recover
+
+This script queries the Gitea API for all runners, identifies stale ones,
+and deletes them via ``DELETE /api/v1/admin/actions/runners/{id}``.
+
+Usage::
+
+    python3 scripts/cleanup_stale_runners.py --gitea-url https://git.example.com --token <admin-token>
+    python3 scripts/cleanup_stale_runners.py --gitea-url https://git.example.com --token <admin-token> --dry-run
+    python3 scripts/cleanup_stale_runners.py --gitea-url https://git.example.com --token <admin-token> \\
+        --stale-threshold 3600
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+import urllib.error
+import urllib.request  # noqa: PTH123  # nosec B404
+from typing import Any
+
+
+def _api_request(base_url: str, token: str, method: str, path: str) -> Any:
+    url = f"{base_url.rstrip('/')}/api/v1{path}"
+    req = urllib.request.Request(url, method=method)  # nosec B310
+    req.add_header("Authorization", f"token {token}")
+    req.add_header("Accept", "application/json")
+    try:
+        with urllib.request.urlopen(req) as resp:  # noqa: PTH123  # nosec B310
+            if resp.status == 204:
+                return None
+            raw = resp.read()
+            return json.loads(raw) if raw else None
+    except urllib.error.HTTPError as e:
+        detail = e.read().decode("utf-8", errors="replace")
+        raise RuntimeError(f"Gitea API error {e.code}: {detail}") from e
+
+
+def list_runners(base_url: str, token: str) -> list[dict[str, Any]]:
+    data = _api_request(base_url, token, "GET", "/admin/actions/runners")
+    if data is None:
+        return []
+    if isinstance(data, list):
+        return data
+    if isinstance(data, dict):
+        return data.get("runners", [])
+    return []
+
+
+def delete_runner(base_url: str, token: str, runner_id: int) -> bool:
+    try:
+        _api_request(base_url, token, "DELETE", f"/admin/actions/runners/{runner_id}")
+        return True
+    except RuntimeError as e:
+        print(f"  ERROR deleting runner {runner_id}: {e}", file=sys.stderr)
+        return False
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description="Clean up stale Gitea runner registrations")
+    parser.add_argument("--gitea-url", required=True, help="Gitea base URL")
+    parser.add_argument("--token", required=True, help="Gitea admin API token")
+    parser.add_argument(
+        "--stale-threshold",
+        type=int,
+        default=3600,
+        help="Seconds since last_online before a runner is considered stale (default: 3600 = 1h)",
+    )
+    parser.add_argument("--dry-run", action="store_true", help="List stale runners without deleting")
+    args = parser.parse_args(argv)
+
+    runners = list_runners(args.gitea_url, args.token)
+    if not runners:
+        print("No runners found.")
+        return 0
+
+    now = int(time.time())
+    stale: list[dict[str, Any]] = []
+    online: list[dict[str, Any]] = []
+
+    for runner in runners:
+        last_online = runner.get("last_online", 0) or 0
+        seconds_since = now - last_online
+        runner["seconds_since_online"] = seconds_since
+        if seconds_since > args.stale_threshold:
+            stale.append(runner)
+        else:
+            online.append(runner)
+
+    print(f"Total runners: {len(runners)}")
+    print(f"Online (within {args.stale_threshold}s): {len(online)}")
+    print(f"Stale (>{args.stale_threshold}s): {len(stale)}")
+    print()
+
+    if not stale:
+        print("No stale runners to clean up.")
+        return 0
+
+    print("Stale runners:")
+    for r in stale:
+        rid = r.get("id", "?")
+        name = r.get("name", "?")
+        uuid = r.get("uuid", "?")[:8]
+        secs = r.get("seconds_since_online", 0)
+        hours = secs / 3600
+        print(f"  id={rid} name={name} uuid={uuid}... offline={hours:.1f}h ago")
+
+    if args.dry_run:
+        print("\n--dry-run: not deleting. Remove --dry-run to clean up.")
+        return 0
+
+    print(f"\nDeleting {len(stale)} stale runners...")
+    deleted = 0
+    for r in stale:
+        rid = r.get("id")
+        if rid is None:
+            continue
+        if delete_runner(args.gitea_url, args.token, rid):
+            deleted += 1
+            print(f"  Deleted runner id={rid} ({r.get('name', '?')})")
+
+    print(f"\nDone: {deleted}/{len(stale)} stale runners deleted.")
+    return 0 if deleted == len(stale) else 1
+
+
+if __name__ == "__main__":  # pragma: no cover
+    sys.exit(main())
@@ -0,0 +1,217 @@
+"""Tests for cleanup_stale_runners.py."""
+
+from __future__ import annotations
+
+import time
+from unittest.mock import MagicMock, patch
+
+from scripts.cleanup_stale_runners import (
+    _api_request,
+    delete_runner,
+    list_runners,
+    main,
+)
+
+
+class TestListRunners:
+    """Tests for list_runners()."""
+
+    @patch("scripts.cleanup_stale_runners._api_request")
+    def test_returns_list_of_runners(self, mock_req: MagicMock) -> None:
+        mock_req.return_value = [{"id": 1, "name": "runner-1"}, {"id": 2, "name": "runner-2"}]
+        result = list_runners("https://git.example.com", "token")
+        assert len(result) == 2
+        assert result[0]["id"] == 1
+
+    @patch("scripts.cleanup_stale_runners._api_request")
+    def test_returns_empty_on_none(self, mock_req: MagicMock) -> None:
+        mock_req.return_value = None
+        result = list_runners("https://git.example.com", "token")
+        assert result == []
+
+    @patch("scripts.cleanup_stale_runners._api_request")
+    def test_extracts_runners_from_dict(self, mock_req: MagicMock) -> None:
+        mock_req.return_value = {"runners": [{"id": 1}]}
+        result = list_runners("https://git.example.com", "token")
+        assert len(result) == 1
+        assert result[0]["id"] == 1
+
+    @patch("scripts.cleanup_stale_runners._api_request")
+    def test_returns_empty_on_non_list_non_dict(self, mock_req: MagicMock) -> None:
+        mock_req.return_value = "not a list"
+        result = list_runners("https://git.example.com", "token")
+        assert result == []
+
+
+class TestDeleteRunner:
+    """Tests for delete_runner()."""
+
+    @patch("scripts.cleanup_stale_runners._api_request")
+    def test_returns_true_on_success(self, mock_req: MagicMock) -> None:
+        mock_req.return_value = None
+        assert delete_runner("https://git.example.com", "token", 42) is True
+
+    @patch("scripts.cleanup_stale_runners._api_request")
+    def test_returns_false_on_error(self, mock_req: MagicMock) -> None:
+        mock_req.side_effect = RuntimeError("API error 404: not found")
+        assert delete_runner("https://git.example.com", "token", 42) is False
+
+
+class TestApiRequest:
+    """Tests for _api_request()."""
+
+    @patch("scripts.cleanup_stale_runners.urllib.request.urlopen")
+    def test_returns_json_on_success(self, mock_urlopen: MagicMock) -> None:
+        mock_resp = MagicMock()
+        mock_resp.status = 200
+        mock_resp.read.return_value = b'{"key": "value"}'
+        mock_urlopen.return_value.__enter__.return_value = mock_resp
+        result = _api_request("https://git.example.com", "token", "GET", "/test")
+        assert result == {"key": "value"}
+
+    @patch("scripts.cleanup_stale_runners.urllib.request.urlopen")
+    def test_returns_none_on_204(self, mock_urlopen: MagicMock) -> None:
+        mock_resp = MagicMock()
+        mock_resp.status = 204
+        mock_urlopen.return_value.__enter__.return_value = mock_resp
+        result = _api_request("https://git.example.com", "token", "DELETE", "/test/1")
+        assert result is None
+
+    @patch("scripts.cleanup_stale_runners.urllib.request.urlopen")
+    def test_returns_none_on_empty_body(self, mock_urlopen: MagicMock) -> None:
+        mock_resp = MagicMock()
+        mock_resp.status = 200
+        mock_resp.read.return_value = b""
+        mock_urlopen.return_value.__enter__.return_value = mock_resp
+        result = _api_request("https://git.example.com", "token", "GET", "/test")
+        assert result is None
+
+    @patch("scripts.cleanup_stale_runners.urllib.request.urlopen")
+    def test_raises_on_http_error(self, mock_urlopen: MagicMock) -> None:
+        import urllib.error
+
+        mock_error = urllib.error.HTTPError(
+            "url",
+            404,
+            "Not Found",
+            {},
+            None,
+        )
+        mock_error.read = MagicMock(return_value=b'{"message": "not found"}')
+        mock_urlopen.side_effect = mock_error
+        import pytest
+
+        with pytest.raises(RuntimeError, match="404"):
+            _api_request("https://git.example.com", "token", "GET", "/test")
+
+
+class TestMain:
+    """Tests for main()."""
+
+    @patch("scripts.cleanup_stale_runners.list_runners")
+    def test_no_runners(self, mock_list: MagicMock) -> None:
+        mock_list.return_value = []
+        rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
+        assert rc == 0
+
+    @patch("scripts.cleanup_stale_runners.list_runners")
+    def test_no_stale_runners(self, mock_list: MagicMock) -> None:
+        now = int(time.time())
+        mock_list.return_value = [
+            {"id": 1, "name": "runner-1", "last_online": now - 60},
+        ]
+        rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
+        assert rc == 0
+
+    @patch("scripts.cleanup_stale_runners.list_runners")
+    def test_dry_run_does_not_delete(self, mock_list: MagicMock) -> None:
+        now = int(time.time())
+        mock_list.return_value = [
+            {"id": 1, "name": "runner-1", "last_online": now - 7200},
+        ]
+        with patch("scripts.cleanup_stale_runners.delete_runner") as mock_del:
+            rc = main(
+                [
+                    "--gitea-url",
+                    "https://git.example.com",
+                    "--token",
+                    "t",
+                    "--dry-run",
+                ]
+            )
+            assert rc == 0
+            mock_del.assert_not_called()
+
+    @patch("scripts.cleanup_stale_runners.list_runners")
+    @patch("scripts.cleanup_stale_runners.delete_runner")
+    def test_deletes_stale_runners(self, mock_del: MagicMock, mock_list: MagicMock) -> None:
+        now = int(time.time())
+        mock_list.return_value = [
+            {"id": 1, "name": "runner-1", "last_online": now - 60},
+            {"id": 2, "name": "runner-2", "last_online": now - 7200},
+            {"id": 3, "name": "runner-3", "last_online": now - 9999},
+        ]
+        mock_del.return_value = True
+        rc = main(
+            [
+                "--gitea-url",
+                "https://git.example.com",
+                "--token",
+                "t",
+                "--stale-threshold",
+                "3600",
+            ]
+        )
+        assert rc == 0
+        assert mock_del.call_count == 2
+
+    @patch("scripts.cleanup_stale_runners.list_runners")
+    @patch("scripts.cleanup_stale_runners.delete_runner")
+    def test_returns_1_on_partial_failure(self, mock_del: MagicMock, mock_list: MagicMock) -> None:
+        now = int(time.time())
+        mock_list.return_value = [
+            {"id": 1, "name": "runner-1", "last_online": now - 7200},
+            {"id": 2, "name": "runner-2", "last_online": now - 7200},
+        ]
+        mock_del.side_effect = [True, False]
+        rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
+        assert rc == 1
+
+    @patch("scripts.cleanup_stale_runners.list_runners")
+    def test_runner_with_zero_last_online(self, mock_list: MagicMock) -> None:
+        """Runners with last_online=0 should be considered stale."""
+        mock_list.return_value = [
+            {"id": 1, "name": "runner-1", "last_online": 0},
+        ]
+        with patch("scripts.cleanup_stale_runners.delete_runner") as mock_del:
+            mock_del.return_value = True
+            rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
+            assert rc == 0
+            mock_del.assert_called_once()
+
+    @patch("scripts.cleanup_stale_runners.list_runners")
+    def test_runner_with_missing_last_online(self, mock_list: MagicMock) -> None:
+        """Runners with missing last_online should be considered stale."""
+        mock_list.return_value = [
+            {"id": 1, "name": "runner-1"},
+        ]
+        with patch("scripts.cleanup_stale_runners.delete_runner") as mock_del:
+            mock_del.return_value = True
+            rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
+            assert rc == 0
+            mock_del.assert_called_once()
+
+    @patch("scripts.cleanup_stale_runners.list_runners")
+    @patch("scripts.cleanup_stale_runners.delete_runner")
+    def test_skips_runner_with_none_id(self, mock_del: MagicMock, mock_list: MagicMock) -> None:
+        """Runners with id=None should be skipped during deletion."""
+        now = int(time.time())
+        mock_list.return_value = [
+            {"id": None, "name": "bad-runner", "last_online": now - 7200},
+            {"id": 2, "name": "runner-2", "last_online": now - 7200},
+        ]
+        mock_del.return_value = True
+        rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
+        # 1/2 deleted (None id skipped), so rc=1 (partial)
+        assert rc == 1
+        mock_del.assert_called_once_with("https://git.example.com", "t", 2)
@@ -1,3 +1,3 @@
 """Gitea Runner Manager — lean CLI for managing Gitea Actions runners."""

-__version__ = "0.18.2"
+__version__ = "0.20.0"
@@ -145,11 +145,25 @@ def cli(ctx: click.Context, become_password_file: str | None, verbose: bool) ->
        "Example: docker:docker://alpine:latest"
    ),
 )
+@click.option(
+    "--force-reregister/--no-force-reregister",
+    default=False,
+    help=_("Force re-registration even if .runner file exists (env: GITEA_FORCE_REREGISTER)"),
+)
@click.option(
    "--ask-become-pass/--no-ask-become-pass",
    default=True,
    help=_("Prompt for sudo password (default)"),
 )
+@click.option(
+    "--auto-recover-token",
+    default=None,
+    help=_(
+        "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). "
+        "When set, the healthcheck can automatically re-register the runner "
+        "if it becomes unregistered. Requires admin or org-level access."
+    ),
+)
@_handle_errors("Installation failed: {error}")
 def install(
    host: str,
@@ -161,10 +175,14 @@ def install(
    admin_token: str | None,
    integration_retries: int,
    labels: str | None,
+    force_reregister: bool,
    ask_become_pass: bool,
+    auto_recover_token: str | None,
 ) -> None:
    if labels is None:
        labels = os.getenv("GITEA_RUNNER_LABELS")
+    if auto_recover_token is None:
+        auto_recover_token = os.getenv("GITEA_AUTO_RECOVER_TOKEN")
    manager = RunnerManager()
    manager.install(
        host=host,
@@ -176,9 +194,11 @@ def install(
        admin_token=admin_token,
        integration_retries=integration_retries,
        labels=labels,
+        force_reregister=force_reregister,
        ask_become_pass=ask_become_pass,
        become_password_file=_get_become_password_file(),
        verbose=_get_verbose(),
+        auto_recover_token=auto_recover_token,
    )


@@ -506,7 +526,7 @@ def list_runners(ask_become_pass: bool, no_status: bool) -> None:
    click.echo(f"{_('NAME'):<18} {_('HOST'):<16} {_('USER'):<10} {_('LABELS'):<30} {_('STATUS')}")
    click.echo("-" * 90)
    for r in runners:
-        click.echo(f"{r['name']:<18} {r['host']:<16} {r['user']:<10} {r['labels']:<30} {r['status']}")
+        click.echo(f"{r['name']:<18} {r['host']:<16} {r['user']:<10} {(r['labels'] or ''):<30} {r['status']}")


@cli.command(name="trigger-workflow", help=_("Trigger a Gitea Actions workflow via the API."))
@@ -87,8 +87,10 @@ class RunnerManager:
        integration_retries: int = 3,
        ask_become_pass: bool = False,
        labels: str | None = None,
+        force_reregister: bool = False,
        become_password_file: str | None = None,
        verbose: bool = False,
+        auto_recover_token: str | None = None,
    ) -> None:
        """Install a runner on a remote host using Ansible."""
        if not name:
@@ -98,16 +100,19 @@ class RunnerManager:
        if not token:
            raise AnsibleError(_("GITEA_REGISTRATION_TOKEN must be set (or pass --token)"))

-        extra_vars: dict[str, str | int] = {
+        extra_vars: dict[str, str | int | bool] = {
            "registration_token": token,
            "gitea_runner_name": name,
            "gitea_url": gitea_url,
            "gitea_runner_integration_retries": integration_retries,
+            "gitea_runner_force_reregister": force_reregister,
        }
        if admin_token:
            extra_vars["gitea_admin_token"] = admin_token
        if labels is not None:
            extra_vars["gitea_runner_labels"] = labels
+        if auto_recover_token:
+            extra_vars["gitea_runner_auto_recover_api_token"] = auto_recover_token

        with track_steps() as tracker:
            tracker.begin(_("Installing Gitea Runner on {host}", host=host))
@@ -367,6 +367,14 @@
    "ru": "Токен регистрации (env: GITEA_REGISTRATION_TOKEN)",
    "zh": "注册令牌（环境变量: GITEA_REGISTRATION_TOKEN）"
  },
+  "Force re-registration even if .runner file exists (env: GITEA_FORCE_REREGISTER)": {
+    "bg": "Принудителна повторна регистрация, дори ако .runner файлът съществува (env: GITEA_FORCE_REREGISTER)",
+    "de": "Erneute Registrierung erzwingen, auch wenn .runner-Datei existiert (env: GITEA_FORCE_REREGISTER)",
+    "en": "Force re-registration even if .runner file exists (env: GITEA_FORCE_REREGISTER)",
+    "pl": "Wymuś ponowną rejestrację, nawet jeśli plik .runner istnieje (env: GITEA_FORCE_REREGISTER)",
+    "ru": "Принудительно повторно зарегистрировать, даже если файл .runner существует (env: GITEA_FORCE_REREGISTER)",
+    "zh": "即使存在 .runner 文件也强制重新注册（环境变量: GITEA_FORCE_REREGISTER）"
+  },
  "Remove a registered Gitea Runner completely.": {
    "bg": "Пълно премахване на регистриран Gitea Runner.",
    "de": "Einen registrierten Gitea Runner vollständig entfernen.",
@@ -774,5 +782,13 @@
    "pl": "Workflow uruchomiony pomyślnie. ID uruchomienia: {run_id}",
    "ru": "Workflow успешно запущен. ID запуска: {run_id}",
    "zh": "工作流触发成功。运行 ID：{run_id}"
+  },
+  "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.": {
+    "bg": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
+    "de": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
+    "en": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
+    "pl": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
+    "ru": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
+    "zh": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access."
  }
 }
@@ -50,9 +50,11 @@ class TestCLI:
            admin_token="",
            integration_retries=3,
            labels=None,
+            force_reregister=False,
            ask_become_pass=True,
            become_password_file=None,
            verbose=False,
+            auto_recover_token=None,
        )

    @patch("grm.cli.RunnerManager")
@@ -73,9 +75,11 @@ class TestCLI:
            admin_token="",
            integration_retries=3,
            labels=None,
+            force_reregister=False,
            ask_become_pass=False,
            become_password_file=None,
            verbose=False,
+            auto_recover_token=None,
        )

    @patch("grm.cli.RunnerManager")
@@ -114,9 +118,11 @@ class TestCLI:
                admin_token=None,
                integration_retries=3,
                labels=None,
+                force_reregister=False,
                ask_become_pass=True,
                become_password_file=None,
                verbose=False,
+                auto_recover_token=None,
            )

    @patch("grm.cli.RunnerManager")
@@ -165,9 +171,11 @@ class TestCLI:
            admin_token="",
            integration_retries=3,
            labels=None,
+            force_reregister=False,
            ask_become_pass=True,
            become_password_file=None,
            verbose=False,
+            auto_recover_token=None,
        )

    @patch("grm.cli.RunnerManager")
@@ -188,9 +196,11 @@ class TestCLI:
            admin_token="",
            integration_retries=3,
            labels=None,
+            force_reregister=False,
            ask_become_pass=True,
            become_password_file=None,
            verbose=False,
+            auto_recover_token=None,
        )

    @patch("grm.cli.RunnerManager")
@@ -226,9 +236,92 @@ class TestCLI:
            admin_token="",
            integration_retries=3,
            labels="docker:docker://alpine:latest",
+            force_reregister=False,
            ask_become_pass=True,
            become_password_file=None,
            verbose=False,
+            auto_recover_token=None,
+        )
+
+    @patch("grm.cli.RunnerManager")
+    def test_install_force_reregister(self, mock_manager_class: MagicMock) -> None:
+        mock_manager = MagicMock()
+        mock_manager_class.return_value = mock_manager
+
+        runner = CliRunner(env=_TEST_ENV)
+        result = runner.invoke(cli, ["install", "host1", "--user", "ubuntu", "--token", "tok", "--force-reregister"])
+        assert result.exit_code == 0
+        mock_manager.install.assert_called_once_with(
+            host="host1",
+            user="ubuntu",
+            key=None,
+            name=None,
+            token="tok",
+            gitea_url="https://git.example.com",
+            admin_token="",
+            integration_retries=3,
+            labels=None,
+            force_reregister=True,
+            ask_become_pass=True,
+            become_password_file=None,
+            verbose=False,
+            auto_recover_token=None,
+        )
+
+    @patch("grm.cli.RunnerManager")
+    def test_install_with_auto_recover_token(self, mock_manager_class: MagicMock) -> None:
+        """--auto-recover-token passes the token to the manager for healthcheck auto-recovery."""
+        mock_manager = MagicMock()
+        mock_manager_class.return_value = mock_manager
+
+        runner = CliRunner(env=_TEST_ENV)
+        result = runner.invoke(
+            cli,
+            ["install", "host1", "--user", "ubuntu", "--token", "tok", "--auto-recover-token", "api-tok"],
+        )
+        assert result.exit_code == 0
+        mock_manager.install.assert_called_once_with(
+            host="host1",
+            user="ubuntu",
+            key=None,
+            name=None,
+            token="tok",
+            gitea_url="https://git.example.com",
+            admin_token="",
+            integration_retries=3,
+            labels=None,
+            force_reregister=False,
+            ask_become_pass=True,
+            become_password_file=None,
+            verbose=False,
+            auto_recover_token="api-tok",
+        )
+
+    @patch("grm.cli.RunnerManager")
+    def test_install_auto_recover_token_from_env(self, mock_manager_class: MagicMock) -> None:
+        """GITEA_AUTO_RECOVER_TOKEN env var is used when --auto-recover-token is not passed."""
+        mock_manager = MagicMock()
+        mock_manager_class.return_value = mock_manager
+
+        env = {**_TEST_ENV, "GITEA_AUTO_RECOVER_TOKEN": "env-tok"}
+        runner = CliRunner(env=env)
+        result = runner.invoke(cli, ["install", "host1", "--user", "ubuntu", "--token", "tok"])
+        assert result.exit_code == 0
+        mock_manager.install.assert_called_once_with(
+            host="host1",
+            user="ubuntu",
+            key=None,
+            name=None,
+            token="tok",
+            gitea_url="https://git.example.com",
+            admin_token="",
+            integration_retries=3,
+            labels=None,
+            force_reregister=False,
+            ask_become_pass=True,
+            become_password_file=None,
+            verbose=False,
+            auto_recover_token="env-tok",
        )

    @patch("grm.cli.RunnerManager")
@@ -250,9 +343,11 @@ class TestCLI:
            admin_token="",
            integration_retries=3,
            labels="",
+            force_reregister=False,
            ask_become_pass=True,
            become_password_file=None,
            verbose=False,
+            auto_recover_token=None,
        )

    @patch("grm.cli.RunnerManager")
@@ -274,9 +369,11 @@ class TestCLI:
            admin_token="",
            integration_retries=3,
            labels="docker:docker://alpine:latest",
+            force_reregister=False,
            ask_become_pass=True,
            become_password_file=None,
            verbose=False,
+            auto_recover_token=None,
        )

    @patch("grm.cli.RunnerManager")
@@ -307,9 +404,11 @@ class TestCLI:
                admin_token="",
                integration_retries=3,
                labels=None,
+                force_reregister=False,
                ask_become_pass=True,
                become_password_file=pw_file,
                verbose=False,
+                auto_recover_token=None,
            )
        finally:
            import os
@@ -334,9 +433,11 @@ class TestCLI:
            admin_token="",
            integration_retries=3,
            labels=None,
+            force_reregister=False,
            ask_become_pass=True,
            become_password_file=None,
            verbose=True,
+            auto_recover_token=None,
        )

    @patch("grm.cli.RunnerManager")
@@ -741,6 +842,27 @@ class TestCLI:
        assert "active" in result.output
        mock_manager.list_runners.assert_called_once_with(become_pass=None, no_status=False)

+    @patch("grm.cli.RunnerManager")
+    def test_list_with_none_labels(self, mock_manager_class: MagicMock) -> None:
+        """Runners with labels=None should not crash the list command."""
+        mock_manager = MagicMock()
+        mock_manager.list_runners.return_value = [
+            {
+                "name": "r1",
+                "host": "10.0.0.1",
+                "user": "ubuntu",
+                "labels": None,
+                "status": "active",
+            },
+        ]
+        mock_manager_class.return_value = mock_manager
+
+        runner = CliRunner()
+        result = runner.invoke(cli, ["list"])
+        assert result.exit_code == 0
+        assert "r1" in result.output
+        assert "active" in result.output
+
    @patch("grm.cli.RunnerManager")
    def test_list_no_status(self, mock_manager_class: MagicMock) -> None:
        """--no-status skips SSH checks and shows registry only."""
@@ -51,6 +51,7 @@ class TestRunnerManager:
        assert manager._captured_extra_vars["registration_token"] == "tok"
        assert manager._captured_extra_vars["gitea_runner_name"] == "192.168.1.10"
        assert manager._captured_extra_vars["gitea_url"] == "https://git.example.com"
+        assert manager._captured_extra_vars["gitea_runner_force_reregister"] is False
        assert "Installing Gitea Runner on 192.168.1.10" in mock_executor.run.call_args.kwargs["description"]
        mock_registry.add.assert_called_once_with(
            name="192.168.1.10",
@@ -76,6 +77,7 @@ class TestRunnerManager:
        assert "/key" in cmd_str
        assert manager._captured_extra_vars["registration_token"] == "preset"
        assert manager._captured_extra_vars["gitea_runner_name"] == "my-runner"
+        assert manager._captured_extra_vars["gitea_runner_force_reregister"] is False
        assert "--ask-become-pass" not in cmd_str
        mock_registry.add.assert_called_once_with(
            name="my-runner",
@@ -97,6 +99,52 @@ class TestRunnerManager:
        cmd_str = " ".join(cmd)
        assert "--ask-become-pass" in cmd_str

+    def test_install_force_reregister(self) -> None:
+        mock_registry = MagicMock()
+        manager = RunnerManager(registry=mock_registry)
+        mock_executor = MagicMock()
+        manager._executor = mock_executor
+
+        manager.install(
+            "host1",
+            "root",
+            token="tok",
+            gitea_url="https://git.example.com",
+            force_reregister=True,
+        )
+        assert manager._captured_extra_vars["gitea_runner_force_reregister"] is True
+
+    def test_install_with_auto_recover_token(self) -> None:
+        """auto_recover_token is passed as extra_var to Ansible."""
+        mock_registry = MagicMock()
+        manager = RunnerManager(registry=mock_registry)
+        mock_executor = MagicMock()
+        manager._executor = mock_executor
+
+        manager.install(
+            "host1",
+            "root",
+            token="tok",
+            gitea_url="https://git.example.com",
+            auto_recover_token="api-tok",
+        )
+        assert manager._captured_extra_vars["gitea_runner_auto_recover_api_token"] == "api-tok"
+
+    def test_install_without_auto_recover_token(self) -> None:
+        """When auto_recover_token is None, the extra_var is not set."""
+        mock_registry = MagicMock()
+        manager = RunnerManager(registry=mock_registry)
+        mock_executor = MagicMock()
+        manager._executor = mock_executor
+
+        manager.install(
+            "host1",
+            "root",
+            token="tok",
+            gitea_url="https://git.example.com",
+        )
+        assert "gitea_runner_auto_recover_api_token" not in manager._captured_extra_vars
+
    def test_install_missing_gitea_url(self) -> None:
        manager = RunnerManager()
        with pytest.raises(AnsibleError, match="GITEA_URL must be set"):
Author	SHA1	Message	Date
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	cea84b1d89	fix: disable systemd-oomd memory pressure kill for runner users The default systemd user service configuration (10-oomd-user-service-defaults.conf) sets ManagedOOMMemoryPressure=kill with ManagedOOMMemoryPressureLimit=50%. Under parallel DinD load (6 molecule tests), memory pressure exceeds 50% for 30 seconds, causing systemd-oomd to SIGKILL molecule containers (rc=137). This was the root cause of recurring "container is not running" and "Failed to create temporary directory" errors in CI. Fix: Create a per-user systemd override that sets: - ManagedOOMMemoryPressure=auto (don't kill on pressure) - ManagedOOMMemoryPressureLimit=100% (never trigger) - OOMScoreAdjust=-500 (very unlikely to be OOM-killed) The runner users run Docker-in-Docker workloads that legitimately use large amounts of memory. systemd-oomd's aggressive memory pressure killing is inappropriate for CI runner workloads. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 23:09:46 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	645d87a3cf	feat: add debug logging to healthcheck (image count, disk, docker df) Logs image count and docker system df output at each healthcheck run to diagnose "No such image" failures. When the gentle prune removes images, the log shows the image count dropping to 0, making the root cause immediately visible instead of requiring forensics. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 19:11:11 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	e58f4c9da4	fix: stop gentle prune from removing CI runner images The gentle prune (70% disk threshold) was running docker image prune -af --filter "until=1h" every 2 minutes. This removed the CI runner image (ci-full:latest, ~2.3GB) between jobs, causing "No such image" errors when act_runner tried to create job containers with forcePull=false. The host disk normally sits at 72-74%, above the 70% warn threshold, so the gentle prune was running constantly. With the CI image unused between jobs, it was pruned after 1 hour. Fix: remove docker image prune from the gentle path. Only prune containers, volumes, and networks at 70%. Images are only pruned at the 75% critical threshold when disk is genuinely full. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 18:50:41 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	9c786ec4df	revert: restore original healthcheck thresholds (2min, 70%, 75%) The 1min interval + 70% critical threshold caused more problems than it solved. The host disk normally sits at ~74% — just under the original 75% critical threshold. Lowering to 70% triggered aggressive docker system prune -af --volumes every single minute, which: 1. Wiped all cached images, forcing all 6 parallel slots to re-pull 2-3GB each simultaneously — increasing disk pressure, not reducing it 2. Caused network prune to remove networks mid-job → "network not found" errors in Gitea Actions Reverting to 2min/70%warn/75%critical. Images stay cached, only gentle until=1h pruning runs at 70%, and full prune only triggers at 75% (above the normal 74% baseline). The network prune --filter "until=1h" fix from the previous commit is kept — that was a genuine bug fix. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 11:54:41 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	d57d6a3591	fix: use until=1h filter for network prune in critical path The critical disk prune path was doing `docker network prune -f` without a filter, which removes ALL unused networks. This races with Gitea Actions jobs that are mid-startup: the runner creates a network, then the healthcheck prunes it before the container attaches, causing "network not found" errors. Use `--filter "until=1h"` (same as the warn path) to only remove networks older than 1 hour. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 10:34:45 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	c7323d5f10	fix: remove userland-proxy=false (incompatible with rootless Docker) userland-proxy: false requires bridge-nf-call-iptables which is not available in rootless Docker. This caused dockerd to fail with: "cannot restrict inter-container communication or run without the userland proxy: stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory" Keep log-opts (max-size=10m, max-file="3") which works correctly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 09:43:35 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	8047ea6061	fix: quote max-file as string in daemon.json Docker's daemon.json parser expects max-file as a string, not an integer. The unquoted Jinja template rendered "max-file": 3 which caused dockerd to fail with "cannot unmarshal number into Go struct field LogConfig.log-opts of type string". Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 09:37:36 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	3f22910a7d	fix: reduce healthcheck interval to 1min and disk thresholds Runners are consistently running out of disk during parallel molecule tests. The 2min healthcheck interval is too coarse — a single molecule scenario can fill 10+ GB in 2 minutes. Reduce to 1min and lower disk thresholds (60% warn, 70% critical) so pruning kicks in earlier. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 09:04:24 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	c192929e34	fix: tune rootless Docker daemon for parallel DinD reliability Add userland-proxy=false and log-opts (10m, 3 files) to the rootless Docker daemon.json. The userland proxy adds overhead under parallel container creation and can become a bottleneck during CI molecule tests. Log rotation prevents disk exhaustion from container logs under heavy DinD load, which caused the daemon to become unresponsive (context deadline exceeded on /_ping). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 01:19:13 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	20aefa57c0	fix: remove molecule-tests from auto-merge needs to prevent skip cascade CI / validate (pull_request) Successful in 1m45s Details CI / molecule-tests (1) (pull_request) Skipped Details CI / molecule-tests (2) (pull_request) Skipped Details CI / molecule-tests (3) (pull_request) Skipped Details CI / molecule-tests (4) (pull_request) Skipped Details CI / auto-merge (pull_request) Successful in 1m11s Details Gitea Actions skips dependent jobs of skipped jobs without evaluating if: conditions. When molecule-tests is skipped (ansible-changed=false), auto-merge was also skipped because it had molecule-tests in needs. Remove molecule-tests from needs and rely on validate success only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 00:43:31 +02:00
Emil Simeonov	811d07bfe1	docs: trigger CI re-run after PR title fix	2026-08-10 00:43:31 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	3c25fa6fdc	ci: wire devx lint checks into CI and pre-commit hooks - Add 5 lint checks to make lint-all: check-ansible-no-log, check-ansible-no-state-absent-on-db, check-ansible-patterns, check-jinja-expr, check-ansible-set-fact-to-json - Add corresponding pre-commit hooks with file-pattern triggers - Bump devx pin from v0.48.1 to v0.50.0 Closes GRM-161 Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 00:43:31 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	9a33287258	fix(healthcheck): lower critical prune threshold to 75% At 76%, molecule containers fail with "container is not running" because overlay2 runs out of space under parallel DinD load. The gentle prune (until=1h) at 75% was insufficient — runners stayed at 76% because nothing was older than 1h on busy runners. Lower the critical threshold to 75% so the full prune (no until filter) triggers immediately when disk reaches the level where tests fail. Also lower the gentle prune threshold to 70% to start cleanup earlier. This matches the infra CI disk gate which skips tests at 75%+. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 00:43:31 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	7af29f4e3c	feat(healthcheck): add two-tier disk prune with critical threshold The healthcheck previously used a single 75% threshold with an until=1h filter. On busy runners with constant CI jobs, nothing is older than 1h, so the prune barely frees anything — disk stays at ~76% and the CI disk gate (80%) skips all molecule tests. Two-tier approach: - 75-79%: gentle prune (until=1h) — same as before - 80%+: full prune (no until filter) — removes ALL stopped containers and unused images regardless of age, plus builder cache and networks This ensures that when disk pressure is critical, the healthcheck actually reclaims space instead of skipping everything that's recently created. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 00:43:31 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	1c8edb55d8	fix(ci): use pushd/popd instead of molecule --cwd Molecule doesn't support --cwd flag. Use pushd/popd to change directory before running molecule test/destroy. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 00:43:20 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	26a32dcf28	fix(ci): use shell: bash for molecule test step The inline molecule runner uses bashisms (here-strings `<<<`, parameter expansion) that dash/sh doesn't support. Gitea Actions defaults to sh, causing "Syntax error: redirection unexpected". Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 00:43:20 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	aec957f06f	refactor: remove molecule_ci_guard — adds complexity, no value The cross-runner cancellation guard was killing healthy molecule slots when unhealthy slots failed, defeating fail-fast: false. Replace with direct molecule test execution. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 00:43:20 +02:00
gitea-actions-bot	28214de583	chore: update badge URLs to commit `918ffe7d` [skip ci]	2026-08-09 11:16:42 +00:00
grm-ci-bot	7fd023d192	release: v0.20.0 [skip ci]	2026-08-09 11:16:01 +00:00
emilandemo	2ffedaa793	GRM-159: feat(healthcheck): add two-tier disk prune with critical threshold Post-merge / detect-and-configure (push) Successful in 1m38s Details Post-merge / release-and-maintain (push) Successful in 2m26s Details Co-authored-by: emil User <emil.simeonov@tutanota.com>	2026-08-09 11:12:37 +00:00
gitea-actions-bot	e7b2d4e6af	chore: update badge URLs to commit `e0611a67` [skip ci]	2026-08-08 23:39:03 +00:00
grm-ci-bot	bda2af91bc	release: v0.19.0 [skip ci]	2026-08-08 23:38:33 +00:00
emilandemo	c72dc97f63	GRM-158: feat: use Gitea mirror for Ansible collection installs Post-merge / detect-and-configure (push) Successful in 1m8s Details Post-merge / release-and-maintain (push) Successful in 1m9s Details Co-authored-by: emil User <emil.simeonov@tutanota.com>	2026-08-08 23:36:41 +00:00
gitea-actions-bot	db9fb6f162	chore: update badge URLs to commit `d9e3ee2e` [skip ci]	2026-08-06 19:16:49 +00:00
grm-ci-bot	d102453960	release: v0.18.8 [skip ci]	2026-08-06 19:16:18 +00:00
kiretoandemo	42409d9e47	GRM-160: fix: pin containerd.io to compatible version for Docker 28.x Post-merge / detect-and-configure (push) Successful in 3m19s Details Post-merge / release-and-maintain (push) Successful in 1m19s Details Co-authored-by: kireto <kireto@oblachno.com>	2026-08-06 19:12:07 +00:00
gitea-actions-bot	91e880f05c	chore: update badge URLs to commit `7b6c9f92` [skip ci]	2026-08-06 09:31:04 +00:00
grm-ci-bot	2d2eaa3291	release: v0.18.7 [skip ci]	2026-08-06 09:30:18 +00:00
kiretoandemo	8176a62885	GRM-159: fix: move StartLimit to [Unit] and make prune timer reload conditional Post-merge / detect-and-configure (push) Successful in 3m49s Details Post-merge / release-and-maintain (push) Successful in 3m43s Details Co-authored-by: kireto <kireto@oblachno.com>	2026-08-06 09:23:29 +00:00
gitea-actions-bot	443756a508	chore: update badge URLs to commit `36201d0d` [skip ci]	2026-08-06 00:00:05 +00:00
grm-ci-bot	340222e041	release: v0.18.6 [skip ci]	2026-08-05 23:59:22 +00:00
kiretoandemo	179e47bbb2	GRM-158: fix: pre-configure daemon.json before rootless setuptool + add DBUS_SESSION_BUS_ADDRESS Post-merge / detect-and-configure (push) Successful in 1m2s Details Post-merge / release-and-maintain (push) Successful in 1m38s Details Co-authored-by: kireto <kireto@oblachno.com>	2026-08-05 23:57:25 +00:00
gitea-actions-bot	68d16577b3	chore: update badge URLs to commit `169df915` [skip ci]	2026-08-05 20:21:20 +00:00
grm-ci-bot	38607d9f29	release: v0.18.5 [skip ci]	2026-08-05 20:20:43 +00:00
kiretoandemo	90139b306b	GRM-157: fix: pin Docker 28.x + disable containerd snapshotter + tune prune/disk Post-merge / detect-and-configure (push) Successful in 1m8s Details Post-merge / release-and-maintain (push) Successful in 1m31s Details Co-authored-by: kireto <kireto@oblachno.com>	2026-08-05 20:18:38 +00:00
gitea-actions-bot	dd475bec0d	chore: update badge URLs to commit `83a5b577` [skip ci]	2026-08-05 13:53:01 +00:00
grm-ci-bot	0f0ada3576	release: v0.18.4 [skip ci]	2026-08-05 13:52:28 +00:00
emo	185e41c49e	GRM-156: fix: harden rootless Docker daemon resilience on CI runners Post-merge / detect-and-configure (push) Successful in 1m9s Details Post-merge / release-and-maintain (push) Successful in 1m15s Details	2026-08-05 13:50:34 +00:00
gitea-actions-bot	103741b3ab	chore: update badge URLs to commit `7136903b` [skip ci]	2026-08-04 14:04:34 +00:00
grm-ci-bot	b770d1debf	release: v0.18.3 [skip ci]	2026-08-04 14:03:59 +00:00
gitea-adminandemo	2f11489be0	GRM-2: fix: switch default network driver to slirp4netns (pasta TCP RST bug) Post-merge / detect-and-configure (push) Successful in 1m8s Details Post-merge / release-and-maintain (push) Successful in 1m22s Details Co-authored-by: oblachno Admin <admin@oblachno.oblachno.fyi>	2026-08-04 14:01:56 +00:00
gitea-actions-bot	8e9e58fedb	chore: update badge URLs to commit 69e62973 [skip ci]	2026-07-17 02:03:11 +00:00
emil	70d985ebaa	GRM-155: chore: bump devx to v0.47.2 Post-merge / detect-and-configure (push) Waiting to run Details Post-merge / release-and-maintain (push) Waiting to run Details	2026-07-17 02:00:23 +00:00
gitea-actions-bot	66f071b676	chore: update badge URLs to commit 7ee78d01 [skip ci]	2026-07-16 17:42:23 +00:00