GRM-163: Fix systemctl --user tasks in gitea_runner role using runuser #255

Merged
emo merged 5 commits from GRM-163-runner-self-healing-v2 into master 2026-08-24 03:15:19 +00:00
Owner

Summary

Fix the gitea_runner role to use runuser for systemctl --user invocations instead of become: true, which was causing silent failures of systemd user timers.

Changes

  • Use runuser -u <user> -- env XDG_RUNTIME_DIR=... DBUS_SESSION_BUS_ADDRESS=... systemctl --user ... pattern in prune.yml, healthcheck.yml, service.yml, rootless_docker.yml, and register.yml
  • Fix service startup order — service start is handled by service.yml (included after register.yml)
  • Add Docker restart override for runner self-healing
  • Pin devx to v0.50.1 (v0.50.5 tag does not exist)
  • Remove non-existent wait_for_checks step from auto-merge workflow

Verification

  • All 6 molecule tests pass
  • CI validate passes

Closes GRM-163

## Summary Fix the gitea_runner role to use `runuser` for `systemctl --user` invocations instead of `become: true`, which was causing silent failures of systemd user timers. ## Changes - Use `runuser -u <user> -- env XDG_RUNTIME_DIR=... DBUS_SESSION_BUS_ADDRESS=... systemctl --user ...` pattern in prune.yml, healthcheck.yml, service.yml, rootless_docker.yml, and register.yml - Fix service startup order — service start is handled by service.yml (included after register.yml) - Add Docker restart override for runner self-healing - Pin devx to v0.50.1 (v0.50.5 tag does not exist) - Remove non-existent `wait_for_checks` step from auto-merge workflow ## Verification - All 6 molecule tests pass - CI validate passes Closes GRM-163
emo reviewed 2026-08-24 01:45:05 +00:00
emo left a comment
Owner

Automated PR Review

  • Architecture compliance: OK
  • Best practices: OK
  • Security: OK
  • i18n: OK
  • Resource management: OK
  • Documentation: OK
  • Tests: OK
  • Commit conventions: OK

No issues found by automated checks.


Auto-merge: If all CI checks pass, this PR will be merged automatically.

## Automated PR Review - Architecture compliance: OK - Best practices: OK - Security: OK - i18n: OK - Resource management: OK - Documentation: OK - Tests: OK - Commit conventions: OK No issues found by automated checks. --- **Auto-merge:** If all CI checks pass, this PR will be merged automatically.
emo approved these changes 2026-08-24 01:52:29 +00:00
Dismissed
emo left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
emil added the ready-to-merge label 2026-08-24 01:57:23 +00:00
emo reviewed 2026-08-24 01:59:16 +00:00
emo left a comment
Owner

Automated PR Review

  • Architecture compliance: OK
  • Best practices: OK
  • Security: OK
  • i18n: OK
  • Resource management: OK
  • Documentation: OK
  • Tests: OK
  • Commit conventions: OK

No issues found by automated checks.


Auto-merge: If all CI checks pass, this PR will be merged automatically.

## Automated PR Review - Architecture compliance: OK - Best practices: OK - Security: OK - i18n: OK - Resource management: OK - Documentation: OK - Tests: OK - Commit conventions: OK No issues found by automated checks. --- **Auto-merge:** If all CI checks pass, this PR will be merged automatically.
emo approved these changes 2026-08-24 02:07:31 +00:00
Dismissed
emo left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
emo reviewed 2026-08-24 02:12:41 +00:00
emo left a comment
Owner

Automated PR Review

  • Architecture compliance: OK
  • Best practices: OK
  • Security: OK
  • i18n: OK
  • Resource management: OK
  • Documentation: OK
  • Tests: OK
  • Commit conventions: OK

No issues found by automated checks.


Auto-merge: If all CI checks pass, this PR will be merged automatically.

## Automated PR Review - Architecture compliance: OK - Best practices: OK - Security: OK - i18n: OK - Resource management: OK - Documentation: OK - Tests: OK - Commit conventions: OK No issues found by automated checks. --- **Auto-merge:** If all CI checks pass, this PR will be merged automatically.
emo approved these changes 2026-08-24 02:18:11 +00:00
Dismissed
emo left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
emil force-pushed GRM-163-runner-self-healing-v2 from 477d425745 to 56f53cb32e 2026-08-24 02:24:05 +00:00 Compare
emo reviewed 2026-08-24 02:25:43 +00:00
emo left a comment
Owner

Automated PR Review

  • Architecture compliance: OK
  • Best practices: OK
  • Security: OK
  • i18n: OK
  • Resource management: OK
  • Documentation: OK
  • Tests: OK
  • Commit conventions: OK

No issues found by automated checks.


Auto-merge: If all CI checks pass, this PR will be merged automatically.

## Automated PR Review - Architecture compliance: OK - Best practices: OK - Security: OK - i18n: OK - Resource management: OK - Documentation: OK - Tests: OK - Commit conventions: OK No issues found by automated checks. --- **Auto-merge:** If all CI checks pass, this PR will be merged automatically.
emo approved these changes 2026-08-24 02:31:16 +00:00
Dismissed
emo left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
emil added 3 commits 2026-08-24 02:36:49 +00:00
The become_user pattern with sudo doesn't reliably connect to the
user's D-Bus session, causing systemd user timers (docker-prune,
runner-healthcheck) to silently fail to enable. This led to stale
Docker images accumulating and filling the disk on CI runners.

Switch all systemctl --user invocations to use runuser with explicit
XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS environment variables,
which correctly connects to the user's systemd session.

Affected tasks: prune timer, healthcheck timer, gitea-runner service,
rootless Docker daemon start/enable/restart, runner service restart.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Three fixes for runner reliability:

1. Fix service startup order (install_runner.yml)
   - Move service.yml include BEFORE register.yml
   - Previously, register.yml tried to start gitea-runner.service
     before service.yml created the systemd unit file, causing
     "Unit gitea-runner.service not found" on first install
   - This only failed when registration actually happened (new runner);
     on idempotent re-runs registration was skipped so the bug was masked

2. Restart runner after (re-)registration (register.yml)
   - Add "Restart runner service after (re-)registration" task
   - When a runner is re-registered (force-reregister or auto-recovery),
     the service must be restarted to pick up the new .runner file
   - The old premature start task was already removed; this adds it back
     in the correct position (after service.yml has created the unit)

3. Docker daemon restart override (rootless_docker.yml)
   - Add Restart=always, RestartSec=5, StartLimitIntervalSec=300,
     StartLimitBurst=10 to both Docker service override blocks
   - Upstream dockerd-rootless-setuptool.sh defaults to
     StartLimitBurst=3 in 60s — too aggressive for OOM-prone hosts
   - If Docker crashes 3 times in a minute, it stops restarting
     permanently, taking the runner down with it
   - New settings allow 10 restarts in 5 minutes, matching the
     runner service's own restart policy

Note: Auto-recovery (--auto-recover-token) already exists in the CLI
and healthcheck script but defaults to disabled. Operators should pass
--auto-recover-token <token> during grm install to enable automatic
re-registration when runners become unregistered.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fix: pin devx to v0.50.1 (v0.50.5 tag does not exist)
CI / validate (pull_request) Successful in 3m20s
CI / molecule-tests (1) (pull_request) Successful in 4m22s
CI / auto-merge (pull_request) Failing after 1m6s
CI / molecule-tests (4) (pull_request) Successful in 6m4s
CI / molecule-tests (3) (pull_request) Successful in 8m42s
CI / molecule-tests (2) (pull_request) Successful in 9m3s
150d3267d1
emil force-pushed GRM-163-runner-self-healing-v2 from 56f53cb32e to 150d3267d1 2026-08-24 02:36:49 +00:00 Compare
emil dismissed emo's review 2026-08-24 02:36:49 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

emo reviewed 2026-08-24 02:40:02 +00:00
emo left a comment
Owner

Automated PR Review

  • Architecture compliance: OK
  • Best practices: OK
  • Security: OK
  • i18n: OK
  • Resource management: OK
  • Documentation: WARNING — Ansible role changed but no docs updated
  • Tests: OK
  • Commit conventions: OK

No issues found by automated checks.


Auto-merge: If all CI checks pass, this PR will be merged automatically.

## Automated PR Review - Architecture compliance: OK - Best practices: OK - Security: OK - i18n: OK - Resource management: OK - Documentation: WARNING — Ansible role changed but no docs updated - Tests: OK - Commit conventions: OK No issues found by automated checks. --- **Auto-merge:** If all CI checks pass, this PR will be merged automatically.
emo approved these changes 2026-08-24 02:45:37 +00:00
Dismissed
emo left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
emil added 1 commit 2026-08-24 02:50:27 +00:00
chore: re-trigger auto-merge
CI / validate (pull_request) Successful in 1m37s
CI / molecule-tests (1) (pull_request) Successful in 4m27s
CI / auto-merge (pull_request) Failing after 1m6s
CI / molecule-tests (3) (pull_request) Successful in 5m53s
CI / molecule-tests (2) (pull_request) Successful in 9m10s
CI / molecule-tests (4) (pull_request) Successful in 9m13s
4083835ca3
emo reviewed 2026-08-24 02:52:03 +00:00
emo left a comment
Owner

Automated PR Review

  • Architecture compliance: OK
  • Best practices: OK
  • Security: OK
  • i18n: OK
  • Resource management: OK
  • Documentation: WARNING — Ansible role changed but no docs updated
  • Tests: OK
  • Commit conventions: OK

No issues found by automated checks.


Auto-merge: If all CI checks pass, this PR will be merged automatically.

## Automated PR Review - Architecture compliance: OK - Best practices: OK - Security: OK - i18n: OK - Resource management: OK - Documentation: WARNING — Ansible role changed but no docs updated - Tests: OK - Commit conventions: OK No issues found by automated checks. --- **Auto-merge:** If all CI checks pass, this PR will be merged automatically.
emo approved these changes 2026-08-24 02:57:38 +00:00
Dismissed
emo left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
emil added 1 commit 2026-08-24 03:04:14 +00:00
fix: wait for molecule tests before auto-merge attempt
CI / validate (pull_request) Successful in 1m39s
CI / molecule-tests (3) (pull_request) Successful in 6m9s
CI / molecule-tests (2) (pull_request) Successful in 6m18s
CI / molecule-tests (1) (pull_request) Successful in 6m52s
CI / molecule-tests (4) (pull_request) Successful in 8m51s
CI / auto-merge (pull_request) Successful in 3m11s
8b751e4eff
emil dismissed emo's review 2026-08-24 03:04:14 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

emo reviewed 2026-08-24 03:05:50 +00:00
emo left a comment
Owner

Automated PR Review

  • Architecture compliance: OK
  • Best practices: OK
  • Security: OK
  • i18n: OK
  • Resource management: OK
  • Documentation: WARNING — Ansible role changed but no docs updated
  • Tests: OK
  • Commit conventions: OK

No issues found by automated checks.


Auto-merge: If all CI checks pass, this PR will be merged automatically.

## Automated PR Review - Architecture compliance: OK - Best practices: OK - Security: OK - i18n: OK - Resource management: OK - Documentation: WARNING — Ansible role changed but no docs updated - Tests: OK - Commit conventions: OK No issues found by automated checks. --- **Auto-merge:** If all CI checks pass, this PR will be merged automatically.
emo approved these changes 2026-08-24 03:13:10 +00:00
emo left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
emo merged commit f672da1753 into master 2026-08-24 03:15:19 +00:00
emo deleted branch GRM-163-runner-self-healing-v2 2026-08-24 03:15:20 +00:00
Sign in to join this conversation.