fix: runner self-healing — fix service startup order, add Docker restart override

Three fixes for runner reliability:

1. Fix service startup order (install_runner.yml)
   - Move service.yml include BEFORE register.yml
   - Previously, register.yml tried to start gitea-runner.service
     before service.yml created the systemd unit file, causing
     "Unit gitea-runner.service not found" on first install
   - This only failed when registration actually happened (new runner);
     on idempotent re-runs registration was skipped so the bug was masked

2. Restart runner after (re-)registration (register.yml)
   - Add "Restart runner service after (re-)registration" task
   - When a runner is re-registered (force-reregister or auto-recovery),
     the service must be restarted to pick up the new .runner file
   - The old premature start task was already removed; this adds it back
     in the correct position (after service.yml has created the unit)

3. Docker daemon restart override (rootless_docker.yml)
   - Add Restart=always, RestartSec=5, StartLimitIntervalSec=300,
     StartLimitBurst=10 to both Docker service override blocks
   - Upstream dockerd-rootless-setuptool.sh defaults to
     StartLimitBurst=3 in 60s — too aggressive for OOM-prone hosts
   - If Docker crashes 3 times in a minute, it stops restarting
     permanently, taking the runner down with it
   - New settings allow 10 restarts in 5 minutes, matching the
     runner service's own restart policy

Note: Auto-recovery (--auto-recover-token) already exists in the CLI
and healthcheck script but defaults to disabled. Operators should pass
--auto-recover-token <token> during grm install to enable automatic
re-registration when runners become unregistered.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
Emil Simeonov
2026-08-17 18:45:39 +02:00
co-authored by Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
parent df271411fa
commit 24d096d802
3 changed files with 24 additions and 6 deletions
@@ -13,9 +13,9 @@
- name: Include validation
ansible.builtin.include_tasks: validate.yml
- name: Include service setup
ansible.builtin.include_tasks: service.yml
- name: Include registration
ansible.builtin.include_tasks: register.yml
when: not gitea_runner_skip_registration
- name: Include service setup
ansible.builtin.include_tasks: service.yml
+13 -3
View File
@@ -44,6 +44,16 @@
('already exists' not in gitea_runner_register_output.stdout | default(''))
timeout: 60
# Note: service start is handled by service.yml (included after register.yml
# in install_runner.yml). Starting here fails because the systemd unit file
# has not been created yet.
- name: Restart runner service after (re-)registration
ansible.builtin.command: systemctl --user restart gitea-runner
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
- gitea_runner_docker_rootless_setup
- gitea_runner_register_output is defined
- gitea_runner_register_output.rc | default(1) == 0
@@ -143,6 +143,10 @@
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
{% endif %}
Restart=always
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=10
mode: "0644"
owner: "{{ gitea_runner_service_user }}"
group: "{{ gitea_runner_service_user }}"
@@ -254,6 +258,10 @@
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
{% endif %}
Restart=always
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=10
mode: "0644"
owner: "{{ gitea_runner_service_user }}"
group: "{{ gitea_runner_service_user }}"