fix: runner self-healing — fix service startup order, add Docker restart override
Three fixes for runner reliability:
1. Fix service startup order (install_runner.yml)
- Move service.yml include BEFORE register.yml
- Previously, register.yml tried to start gitea-runner.service
before service.yml created the systemd unit file, causing
"Unit gitea-runner.service not found" on first install
- This only failed when registration actually happened (new runner);
on idempotent re-runs registration was skipped so the bug was masked
2. Restart runner after (re-)registration (register.yml)
- Add "Restart runner service after (re-)registration" task
- When a runner is re-registered (force-reregister or auto-recovery),
the service must be restarted to pick up the new .runner file
- The old premature start task was already removed; this adds it back
in the correct position (after service.yml has created the unit)
3. Docker daemon restart override (rootless_docker.yml)
- Add Restart=always, RestartSec=5, StartLimitIntervalSec=300,
StartLimitBurst=10 to both Docker service override blocks
- Upstream dockerd-rootless-setuptool.sh defaults to
StartLimitBurst=3 in 60s — too aggressive for OOM-prone hosts
- If Docker crashes 3 times in a minute, it stops restarting
permanently, taking the runner down with it
- New settings allow 10 restarts in 5 minutes, matching the
runner service's own restart policy
Note: Auto-recovery (--auto-recover-token) already exists in the CLI
and healthcheck script but defaults to disabled. Operators should pass
--auto-recover-token <token> during grm install to enable automatic
re-registration when runners become unregistered.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
parent
d16f53dfd7
commit
fe9f66afb5
@@ -13,9 +13,9 @@
|
||||
- name: Include validation
|
||||
ansible.builtin.include_tasks: validate.yml
|
||||
|
||||
- name: Include service setup
|
||||
ansible.builtin.include_tasks: service.yml
|
||||
|
||||
- name: Include registration
|
||||
ansible.builtin.include_tasks: register.yml
|
||||
when: not gitea_runner_skip_registration
|
||||
|
||||
- name: Include service setup
|
||||
ansible.builtin.include_tasks: service.yml
|
||||
|
||||
@@ -143,6 +143,10 @@
|
||||
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
|
||||
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
|
||||
{% endif %}
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=10
|
||||
mode: "0644"
|
||||
owner: "{{ gitea_runner_service_user }}"
|
||||
group: "{{ gitea_runner_service_user }}"
|
||||
@@ -248,6 +252,10 @@
|
||||
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
|
||||
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
|
||||
{% endif %}
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=10
|
||||
mode: "0644"
|
||||
owner: "{{ gitea_runner_service_user }}"
|
||||
group: "{{ gitea_runner_service_user }}"
|
||||
|
||||
Reference in New Issue
Block a user