Compare commits

..
Author SHA1 Message Date
Emil Simeonov 64240b4f51 fix: remove non-existent wait_for_checks step from auto-merge
CI / validate (pull_request) Successful in 1m42s
CI / molecule-tests (1) (pull_request) Successful in 4m20s
CI / auto-merge (pull_request) Failing after 1m7s
CI / molecule-tests (3) (pull_request) Successful in 5m59s
CI / molecule-tests (2) (pull_request) Successful in 9m21s
CI / molecule-tests (4) (pull_request) Successful in 9m31s
2026-08-24 02:25:43 +02:00
Emil Simeonov 20951179a9 fix: pin devx to v0.50.1 (v0.50.5 tag does not exist)
CI / molecule-tests (1) (pull_request) Successful in 4m23s
CI / auto-merge (pull_request) Failing after 1m4s
CI / validate (pull_request) Successful in 3m23s
CI / molecule-tests (4) (pull_request) Successful in 6m6s
CI / molecule-tests (2) (pull_request) Successful in 9m3s
CI / molecule-tests (3) (pull_request) Successful in 9m31s
2026-08-24 02:11:52 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> 9718acb3d8 fix: use runuser for systemctl --user tasks in gitea_runner role
CI / validate (pull_request) Failing after 19s
CI / molecule-tests (1) (pull_request) Skipped
CI / molecule-tests (2) (pull_request) Skipped
CI / molecule-tests (3) (pull_request) Skipped
CI / molecule-tests (4) (pull_request) Skipped
CI / auto-merge (pull_request) Skipped
The become_user pattern with sudo doesn't reliably connect to the
user's D-Bus session, causing systemd user timers (docker-prune,
runner-healthcheck) to silently fail to enable. This led to stale
Docker images accumulating and filling the disk on CI runners.

Switch all systemctl --user invocations to use runuser with explicit
XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS environment variables,
which correctly connects to the user's systemd session.

Affected tasks: prune timer, healthcheck timer, gitea-runner service,
rootless Docker daemon start/enable/restart, runner service restart.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-23 20:20:36 +02:00
Emil SimeonovandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> 24d096d802 fix: runner self-healing — fix service startup order, add Docker restart override
Three fixes for runner reliability:

1. Fix service startup order (install_runner.yml)
   - Move service.yml include BEFORE register.yml
   - Previously, register.yml tried to start gitea-runner.service
     before service.yml created the systemd unit file, causing
     "Unit gitea-runner.service not found" on first install
   - This only failed when registration actually happened (new runner);
     on idempotent re-runs registration was skipped so the bug was masked

2. Restart runner after (re-)registration (register.yml)
   - Add "Restart runner service after (re-)registration" task
   - When a runner is re-registered (force-reregister or auto-recovery),
     the service must be restarted to pick up the new .runner file
   - The old premature start task was already removed; this adds it back
     in the correct position (after service.yml has created the unit)

3. Docker daemon restart override (rootless_docker.yml)
   - Add Restart=always, RestartSec=5, StartLimitIntervalSec=300,
     StartLimitBurst=10 to both Docker service override blocks
   - Upstream dockerd-rootless-setuptool.sh defaults to
     StartLimitBurst=3 in 60s — too aggressive for OOM-prone hosts
   - If Docker crashes 3 times in a minute, it stops restarting
     permanently, taking the runner down with it
   - New settings allow 10 restarts in 5 minutes, matching the
     runner service's own restart policy

Note: Auto-recovery (--auto-recover-token) already exists in the CLI
and healthcheck script but defaults to disabled. Operators should pass
--auto-recover-token <token> during grm install to enable automatic
re-registration when runners become unregistered.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-17 18:45:39 +02:00
gitea-actions-bot df271411fa chore: update badge URLs to commit e96a599d [skip ci] 2026-08-12 22:43:40 +00:00
emil 8b256ceef3 GRM-162: refactor: convert workflows to use composite actions 2026-08-12 22:41:25 +00:00
emil 6cea042330 GRM-161: Fix register.yml service start and bump devx to v0.50.5 (#238)
- Fix register.yml premature service start (removed systemctl start before unit file exists)
- Replace broken inline Python in ci.yml auto-merge with devx.ci.wait_for_checks
- Bump devx from v0.50.0 to v0.50.5

GRM-161
2026-08-12 22:35:23 +00:00
gitea-actions-bot d1f38243d6 chore: update badge URLs to commit 35ed01b7 [skip ci] 2026-08-12 13:53:53 +00:00
emil 8a85b2479f GRM-160: fix: wait for molecule tests before auto-merge attempt 2026-08-12 13:51:47 +00:00
emil 3686ab5170 GRM-159: Docker daemon hardening, healthcheck improvements, and devx lint checks 2026-08-12 13:45:18 +00:00
gitea-actions-bot d393597a76 chore: update badge URLs to commit 3720ac3b [skip ci] 2026-08-07 10:34:46 +00:00
grm-ci-bot 6eba032ec2 release: v0.18.3 [skip ci] 2026-08-07 10:34:08 +00:00
gitea-admin fd351714bb GRM-156: fix: switch default network driver to slirp4netns (pasta TCP RST bug)
Co-authored-by: oblachno Admin <admin@oblachno.oblachno.fyi>
2026-08-07 10:27:28 +00:00
26 changed files with 540 additions and 307 deletions
+47
View File
@@ -0,0 +1,47 @@
name: 'Notify on failure'
description: 'Create a Gitea issue when a CI workflow fails (calls devx.ci.notify_failure)'
# Composite action for the common "Notify on failure" step pattern.
# Replaces the repeated inline:
# - name: Notify on failure
# if: failure()
# env:
# CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
# run: |
# . .venv/bin/activate 2>/dev/null || true
# export PATH="$HOME/.local/bin:$PATH"
# python3 -m devx.ci.notify_failure \
# --repo "${{ github.repository }}" \
# --run-id "${{ github.run_id }}" \
# --workflow "ci/validate" \
# --commit "${{ github.sha }}" \
# --auto-login
#
# Gitea 1.27 notes:
# - `if: failure()` is evaluated in the calling workflow's context and
# propagates correctly to composite action steps.
# - `secrets` are not accessible here; the calling workflow's top-level
# `env:` CI_GITEA_API_TOKEN is used via `${{ env.* }}`.
inputs:
workflow:
description: 'Workflow/job name used in the Gitea issue title (e.g., ci/validate)'
required: true
runs:
using: 'composite'
steps:
- name: Notify on failure
if: failure()
shell: bash
env:
CI_GITEA_API_TOKEN: ${{ env.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "${{ inputs.workflow }}" \
--commit "${{ github.sha }}" \
--auto-login
+89
View File
@@ -0,0 +1,89 @@
name: 'Quality checks'
description: 'Run lint, unit tests with coverage, test speed, docs, translations, and security scan'
# Composite action for the 6-step quality check sequence used by the
# devx validate job. Replaces the inline block:
# - Lint all
# - Unit tests with 100% coverage
# - Check unit test speed
# - Documentation gate (coverage + stale refs + lint + version refs + prose)
# - Translation completeness check
# - Dependency security scan
#
# Each step activates the venv defensively (`. .venv/bin/activate 2>/dev/null
# || true`) so the action works whether or not the setup step created a
# venv at the repo root (pre-built CI images symlink /opt/venv to .venv).
#
# Gitea 1.27 notes:
# - Every `run` step needs explicit `shell:`.
# - Inputs are string-typed; numeric thresholds are passed through as
# strings to `devx.tools.check_test_speed`.
inputs:
package:
description: 'Package name for doc version checks (e.g., devx, grm). Empty = no DEVX_DOC_VERSIONS_PKG override.'
required: false
default: ''
test-speed-max:
description: 'Max total test seconds (passed to check_test_speed --max-seconds)'
required: false
default: '15'
test-speed-max-single:
description: 'Max single test seconds (passed to check_test_speed --max-single-seconds)'
required: false
default: '0.5'
translations-file:
description: 'Path to translations.json (empty = default location src/devx/translations.json)'
required: false
default: ''
runs:
using: 'composite'
steps:
- name: Lint all
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
make lint-all
- name: Unit tests with 100% coverage
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
make pytest-cov
- name: Check unit test speed
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
python3 -m devx.tools.check_test_speed \
--max-seconds "${{ inputs.test-speed-max }}" \
--max-single-seconds "${{ inputs.test-speed-max-single }}"
- name: Documentation gate (coverage + stale refs + lint + version refs + prose)
shell: bash
env:
DEVX_DOC_COVERAGE_STRICT: "1"
DEVX_VALE_LEVEL: warning
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
if [ -n "${{ inputs.package }}" ]; then
export DEVX_DOC_VERSIONS_PKG="${{ inputs.package }}"
fi
make devx-docs-check
- name: Translation completeness check
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
if [ -n "${{ inputs.translations-file }}" ]; then
python3 -m devx.ci.check_translations --translations "${{ inputs.translations-file }}"
else
python3 -m devx.ci.check_translations
fi
- name: Dependency security scan
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
# Install pip in venv if missing (needed by pip-audit)
.venv/bin/python -m ensurepip 2>/dev/null || true
PIPAPI_PYTHON_LOCATION=$PWD/.venv/bin/python \
pip-audit --desc --skip-editable 2>&1 || true
+37
View File
@@ -0,0 +1,37 @@
name: 'Set up environment'
description: 'Set up CI environment with venv and PATH (calls make setup-image)'
# Composite action for the common "Set up environment" step pattern.
# Replaces the repeated inline:
# - name: Set up environment
# env:
# CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
# run: make setup-image
#
# Gitea 1.27 notes:
# - Every `run` step needs explicit `shell:`.
# - Composite actions cannot access `secrets` directly; they read from
# the `env:` context which the calling workflow must populate.
# - The calling workflow's top-level `env:` block (CI_GITEA_API_TOKEN,
# CI_GITEA_USERNAME) is visible here via `${{ env.* }}`.
inputs:
extras:
description: 'Extra pip install groups passed to make setup-image (e.g., ci,lint,release)'
required: false
default: ''
runs:
using: 'composite'
steps:
- name: Set up environment
shell: bash
env:
CI_GITEA_API_TOKEN: ${{ env.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ env.CI_GITEA_USERNAME }}
run: |
if [ -n "${{ inputs.extras }}" ]; then
make setup-image EXTRAS="${{ inputs.extras }}"
else
make setup-image
fi
+75 -73
View File
@@ -18,7 +18,11 @@ jobs:
# Saves ~5x checkout+setup overhead vs 6 separate jobs.
validate:
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
credentials:
username: ${{ vars.CI_GITEA_USERNAME }}
password: ${{ secrets.CI_GITEA_API_TOKEN }}
timeout-minutes: 15
defaults:
run:
@@ -32,45 +36,14 @@ jobs:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
run: make setup-image EXTRAS=ci,lint
# --- quality steps ---
- name: Lint all
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
make lint-all
- name: Unit tests with 100% coverage
run: |
. .venv/bin/activate 2>/dev/null || true
make pytest-cov
- name: Documentation gate (coverage + stale refs + lint + version refs + prose)
env:
DEVX_DOC_COVERAGE_STRICT: "1"
DEVX_DOC_VERSIONS_PKG: grm
DEVX_VALE_LEVEL: warning
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
make devx-docs-check
- name: Translation completeness check
run: |
. .venv/bin/activate 2>/dev/null || true
python3 -m devx.ci.check_translations --translations src/grm/translations.json
- name: Check unit test speed
run: |
. .venv/bin/activate 2>/dev/null || true
python3 -m devx.tools.check_test_speed --max-seconds 4 --max-single-seconds 0.5
- name: Dependency security scan
run: |
. .venv/bin/activate 2>/dev/null || true
# Install pip in venv if missing (needed by pip-audit)
.venv/bin/python -m ensurepip 2>/dev/null || true
PIPAPI_PYTHON_LOCATION=$PWD/.venv/bin/python \
pip-audit --desc --skip-editable 2>&1 || true
- uses: ./.gitea/actions/setup-env
with:
extras: "ci,lint"
- uses: ./.gitea/actions/quality-checks
with:
package: grm
test-speed-max: "4"
translations-file: src/grm/translations.json
- name: Workflow dry-run validation
run: |
. .venv/bin/activate 2>/dev/null || true
@@ -140,24 +113,19 @@ jobs:
--owner "${{ github.repository_owner }}" \
--repo "${{ github.event.repository.name }}" \
--github-output
- name: Notify on failure
if: failure()
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure --auto-login \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "ci/validate" \
--commit "${{ github.sha }}"
- uses: ./.gitea/actions/notify-failure
with:
workflow: "ci/validate"
molecule-tests:
needs: [validate]
if: needs.validate.outputs.ansible-changed == 'true'
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
credentials:
username: ${{ vars.CI_GITEA_USERNAME }}
password: ${{ secrets.CI_GITEA_API_TOKEN }}
timeout-minutes: 15
strategy:
fail-fast: false
@@ -200,16 +168,12 @@ jobs:
fi
- name: Run molecule tests
if: env.SKIP != 'true' && steps.prune.outputs.should-run != 'false'
shell: bash
env:
GITEA_URL: ${{ github.server_url }}
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
RUN_ID: ${{ github.run_id }}
ANSIBLE_INJECT_INVOCATION: "1"
JOB_NAME: ${{ github.job }}
MATRIX_INDEX: ${{ matrix.runner-index }}
GITEA_REPOSITORY: ${{ github.repository }}
DOCKER_HOST: unix:///var/run/docker.sock
ANSIBLE_INJECT_INVOCATION: "1"
run: |
. .venv/bin/activate 2>/dev/null || true
if [ -z "$TEST_PAIRS" ]; then exit 0; fi
@@ -220,21 +184,61 @@ jobs:
_TOKEN="$CI_GITEA_API_TOKEN"; [ -z "$_TOKEN" ] && _TOKEN="$CI_GITEA_TOKEN"
[ -z "$_TOKEN" ] && { echo "Gitea API token not set — skipping Docker login"; exit 0; }
echo "$_TOKEN" | docker login git.oblachno.oblachno.fyi -u "$CI_GITEA_USERNAME" --password-stdin
# shellcheck disable=SC2086 # intentional word splitting for argument expansion
python3 -m devx.molecule.molecule_ci_guard $TEST_PAIRS
# Run each molecule test pair sequentially.
# Pairs are 4-part: scenario|platform_name|platform_image|platform_command
# Spaces in platform_command are encoded as __SPACE__.
role_dir="ansible/roles/gitea_runner"
# shellcheck disable=SC2086 # intentional word splitting for pair list
for pair in $TEST_PAIRS; do
IFS='|' read -r scenario platform_name platform_image platform_command <<< "$pair"
platform_command="${platform_command//__SPACE__/ }"
export MOLECULE_PLATFORM_NAME="$platform_name"
export MOLECULE_PLATFORM_IMAGE="$platform_image"
if [ -n "$platform_command" ]; then
export MOLECULE_PLATFORM_COMMAND="$platform_command"
else
unset MOLECULE_PLATFORM_COMMAND
fi
export ANSIBLE_ALLOW_BROKEN_CONDITIONALS=true
echo "--- Running: $scenario on $platform_name ---"
pushd "$role_dir" >/dev/null
if [ "$scenario" = "default" ]; then
molecule test || {
echo "FAILED: $pair — running molecule destroy"
molecule destroy 2>/dev/null || true
popd >/dev/null
exit 1
}
else
molecule test -s "$scenario" || {
echo "FAILED: $pair — running molecule destroy"
molecule destroy -s "$scenario" 2>/dev/null || true
popd >/dev/null
exit 1
}
fi
popd >/dev/null
echo "PASSED: $pair"
docker system prune -af --volumes 2>/dev/null || true
done
echo "All molecule tests passed."
auto-merge:
# Auto-merge runs after validate + molecule-tests pass (or molecule is skipped).
# Uses always() so it evaluates even when molecule-tests is skipped
# (Gitea Actions skips dependent jobs of skipped jobs by default).
needs: [validate, molecule-tests]
# Auto-merge runs after validate passes. molecule-tests is NOT in needs
# because Gitea Actions skips dependent jobs of skipped jobs without
# evaluating if: conditions — having molecule-tests in needs would
# cascade the skip to auto-merge when ansible-changed=false.
needs: [validate]
if: >-
always() &&
github.event_name == 'pull_request' &&
needs.validate.result == 'success' &&
(needs.molecule-tests.result == 'success' || needs.molecule-tests.result == 'skipped')
needs.validate.result == 'success'
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
credentials:
username: ${{ vars.CI_GITEA_USERNAME }}
password: ${{ secrets.CI_GITEA_API_TOKEN }}
timeout-minutes: 10
defaults:
run:
@@ -244,11 +248,9 @@ jobs:
with:
fetch-depth: 0
token: ${{ secrets.CI_GITEA_API_TOKEN }}
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
run: make setup-image EXTRAS=ci
- uses: ./.gitea/actions/setup-env
with:
extras: "ci"
- name: Post approval review
env:
REVIEWER_GITEA_API_TOKEN: ${{ secrets.REVIEWER_GITEA_API_TOKEN }}
+25 -35
View File
@@ -36,7 +36,11 @@ env:
jobs:
detect-and-configure:
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
credentials:
username: ${{ vars.CI_GITEA_USERNAME }}
password: ${{ secrets.CI_GITEA_API_TOKEN }}
timeout-minutes: 10
defaults:
run:
@@ -49,11 +53,9 @@ jobs:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
run: make setup-image EXTRAS=ci
- uses: ./.gitea/actions/setup-env
with:
extras: "ci"
- name: Ensure branch protection and labels
env:
DEVX_REPO_NAME: grm
@@ -87,24 +89,19 @@ jobs:
--base "HEAD~1" \
--head "HEAD" \
--github-output
- name: Notify on failure
if: failure()
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure --auto-login \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "post-merge/detect-and-configure" \
--commit "${{ github.sha }}"
- uses: ./.gitea/actions/notify-failure
with:
workflow: "post-merge/detect-and-configure"
release-and-maintain:
needs: [detect-and-configure]
if: always() && needs.detect-and-configure.result == 'success'
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
credentials:
username: ${{ vars.CI_GITEA_USERNAME }}
password: ${{ secrets.CI_GITEA_API_TOKEN }}
timeout-minutes: 15
outputs:
tag: ${{ steps.release-tag.outputs.tag }}
@@ -117,20 +114,22 @@ jobs:
fetch-depth: 0
ref: master
token: ${{ secrets.CI_GITEA_API_TOKEN }}
- name: Set up environment
- uses: ./.gitea/actions/setup-env
with:
extras: "ci,lint"
- name: Configure git
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
run: make setup-image EXTRAS=ci,lint
- name: Configure git
run: |
git config user.name "grm-ci-bot"
git config user.email "grm-ci-bot@oblachno.fyi"
git remote set-url origin "https://grm-ci-bot:${CI_GITEA_API_TOKEN}@git.oblachno.oblachno.fyi/oblachno-oss/grm.git"
# --- release + publish (only if not a release commit) ---
- name: Run release
id: release-tag
if: needs.detect-and-configure.outputs.is-release == 'false' && needs.detect-and-configure.outputs.user-facing-changed == 'true'
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
DEVX_VERSION_FILE: src/grm/__init__.py
DEVX_TASK_PREFIX: GRM
DEVX_VIKUNJA_PROJECT_ID: 6
@@ -177,15 +176,6 @@ jobs:
git fetch origin master
git reset --hard origin/master
python3 -m devx.ci.push_badges
- name: Notify on failure
if: failure()
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure --auto-login \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "post-merge/release-and-maintain" \
--commit "${{ github.sha }}"
- uses: ./.gitea/actions/notify-failure
with:
workflow: "post-merge/release-and-maintain"
+40
View File
@@ -96,3 +96,43 @@ repos:
types: [python]
pass_filenames: false
stages: [pre-push]
- id: check-ansible-no-log
name: ansible no_log on secret tasks
entry: make check-ansible-no-log
language: system
files: ^ansible/.*\.(yml|yaml)$
pass_filenames: false
stages: [pre-commit]
- id: check-ansible-no-state-absent-on-db
name: no state absent on DB paths
entry: make check-ansible-no-state-absent-on-db
language: system
files: ^ansible/.*\.(yml|yaml)$
pass_filenames: false
stages: [pre-commit]
- id: check-ansible-patterns
name: ansible failure-masking patterns
entry: make check-ansible-patterns
language: system
files: ^ansible/.*\.(yml|yaml)$
pass_filenames: false
stages: [pre-commit]
- id: check-jinja-expr
name: jinja2 expression validation
entry: make check-jinja-expr
language: system
files: ^ansible/.*\.(yml|yaml|j2)$
pass_filenames: false
stages: [pre-commit]
- id: check-ansible-set-fact-to-json
name: set_fact to_json misuse check
entry: make check-ansible-set-fact-to-json
language: system
files: ^ansible/.*\.(yml|yaml)$
pass_filenames: false
stages: [pre-commit]
+3 -3
View File
@@ -278,9 +278,9 @@ via `[tool.devx.classify]` in `pyproject.toml`.
- Any new file type not in the allowlist
**devx module structure** (installed from git, not in this repo):
- `devx.ci.*` — CI/CD automation (run by workflows): release, publish, auto_merge, classify_changes, detect_release_commit, push_badges, doc_coverage, sync_wiki, distribute_molecule, molecule_ci_guard, discover_runners, notify_failure, post_merge, pr_review, validate_commit_msg
- `devx.ci.*` — CI/CD automation (run by workflows): release, publish, auto_merge, classify_changes, detect_release_commit, push_badges, doc_coverage, sync_wiki, distribute_molecule, discover_runners, notify_failure, post_merge, pr_review, validate_commit_msg
- `devx.tools.*` — Dev tools (run locally): check_test_speed, configure_repo, install_checkmake, install_tools, setup, generate_badges, create_task, create_pr, pr_status, pr_logs, pr_label, rebase, pr_rebase
- `devx.molecule.*` — Molecule helpers: molecule_all, platforms, discover_runners, distribute_molecule, molecule_ci_guard
- `devx.molecule.*` — Molecule helpers: molecule_all, platforms, discover_runners, distribute_molecule
- `devx.gitea_cli` — Tea CLI wrapper
- `devx.i18n` — i18n translation system
- `devx.config` — Shared configuration (DEVX_* env vars)
@@ -348,7 +348,7 @@ Since devx is installed as a package (via `pip install` from git), it is importa
| PYTHONPATH | When to use | Example modules |
|------------|-------------|-----------------|
| `src` | Module imports from `grm` | `devx.ci.auto_merge`, `devx.ci.pr_review`, `devx.ci.pr_review`, `devx.ci.sync_wiki`, `devx.ci.post_merge`, `devx.ci.classify_changes`, `devx.molecule.discover_runners`, `devx.ci.doc_coverage` |
| (none) | Module has no GRM imports | `devx.ci.detect_release_commit`, `devx.molecule.distribute_molecule`, `devx.molecule.molecule_ci_guard`, `devx.ci.push_badges`, `devx.ci.validate_commit_msg` |
| (none) | Module has no GRM imports | `devx.ci.detect_release_commit`, `devx.molecule.distribute_molecule`, `devx.ci.push_badges`, `devx.ci.validate_commit_msg` |
**In workflows**, always use `env:` blocks (not inline `PYTHONPATH=value`):
```yaml
+30
View File
@@ -2,6 +2,36 @@
All notable changes to this project will be documented in this file.
## [Unreleased]
### CI
- Convert `ci.yml` and `post-merge.yml` to use composite actions
(`setup-env`, `quality-checks`, `notify-failure`) copied from devx,
reducing workflow duplication
### Bug Fixes
- Fix `register.yml` premature service start: removed step that ran
`systemctl --user start gitea-runner` before the systemd unit file was
created by `service.yml` (included after `register.yml`). First-time
installs were failing with "Unit gitea-runner.service not found".
- Fix `ci.yml` auto-merge IndentationError: replaced broken inline Python
polling script with `devx.ci.wait_for_checks` (the inline Python had
YAML run-block indentation leaking into `python3 -c` string).
### Dependencies
- Bump devx from v0.50.0 to v0.50.5 (adds `wait_for_checks` tool,
consolidated Ansible checks, tenacity retry in `install_tools`,
increased download retry attempts/backoff)
## [0.20.0] - 2026-08-09
### Features
- *(healthcheck)* Add two-tier disk prune with critical threshold
## [0.19.0] - 2026-08-08
### Features
+26 -1
View File
@@ -175,11 +175,36 @@ makefile-lint:
echo "checkmake not found, skipping Makefile lint"; \
fi
lint-all: lint ansible-lint makefile-lint workflow-lint check-api-identity-checks
lint-all: lint ansible-lint makefile-lint workflow-lint check-api-identity-checks check-ansible-no-log check-ansible-no-state-absent-on-db check-ansible-patterns check-jinja-expr check-ansible-set-fact-to-json
check-api-identity-checks:
@$(BIN)/python -m devx.tools.check_api_identity_checks
check-ansible-no-log:
@echo "[check-ansible-no-log] Checking Ansible tasks for missing no_log on secret-handling tasks..."
@$(BIN)/python -m devx.tools.check_ansible_no_log
@echo "[check-ansible-no-log] Passed."
check-ansible-no-state-absent-on-db:
@echo "[check-ansible-no-state-absent-on-db] Checking for state: absent on DB data directories..."
@$(BIN)/python -m devx.tools.check_ansible_no_state_absent_on_db
@echo "[check-ansible-no-state-absent-on-db] Passed."
check-ansible-patterns:
@echo "[check-ansible-patterns] Checking for dangerous failure-masking patterns..."
@$(BIN)/python -m devx.tools.check_ansible_patterns
@echo "[check-ansible-patterns] Passed."
check-jinja-expr:
@echo "[check-jinja-expr] Validating Jinja2 expressions in Ansible files..."
@$(BIN)/python -m devx.tools.check_jinja_expr
@echo "[check-jinja-expr] Passed."
check-ansible-set-fact-to-json:
@echo "[check-ansible-set-fact-to-json] Checking set_fact tasks for to_json misuse..."
@$(BIN)/python -m devx.tools.check_ansible_set_fact_to_json
@echo "[check-ansible-set-fact-to-json] Passed."
test-integration:
$(BIN)/pytest tests/integration/ -v --no-cov
+6 -6
View File
@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
[![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/python.svg)](https://www.python.org/downloads/)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/python.svg)](https://www.python.org/downloads/)
## Why GRM?
+19 -4
View File
@@ -36,12 +36,23 @@ gitea_runner_prune_label: "gitea-runner=true"
gitea_runner_service_restart_sec: "5"
# Health check configuration
# 2min interval — catches hung daemons before multiple CI jobs fail between checks.
# The previous 5min interval was too coarse: a stuck daemon could fail 3+ molecule
# jobs in the window between healthcheck runs.
# 2min interval — catches hung daemons before multiple CI jobs fail between
# checks. The 1min interval caused excessive pruning which removed cached
# images, forcing all 6 parallel slots to re-pull simultaneously and
# actually increasing disk pressure.
gitea_runner_healthcheck_interval: "2min"
gitea_runner_healthcheck_boot_delay: "2min"
gitea_runner_healthcheck_disk_threshold: 75
gitea_runner_healthcheck_disk_threshold: 70
# When disk reaches this level, prune EVERYTHING (no until-filter) — the
# runner is dangerously full and the gentle until=1h prune isn't enough.
# This removes all stopped containers and unused images regardless of age.
# At 75%+, molecule containers fail with "container is not running" because
# overlay2 runs out of space under parallel DinD load.
# IMPORTANT: keep at 75 (not lower) — the host disk normally sits at ~74%.
# Lowering to 70 triggers full prune every cycle, wiping cached images and
# forcing all parallel slots to re-pull simultaneously, which increases
# disk pressure rather than reducing it.
gitea_runner_healthcheck_disk_critical: 75
gitea_runner_healthcheck_script_path: "{{ gitea_runner_config_dir }}/healthcheck.sh"
# Auto-recovery: when the healthcheck detects an unregistered runner, it
@@ -70,6 +81,10 @@ gitea_runner_docker_shutdown_timeout: 30
gitea_runner_docker_max_concurrent_downloads: 3
gitea_runner_docker_max_concurrent_uploads: 3
gitea_runner_docker_default_nofile: 65536
# Log file size limits — under parallel DinD load, container logs can fill
# disk and cause the daemon to become unresponsive. Limit log size per container.
gitea_runner_docker_max_log_size: "10m"
gitea_runner_docker_max_log_files: 3
# Admin token for runner deregistration via Gitea API.
# If not set, falls back to registration_token (which likely lacks admin scope).
@@ -10,3 +10,7 @@
- ansible_facts is defined
- ansible_facts['service_mgr'] | default('') == 'systemd'
- gitea_runner_docker_rootless_setup
- name: Reload systemd user daemon
ansible.builtin.systemd:
daemon_reload: true
@@ -40,24 +40,22 @@
mode: "0644"
- name: Reload systemd user daemon for healthcheck timer
ansible.builtin.command: systemctl --user daemon-reload
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user daemon-reload
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
- gitea_runner_docker_rootless_setup
- name: Enable and start healthcheck user timer
ansible.builtin.command: systemctl --user enable --now runner-healthcheck.timer
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user enable --now runner-healthcheck.timer
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
@@ -13,9 +13,9 @@
- name: Include validation
ansible.builtin.include_tasks: validate.yml
- name: Include service setup
ansible.builtin.include_tasks: service.yml
- name: Include registration
ansible.builtin.include_tasks: register.yml
when: not gitea_runner_skip_registration
- name: Include service setup
ansible.builtin.include_tasks: service.yml
+10 -12
View File
@@ -18,12 +18,11 @@
register: gitea_runner_prune_timer
- name: Reload systemd user daemon for prune timer
ansible.builtin.command: systemctl --user daemon-reload
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user daemon-reload
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
@@ -31,12 +30,11 @@
- gitea_runner_prune_service is changed or gitea_runner_prune_timer is changed
- name: Enable and start docker-prune user timer
ansible.builtin.command: systemctl --user enable --now docker-prune.timer
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user enable --now docker-prune.timer
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
@@ -44,13 +44,12 @@
('already exists' not in gitea_runner_register_output.stdout | default(''))
timeout: 60
- name: Ensure runner service is running after registration
ansible.builtin.command: systemctl --user start gitea-runner
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
- name: Restart runner service after (re-)registration
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user restart gitea-runner
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
@@ -143,6 +143,10 @@
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
{% endif %}
Restart=always
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=10
mode: "0644"
owner: "{{ gitea_runner_service_user }}"
group: "{{ gitea_runner_service_user }}"
@@ -180,6 +184,10 @@
"features": {
"containerd-snapshotter": false
},
"log-opts": {
"max-size": "{{ gitea_runner_docker_max_log_size }}",
"max-file": "{{ gitea_runner_docker_max_log_files }}"
},
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
"ipv6": true,
"ip6tables": true,
@@ -212,22 +220,20 @@
- not gitea_runner_rootless_docker_check.stat.exists
- name: Start rootless Docker daemon (systemd user service)
ansible.builtin.command: systemctl --user start docker
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user start docker
changed_when: true
when: gitea_runner_docker_rootless_setup
- name: Enable rootless Docker daemon (systemd user service)
ansible.builtin.command: systemctl --user enable docker
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user enable docker
changed_when: true
when: gitea_runner_docker_rootless_setup
@@ -250,6 +256,10 @@
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
{% endif %}
Restart=always
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=10
mode: "0644"
owner: "{{ gitea_runner_service_user }}"
group: "{{ gitea_runner_service_user }}"
@@ -257,12 +267,11 @@
when: gitea_runner_docker_rootless_setup
- name: Reload systemd user daemon if network config changed
ansible.builtin.command: systemctl --user daemon-reload
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user daemon-reload
changed_when: true
when:
- gitea_runner_docker_rootless_setup
@@ -283,6 +292,10 @@
"features": {
"containerd-snapshotter": false
},
"log-opts": {
"max-size": "{{ gitea_runner_docker_max_log_size }}",
"max-file": "{{ gitea_runner_docker_max_log_files }}"
},
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
"ipv6": true,
"ip6tables": true,
@@ -300,12 +313,11 @@
when: gitea_runner_docker_rootless_setup
- name: Restart rootless Docker if config changed
ansible.builtin.command: systemctl --user restart docker
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user restart docker
changed_when: true
when:
- gitea_runner_docker_rootless_setup
+15 -18
View File
@@ -9,12 +9,11 @@
register: gitea_runner_service_file
- name: Reload systemd user daemon
ansible.builtin.command: systemctl --user daemon-reload
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user daemon-reload
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
@@ -22,12 +21,11 @@
- gitea_runner_service_file is changed
- name: Restart gitea-runner if service file changed
ansible.builtin.command: systemctl --user restart gitea-runner
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user restart gitea-runner
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
@@ -35,12 +33,11 @@
- gitea_runner_service_file is changed
- name: Enable and start gitea-runner user service
ansible.builtin.command: systemctl --user enable --now gitea-runner
become: true
become_user: "{{ gitea_runner_service_user }}"
environment:
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
ansible.builtin.command: >
runuser -u {{ gitea_runner_service_user }} --
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
systemctl --user enable --now gitea-runner
changed_when: true
when:
- gitea_runner_systemd_available.stat.exists
@@ -69,3 +69,31 @@
owner: "{{ gitea_runner_service_user }}"
group: "{{ gitea_runner_service_user }}"
mode: "0755"
- name: Disable systemd-oomd memory pressure kill for runner user
when: gitea_runner_systemd_available.stat.exists
block:
- name: Ensure user service override directory exists
ansible.builtin.file:
path: "/etc/systemd/system/user@{{ gitea_runner_uid }}.service.d"
state: directory
owner: root
group: root
mode: "0755"
- name: Disable ManagedOOMMemoryPressure for runner user
ansible.builtin.copy:
content: |
[Service]
ManagedOOMMemoryPressure=auto
ManagedOOMMemoryPressureLimit=100%
OOMScoreAdjust=-500
dest: "/etc/systemd/system/user@{{ gitea_runner_uid }}.service.d/oomd-override.conf"
owner: root
group: root
mode: "0644"
notify: Reload systemd user daemon
- name: Reload systemd daemon for oomd override
ansible.builtin.systemd:
daemon_reload: true
@@ -11,13 +11,7 @@ Environment=XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
# consuming disk and memory. We stop+rm everything first, then prune the rest.
# Exclude CI job containers (name starts with GITEA-ACTIONS-TASK) — removing
# them kills the active CI job and causes "RWLayer is unexpectedly nil" errors.
# Only remove containers older than 1 hour (grep for "hour/day/week/month/year
# ago" in RunningFor) to avoid killing molecule test containers that CI jobs
# are actively using.
ExecStart=/bin/sh -c 'docker ps -a --format "{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}" 2>/dev/null | grep -v "GITEA-ACTIONS-TASK" | grep -E "(hour|day|week|month|year)s? ago" | awk "{print $1}" | xargs -r docker rm -f 2>/dev/null || true'
ExecStart=/bin/sh -c 'docker ps -a --format "{% raw %}{{.ID}} {{.Names}}{% endraw %}" 2>/dev/null | grep -v "GITEA-ACTIONS-TASK" | awk "{print $1}" | xargs -r docker rm -f 2>/dev/null || true'
ExecStart=/usr/bin/docker system prune -af --filter "until={{ gitea_runner_prune_until }}" --volumes
# Prune networks older than the prune-until threshold to avoid removing
# networks that molecule tests are actively creating (e.g. 'traefik' network
# created during molecule create phase before containers are attached).
ExecStart=/usr/bin/docker network prune -f --filter "until={{ gitea_runner_prune_until }}"
ExecStart=/usr/bin/docker network prune -f
ExecStart=/usr/bin/docker builder prune -f
@@ -48,14 +48,6 @@ fi
# runner (e.g., server restore, runner record deleted, Gitea restart with
# token salt change), the runner logs "unregistered runner" every few seconds.
# A service restart will not fix this; re-registration is required.
#
# Detection method: query the Gitea API to verify the runner's UUID still
# exists. This is more reliable than parsing journal logs (which requires
# journal access permissions that runner users may not have — see the
# 2026-08-08 incident where journalctl --user returned "No journal files
# were opened due to insufficient permissions" for all runner users,
# causing the healthcheck to always report "OK: runner healthy" even
# though all runners were unregistered).
{% if gitea_runner_auto_recover_api_token %}
# Auto-recovery is enabled: fetch a new registration token from the Gitea API
# and re-register the runner automatically. A cooldown prevents infinite loops.
@@ -66,50 +58,10 @@ GITEA_URL="{{ gitea_url }}"
RUNNER_NAME="{{ gitea_runner_name }}"
RUNNER_LABELS="{{ gitea_runner_labels }}"
BINARY="{{ gitea_runner_binary_path }}"
RUNNER_FILE="{{ gitea_runner_data_dir }}/.runner"
{% endif %}
runner_unregistered=0
# Primary detection: query the Gitea API to check if the runner's ID
# still exists in Gitea's runner list. This works regardless of journal
# permissions.
{% if gitea_runner_auto_recover_api_token %}
if [[ -f "$GITEA_API_TOKEN_FILE" && -f "$RUNNER_FILE" ]]; then
API_TOKEN=$(cat "$GITEA_API_TOKEN_FILE" 2>/dev/null || true)
RUNNER_ID=$(python3 -c "import json; print(json.load(open('$RUNNER_FILE')).get('id',''))" 2>/dev/null || true)
if [[ -n "$API_TOKEN" && -n "$RUNNER_ID" ]]; then
# List all runners and check if our ID is present
runner_found=$(curl -sf --connect-timeout 5 --max-time 10 \
-H "Authorization: token $API_TOKEN" \
"${GITEA_URL}/api/v1/admin/actions/runners" 2>/dev/null \
| python3 -c "
import sys, json
try:
data = json.load(sys.stdin)
runners = data if isinstance(data, list) else data.get('runners', [])
ids = [str(r.get('id', '')) for r in runners]
print('1' if '$RUNNER_ID' in ids else '0')
except Exception:
print('0')
" 2>/dev/null || echo "0")
if [[ "$runner_found" != "1" ]]; then
runner_unregistered=1
echo "CRITICAL: runner ID $RUNNER_ID not found in Gitea (unregistered)."
fi
fi
fi
{% endif %}
# Fallback detection: check journal logs (if accessible)
if [[ "$runner_unregistered" -eq 0 ]]; then
recent_errors=$(journalctl --user -u gitea-runner.service --since "5 minutes ago" --no-pager -q 2>/dev/null | grep -c "unregistered runner" || true)
if [[ "$recent_errors" -ge 3 ]]; then
runner_unregistered=1
echo "CRITICAL: runner is unregistered in Gitea (re-login failed $recent_errors times in 5 minutes)."
fi
fi
if [[ "$runner_unregistered" -ge 1 ]]; then
recent_errors=$(journalctl --user -u gitea-runner.service --since "5 minutes ago" --no-pager -q 2>/dev/null | grep -c "unregistered runner" || true)
if [[ "$recent_errors" -ge 3 ]]; then
echo "CRITICAL: runner is unregistered in Gitea (re-login failed $recent_errors times in 5 minutes)."
{% if gitea_runner_auto_recover_api_token %}
# Check cooldown — skip if we recently attempted recovery
if [[ -f "$COOLDOWN_FILE" ]]; then
@@ -183,32 +135,14 @@ if [[ "$runner_unregistered" -ge 1 ]]; then
systemctl --user start gitea-runner.service
sleep 3
# Verify recovery — query the Gitea API to confirm the new ID is registered
NEW_ID=$(python3 -c "import json; print(json.load(open('$RUNNER_FILE')).get('id',''))" 2>/dev/null || true)
if [[ -n "$NEW_ID" ]]; then
new_found=$(curl -sf --connect-timeout 5 --max-time 10 \
-H "Authorization: token $API_TOKEN" \
"${GITEA_URL}/api/v1/admin/actions/runners" 2>/dev/null \
| python3 -c "
import sys, json
try:
data = json.load(sys.stdin)
runners = data if isinstance(data, list) else data.get('runners', [])
ids = [str(r.get('id', '')) for r in runners]
print('1' if '$NEW_ID' in ids else '0')
except Exception:
print('0')
" 2>/dev/null || echo "0")
if [[ "$new_found" == "1" ]]; then
echo "OK: runner recovered and registered with new ID $NEW_ID"
# Clear cooldown on success
rm -f "$COOLDOWN_FILE" 2>/dev/null || true
else
echo "WARN: runner re-registered but ID not found in Gitea API. Will retry after cooldown."
exit 3
fi
# Verify recovery — check if unregistered errors stopped
new_errors=$(journalctl --user -u gitea-runner.service --since "10 seconds ago" --no-pager -q 2>/dev/null | grep -c "unregistered runner" || true)
if [[ "$new_errors" -eq 0 ]]; then
echo "OK: runner recovered and no longer reporting unregistered errors"
# Clear cooldown on success
rm -f "$COOLDOWN_FILE" 2>/dev/null || true
else
echo "WARN: could not read new .runner file after re-registration. Will retry after cooldown."
echo "WARN: runner still showing unregistered errors after re-registration. Will retry after cooldown."
exit 3
fi
{% else %}
@@ -225,22 +159,16 @@ fi
disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
echo "WARN: Disk usage at ${disk_pct}%, pruning all runner resources"
# Force-remove stale containers (including running ones from failed molecule tests)
# that are older than 1 hour. "docker container prune -f" only removes stopped
# containers, so running containers from crashed CI jobs accumulate and consume
# disk/memory. Exclude CI job containers (name starts with GITEA-ACTIONS-TASK).
# Only remove containers older than 1 hour to avoid killing molecule test
# containers that CI jobs are actively using.
docker ps -a --format '{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}' 2>/dev/null \
# Force-remove stale containers (including running ones from failed molecule tests).
# "docker container prune -f" only removes stopped containers, so running
# containers from crashed CI jobs accumulate and consume disk/memory.
# Exclude CI job containers (name starts with GITEA-ACTIONS-TASK).
docker ps -a --format '{% raw %}{{.ID}} {{.Names}}{% endraw %}' 2>/dev/null \
| grep -v 'GITEA-ACTIONS-TASK' \
| grep -E '(hour|day|week|month|year)s? ago' \
| awk '{print $1}' \
| xargs -r docker rm -f 2>/dev/null || true
docker system prune -af --filter "until=1h" --volumes || true
# Prune networks older than 1 hour to avoid removing networks that
# molecule tests are actively creating (e.g. 'traefik' network created
# during molecule create phase before containers are attached).
docker network prune -f --filter "until=1h" || true
docker network prune -f || true
disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
echo "INFO: Disk usage after prune: ${disk_pct}%"
fi
+6 -6
View File
@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
[![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/3720ac3b2d4a1a832197f5c128d50f59f6fff163/python.svg)](https://www.python.org/downloads/)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/e96a599d8e0a5186cd72bebaddbd24b2df2e2058/python.svg)](https://www.python.org/downloads/)
## Overview
+3 -3
View File
@@ -259,9 +259,9 @@ OS platform matrix (defined in `devx.molecule.platforms`), then splits
the resulting test pairs evenly across the requested number of runners.
Each pair is encoded as `scenario|platform_name|platform_image|platform_command`.
`devx.molecule.molecule_ci_guard` runs the actual molecule test for a
given test pair, with CI context (Gitea URL, token, run ID) for
reporting results back to the commit status API.
The CI workflow runs each test pair sequentially via a shell loop that
sets the appropriate `MOLECULE_PLATFORM_*` environment variables and
invokes `molecule test` directly.
### Commit Message Validation
+1 -1
View File
@@ -91,7 +91,7 @@ The `molecule-tests` job uses `fromJSON()` to consume the dynamic matrix, and pa
`devx.molecule.distribute_molecule` discovers all molecule scenarios under `ansible/roles/*/molecule/` and crosses them with the supported OS platform matrix, then splits the resulting test pairs evenly across the requested number of runners. Each pair is encoded as `scenario|platform_name|platform_image|platform_command`.
`devx.molecule.molecule_ci_guard` runs the actual molecule test for a given test pair, with CI context (Gitea URL, token, run ID) for reporting results back to the commit status API.
The CI workflow runs each test pair sequentially via a shell loop that sets the appropriate `MOLECULE_PLATFORM_*` environment variables and invokes `molecule test` directly.
### Path-based CI filtering
+2 -2
View File
@@ -36,7 +36,7 @@ ci = [
"build==1.5.1",
"twine==6.2.0",
# Reusable CI/CD and dev tools (auto-merge, pr-review, pre-push checks, etc.)
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.48.1",
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.50.1",
]
# Lint and type-checking tools (validate job)
lint = [
@@ -56,7 +56,7 @@ molecule = [
dev = [
"grm[ci,lint,molecule]",
# Reusable CI/CD and dev tools (pre-push hooks, create-task, create-pr)
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.48.1",
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.50.1",
# Non-Python dev dependency: checkmake (Makefile linter)
# Install via: go install github.com/checkmake/checkmake/cmd/checkmake@latest
]
+1 -1
View File
@@ -1,3 +1,3 @@
"""Gitea Runner Manager — lean CLI for managing Gitea Actions runners."""
__version__ = "0.19.0"
__version__ = "0.20.0"