release: v0.24.0 [skip ci]

Co-authored-by: emil User <emil.simeonov@tutanota.com>

.devin/skills/devx-workflow/SKILL.md

@@ -2,18 +2,28 @@
 
 Quick reference for devx tools when working on this repo.
 
+## When to Invoke
+
+Invoke this skill when creating PRs, checking CI status, adding
+labels, rebasing branches, or performing any PR lifecycle operation.
+
+## Prerequisites
+
+- `.venv` exists (run `make setup` if not)
+- `.env` with `DEVELOPER_GITEA_API_TOKEN`, `VIKUNJA_TOKEN`
+
 ## PR Workflow (use these, not raw git/tea/MCP)
 
 | Task | Command |
 |------|---------|
-| Create Vikunja task | `make create-task -- --title "..." --description "..."` |
+| Create Vikunja task | `.venv/bin/python -m devx.tools.create_task --title "..." --description "..."` (make target doesn't forward args) |
 | Create PR | `make create-pr` |
 | Push + create PR | `make push-with-pr` |
 | Check CI status | `make devx-pr-status` or `make devx-pr-status PR=42 WAIT=1` |
 | Fetch CI failure logs | `make devx-pr-logs` or `make devx-pr-logs PR=42 JOB=quality TAIL=50` |
 | Add ready-to-merge label | `make devx-pr-label` or `make devx-pr-label PR=42` |
-| Rebase current branch | `make rebase` |
-| Rebase PR via API | `make pr-rebase` or `make pr-rebase PR=42` |
+| Rebase current branch | `make devx-rebase` |
+| Rebase PR via API | `make devx-pr-rebase` or `make pr-rebase PR=42` |
 
 ## Auto-merge Behavior
 

.devin/skills/molecule-testing/SKILL.md

@@ -0,0 +1,70 @@
+# molecule-testing
+
+Authoring and debugging `gitea_runner` molecule scenarios. For running
+tests use the `testing-and-debugging` make targets — this covers
+writing scenarios and fixing DIND/platform issues.
+
+## When to Invoke
+
+- Adding a molecule scenario for the `gitea_runner` role
+- A scenario fails on platform setup, DIND, or registration mocking
+- Reviewing scenario coverage for a role change
+
+## Prerequisites
+
+- Docker running locally
+- `.venv` exists (`make setup`)
+
+## Scenario Layout
+
+`ansible/roles/gitea_runner/molecule/<scenario>/`:
+
+Current scenarios: `default`, `template-content`, `deregister`,
+`multi-instance`, `update`, `remove`, `lifecycle`.
+
+| File | Purpose |
+|------|---------|
+| `molecule.yml` | driver/platforms/provisioner config |
+| `converge.yml` | applies the role |
+| `verify.yml` | assertions scoped to the scenario |
+| `prepare.yml` | optional host prep |
+
+Scenario registration lives in `pyproject.toml` (scenario map used by
+`devx.molecule` distribution in CI) — a new scenario MUST be
+registered there or CI never runs it.
+
+## molecule.yml Conventions
+
+- Platform name/image/command are env-overridable via
+  `${MOLECULE_PLATFORM_*}` so all-platforms runs work.
+- `remote_tmp: /tmp` in provisioner `config_options` — default temp
+  dir breaks in containers.
+- `ANSIBLE_ROLES_PATH` must include the repo roles root.
+- Use `inventory.group_vars` to isolate the scenario: disable
+  unrelated features rather than editing tasks.
+- Runner registration in tests is mocked/faked — scenarios must not
+  require a live Gitea instance; check how existing scenarios stub
+  the registration/token flow before adding API calls.
+
+## Debugging
+
+```bash
+cd ansible/roles/gitea_runner
+molecule test -s <scenario>
+molecule converge -s <scenario>
+molecule login -s <scenario>
+```
+
+- "Failed to create temporary directory" → `remote_tmp: /tmp` missing.
+- Idempotence failures → find the changed task on second converge.
+- Registration/API timeouts → the scenario hit a real endpoint —
+  stub it like the existing scenarios do.
+
+## Common Mistakes
+
+- Adding a scenario without registering it in `pyproject.toml` —
+  silently untested.
+- Hardcoding the platform image — keep `${MOLECULE_PLATFORM_*}`
+  overrides.
+- Calling the real Gitea API in converge — scenarios must be
+  self-contained; mock the registration path.

.devin/skills/runner-ops/SKILL.md

@@ -0,0 +1,73 @@
+# runner-ops
+
+Operating the Gitea Actions runner fleet: registration lifecycle,
+stale-runner cleanup, image pruning, and safe debugging. Core code:
+`src/grm/runner_manager.py`, `src/grm/executor.py`,
+`src/grm/registry.py`.
+
+## When to Invoke
+
+- Runners go offline, stall, or pile up stale registrations
+- Runner hosts need install/update/remove/deregister operations
+- Disk pressure on runner hosts (image/container accumulation)
+- Working on S08 (leases, physical-host admission, disk watermarks)
+
+## Prerequisites
+
+- `.env` with Gitea admin token for API operations
+- SSH access to runner hosts for Ansible-driven lifecycle
+- Runner registrations visible via admin API:
+  `GET /api/v1/admin/actions/runners`
+
+## Architecture
+
+- `RunnerManager` orchestrates install/update/lifecycle via
+  `AnsibleExecutor` against the `gitea_runner` role; `RunnerRegistry`
+  tracks local runner state.
+- Runners execute jobs in Docker (`docker` label) — every job gets a
+  fresh container from `ci-base`/`ci-quality`/`ci-full` images.
+- Molecule jobs nest containers (DIND) — privileged, `SYS_ADMIN`,
+  `/var/lib/docker` volume.
+
+## Lifecycle Operations
+
+| Task | Entry point |
+|------|-------------|
+| Install/update runners | `grm` CLI → `RunnerManager` (Ansible) |
+| Stale registration cleanup | `scripts/cleanup_stale_runners.py` — deletes runners offline >1h via `DELETE /api/v1/admin/actions/runners/{id}` |
+| Image pruning | `scripts/prune_runner_images.py` — reclaims disk from old CI image versions |
+
+Stale registrations accumulate when a host is rebuilt, re-registered,
+or its runner process dies unrecoverably — clean them before capacity
+accounting.
+
+## Debugging a Stuck Runner
+
+1. Check registration state via admin API (offline vs online).
+2. SSH to the host: `systemctl status` the runner service / inspect
+   `docker ps` for orphaned job containers.
+3. Orphaned molecule containers: safe to remove ONLY when no molecule
+   run is active — check runner logs first (`runner-ops` counterpart
+   of "don't force-remove active containers", fixed in GRM-166/167).
+4. Disk pressure: check `/var/lib/docker` usage, then
+   `prune_runner_images.py` — never blanket `docker system prune`
+   while jobs may be mid-flight.
+
+## S08-Relevant Rules
+
+- Runner admission must be per physical host — a runner that shares
+  hardware must declare capacity, not just labels.
+- Cleanup must never remove a container a live job owns — ownership
+  check before any force-removal.
+- Disk watermark logic belongs in the role/scripts, not ad-hoc
+  cron `docker prune`.
+
+## Common Mistakes
+
+- `docker system prune -a` on a runner host — kills in-flight job
+  containers and image cache mid-run.
+- Deleting an offline runner registration while the host still runs
+  the service — it re-registers and duplicates; stop the service
+  first.
+- Treating molecule DIND containers as junk — they belong to an
+  active scenario; check timestamps and runner logs.

.devin/skills/spec-driven-development/SKILL.md

@@ -1,5 +1,14 @@
 # Spec-Driven Development
 
+## When to Invoke
+
+Invoke this skill when starting any change — every PR requires a spec
+at `docs/specs/<TASK-ID>.md` that CI validates before merge.
+
+## Prerequisites
+
+- A Vikunja task ID (`GRM-N`) — see `vikunja-tasks` skill
+
 ## Overview
 
 Every change starts with a spec. No spec, no code. No code, no PR.

.devin/skills/testing-and-debugging/SKILL.md

@@ -3,6 +3,17 @@
 Make targets for testing, debugging, and CI investigation. **Use these
 instead of raw `pytest`, `ruff`, or `molecule` commands.**
 
+## When to Invoke
+
+Invoke this skill when running tests, investigating CI failures, or
+debugging molecule scenarios. Also invoke when asked to "run tests",
+"check coverage", or "debug a failure".
+
+## Prerequisites
+
+- `.venv` exists (run `make setup` if not)
+- For molecule tests: Docker is running
+
 ## Why Make Targets
 
 Make targets encapsulate the correct venv activation, PYTHONPATH, env
@@ -31,9 +42,8 @@ produces false failures (missing dependencies, wrong Python version).
 
 | Task | Command | Notes |
 |------|---------|-------|
-| All scenarios | `make molecule` | All 6 scenarios on Ubuntu 22.04 |
-| All platforms | `make molecule-all` | All 6 scenarios on all 4 OSes |
-| Parallel | `make molecule-all-parallel` | MOLECULE_JOBS=4 |
+| All scenarios | `make molecule` | All 7 scenarios on Ubuntu 22.04 |
+| All platforms | `make molecule-all` | All 7 scenarios on all 4 OSes |
 
 ### Spec-Driven Workflow
 
@@ -46,12 +56,12 @@ CI validates the spec before running expensive jobs.
 **Before pushing any branch:**
 
 ```bash
-make pre-push
+make lint-all && make pytest-cov
 ```
 
-This runs `lint-all` + `pytest-cov`. The pre-push git hook only
-validates the Vikunja task exists — it does NOT run tests. You must
-run `make pre-push` manually.
+This runs all linters + unit tests with coverage. The pre-push git
+hook only validates the Vikunja task exists — it does NOT run tests.
+Run the checks manually (there is no `pre-push` target here).
 
 ## CI Failure Investigation
 
@@ -59,7 +69,7 @@ When investigating a CI failure:
 
 1. **Fetch logs via MCP** — use `mcp_call_tool` with gitea server,
    `actions_run_read` method, `download_job_log` tool
-2. **Reproduce locally** — use `make pytest-cov` or `make lint-ci`
+2. **Reproduce locally** — use `make pytest-cov` or `make lint-all`
    depending on which CI job failed
 3. **Never run raw pytest** — always use the make target
 

.devin/skills/vikunja-tasks/SKILL.md

@@ -0,0 +1,74 @@
+# vikunja-tasks
+
+Vikunja task lifecycle beyond `create`: querying status, closing, and
+recovering when the tracker is unreachable.
+
+## When to Invoke
+
+- Creating, closing, or checking a Vikunja task
+- A spec workflow step needs the task ID or done state
+- `vikunja.oblachno.oblachno.fyi` fails to resolve / times out
+
+## Prerequisites
+
+- `.env` with `VIKUNJA_TOKEN`
+- Project ID comes from `[tool.devx]` in `pyproject.toml`
+  (`DEVX_VIKUNJA_PROJECT_ID`)
+
+## Create
+
+`make create-task` does **not** forward arguments — call the module:
+
+```bash
+.venv/bin/python -m devx.tools.create_task \
+  --title "Task title (no GRM-N prefix)" \
+  --description "<h2>Context</h2><p>...</p>"
+```
+
+Prints `GRM-N` + next steps. Title must not include the task-ID
+prefix (auto-merge prepends it; a manual prefix double-prefixes the
+PR title and fails validation).
+
+## Query / Close
+
+```bash
+# Task details (ID = numeric part of GRM-N)
+curl -sf -H "Authorization: Bearer $VIKUNJA_TOKEN" \
+  "https://vikunja.oblachno.oblachno.fyi/api/v1/tasks/<N>"
+
+# Close: mark done
+curl -sf -X POST -H "Authorization: Bearer $VIKUNJA_TOKEN" \
+  -H "Content-Type: application/json" -d '{"done":true}' \
+  "https://vikunja.oblachno.oblachno.fyi/api/v1/tasks/<N>"
+```
+
+Post-merge automation marks the task done when the PR squash-merges —
+manual close is only needed for abandoned/superseded tasks.
+
+## Task-ID / Spec Collisions
+
+Vikunja IDs can collide with historical spec files (an old task reused
+the number). Convention: preserve the old file as
+`docs/specs/<ID>-<topic>-historical.md`, then write the new spec at
+`docs/specs/<ID>.md`. Check `git log` on the existing spec before
+moving it.
+
+## Tracker Unreachable
+
+If the Vikunja host fails DNS/TLS:
+
+1. Don't block the whole workflow — record the intended task title in
+   the spec draft and retry `create_task` before branching.
+2. Never invent an ID — branch/PR titles must match a real task or
+   `pre_push_check` / auto-merge validation fails.
+3. DNS failures observed so far were transient; retry after a few
+   minutes before escalating.
+
+## Common Mistakes
+
+- `make create-task -- --title ...` — args are dropped; use the module
+  call above (forwarding fix is S11 scope).
+- Including `GRM-N:` in the task title — double prefix breaks
+  auto-merge.
+- Closing a task whose PR is still open — auto-merge's post-merge
+  step handles the close; manual close confuses the audit trail.

.gitea/workflows/ci.yml

@@ -32,6 +32,22 @@ jobs:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
+      # Implements: REQ-1 (GRM-172) — docs-only changes skip the heavy
+      # quality steps. Detection needs only git, so it runs before setup.
+      - name: Detect docs-only change
+        id: docs-only
+        if: github.event_name == 'pull_request'
+        run: |
+          HEAD="${{ github.event.pull_request.head.sha || github.sha }}"
+          DOCS_ONLY=true
+          while IFS= read -r f; do
+            case "$f" in
+              docs/*|*.md|.devin/*) ;;
+              *) DOCS_ONLY=false; break;;
+            esac
+          done < <(git diff --name-only "origin/master...$HEAD")
+          echo "docs-only=$DOCS_ONLY" >> "$GITHUB_OUTPUT"
+          echo "docs-only=$DOCS_ONLY"
       - name: Set up environment
         env:
           CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
@@ -39,11 +55,13 @@ jobs:
         run: make setup-image EXTRAS=ci,lint
       # --- quality steps ---
       - name: Lint all
+        if: steps.docs-only.outputs.docs-only != 'true'
         run: |
           . .venv/bin/activate 2>/dev/null || true
           export PATH="$HOME/.local/bin:$PATH"
           make lint-all
       - name: Unit tests with 100% coverage
+        if: steps.docs-only.outputs.docs-only != 'true'
         run: |
           . .venv/bin/activate 2>/dev/null || true
           make pytest-cov
@@ -57,14 +75,17 @@ jobs:
           export PATH="$HOME/.local/bin:$PATH"
           make devx-docs-check
       - name: Translation completeness check
+        if: steps.docs-only.outputs.docs-only != 'true'
         run: |
           . .venv/bin/activate 2>/dev/null || true
           python3 -m devx.ci.check_translations --translations src/grm/translations.json
       - name: Check unit test speed
+        if: steps.docs-only.outputs.docs-only != 'true'
         run: |
           . .venv/bin/activate 2>/dev/null || true
           python3 -m devx.tools.check_test_speed --max-seconds 4 --max-single-seconds 0.5
       - name: Dependency security scan
+        if: steps.docs-only.outputs.docs-only != 'true'
         run: |
           . .venv/bin/activate 2>/dev/null || true
           # Install pip in venv if missing (needed by pip-audit)
@@ -72,6 +93,7 @@ jobs:
           PIPAPI_PYTHON_LOCATION=$PWD/.venv/bin/python \
             pip-audit --desc --skip-editable 2>&1 || true
       - name: Workflow dry-run validation
+        if: steps.docs-only.outputs.docs-only != 'true'
         run: |
           . .venv/bin/activate 2>/dev/null || true
           export PATH="$HOME/.local/bin:$PATH"
@@ -156,18 +178,9 @@ jobs:
             --owner "${{ github.repository_owner }}" \
             --repo "${{ github.event.repository.name }}" \
             --github-output
-      - name: Notify on failure
-        if: failure()
-        env:
-          CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
-        run: |
-          . .venv/bin/activate 2>/dev/null || true
-          export PATH="$HOME/.local/bin:$PATH"
-          python3 -m devx.ci.notify_failure --auto-login \
-            --repo "${{ github.repository }}" \
-            --run-id "${{ github.run_id }}" \
-            --workflow "ci/validate" \
-            --commit "${{ github.sha }}"
+      # Implements: REQ-2 (GRM-172) — no failure-issue step in PR CI;
+      # auto-created issues are for deploy-pipeline failures only
+      # (post-merge keeps its notification).
 
   molecule-tests:
     needs: [validate]

.gitea/workflows/post-merge.yml

@@ -25,7 +25,9 @@ on:
 
 concurrency:
   group: post-merge-${{ github.ref }}
-  cancel-in-progress: true
+  # Implements: REQ-3 (GRM-172) — queue instead of killing an in-flight
+  # release/publish; a cancelled release can leave tag-without-publish.
+  cancel-in-progress: false
 
 env:
   PIP_BREAK_SYSTEM_PACKAGES: "1"

AGENTS.md

@@ -111,7 +111,7 @@ The auto-merge workflow enforces the APPROVE review check programmatically
 as a defense-in-depth measure, but branch protection is the primary gate.
 
 ### 1. Create Vikunja Task
-Create a task in Vikunja project 6 via `make create-task -- --title "Task title" --description "<h2>...</h2>"` (requires `VIKUNJA_TOKEN` in `.env`). This prints the `GRM-N` identifier and next-step instructions.
+Create a task in Vikunja project 6 via `.venv/bin/python -m devx.tools.create_task --title "Task title" --description "<h2>...</h2>"` (make target does not forward args) (requires `VIKUNJA_TOKEN` in `.env`). This prints the `GRM-N` identifier and next-step instructions.
 
 **IMPORTANT:** The task title must NOT include the `GRM-N:` prefix.
 The `make create-pr` and `check_auto_merge_ready` commands automatically

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.24.0] - 2026-09-22
+
+### Features
+
+- Scoped runner cleanup with ownership leases and disk admission
+
 ## [0.23.3] - 2026-09-18
 
 ### Bug Fixes

README.md

@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
 
 [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
 [![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
-[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
-[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
-[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/python.svg)](https://www.python.org/downloads/)
+[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
+[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
+[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/python.svg)](https://www.python.org/downloads/)
 
 ## Why GRM?
 

ansible/roles/gitea_runner/defaults/main.yml

@@ -31,6 +31,13 @@ gitea_runner_prune_until: "24h"
 # container operations).
 gitea_runner_prune_schedule: "*-*-* 00/6:00:00"
 gitea_runner_prune_label: "gitea-runner=true"
+# Shared scoped cleanup script (runner-cleanup.sh) used by the prune timer
+# and the healthcheck disk-pressure tiers (GRM-173).
+gitea_runner_cleanup_script_path: "{{ gitea_runner_config_dir }}/cleanup.sh"
+# Regex alternation of image refs never removed by cleanup — warm base
+# layers stay warm even under critical disk pressure.
+gitea_runner_keep_images:
+  - "runner-images/"
 
 # Service configuration
 gitea_runner_service_restart_sec: "5"
@@ -42,13 +49,20 @@ gitea_runner_service_restart_sec: "5"
 gitea_runner_healthcheck_interval: "2min"
 gitea_runner_healthcheck_boot_delay: "2min"
 gitea_runner_healthcheck_disk_threshold: 70
-# When disk reaches this level, prune EVERYTHING (no until-filter) — the
-# runner is dangerously full and the gentle until=1h prune isn't enough.
-# This removes all stopped containers and unused images regardless of age.
-# At 75%+, molecule containers fail with "container is not running" because
-# overlay2 runs out of space under parallel DinD load.
+# At this level the cleanup script drops age limits — the runner is
+# dangerously full and the gentle until=1h prune isn't enough. Leases,
+# keep-images, running containers and CI job containers are still honored
+# (GRM-173). At 75%+, molecule containers fail with "container is not
+# running" because overlay2 runs out of space under parallel DinD load.
 gitea_runner_healthcheck_disk_critical: 75
 gitea_runner_healthcheck_script_path: "{{ gitea_runner_config_dir }}/healthcheck.sh"
+# Disk-pressure admission control (GRM-173): at critical disk usage the
+# healthcheck stops gitea-runner.service (no new jobs are fetched) once no
+# CI job container is running, and resumes it automatically after recovery.
+gitea_runner_disk_admission_enabled: true
+# Physical-host admission (GRM-173): act_runner capacity — max parallel
+# tasks per runner. Declared explicitly (upstream default is 1).
+gitea_runner_capacity: 1
 
 # CI job containers older than this many minutes get an exec-responsiveness
 # probe; a timeout writes one diagnostics bundle per container for
@@ -121,6 +135,13 @@ gitea_runner_valid_volumes:
 gitea_runner_containerd_max_compatible_major: 2
 gitea_runner_containerd_max_compatible_minor: 2
 
+# Production-host exclusion (GRM-173): the role fails when the target is a
+# production host — via this flag or the marker file — unless
+# allow_production_host explicitly overrides.
+gitea_runner_on_production_host: false
+gitea_runner_allow_production_host: false
+gitea_runner_production_marker_path: "/etc/oblachno/production-host"
+
 # Docker installation (for rootless dependencies)
 gitea_runner_docker_gpg_key_path: "/etc/apt/keyrings/docker.gpg"
 gitea_runner_docker_apt_arch: "{{ 'amd64' if ansible_facts['architecture'] == 'x86_64' else ansible_facts['architecture'] }}"

ansible/roles/gitea_runner/molecule/template-content/verify.yml

@@ -47,14 +47,39 @@
       ansible.builtin.assert:
         that:
           - "'Type=oneshot' in prune_service.content | b64decode"
-          - "'docker rm -f' in prune_service.content | b64decode"
-          - "'status=exited' in prune_service.content | b64decode"
           - "'GITEA-ACTIONS-TASK' in prune_service.content | b64decode"
-          - "'docker system prune -af' in prune_service.content | b64decode"
-          - "'docker network prune' in prune_service.content | b64decode"
-          - "'docker builder prune' in prune_service.content | b64decode"
+          - "(gitea_runner_cleanup_script_path ~ ' --tier routine') in prune_service.content | b64decode"
         fail_msg: "Prune service template is missing expected directives"
 
+    - name: Read rendered cleanup script
+      ansible.builtin.slurp:
+        src: "{{ gitea_runner_cleanup_script_path }}"
+      register: cleanup_script
+
+    - name: Assert cleanup script honors leases and tiers
+      ansible.builtin.assert:
+        that:
+          - "'org.oblachno.lease-until' in cleanup_script.content | b64decode"
+          - "'lease_active' in cleanup_script.content | b64decode"
+          - "'GITEA-ACTIONS-TASK' in cleanup_script.content | b64decode"
+          - "'runner-images/' in cleanup_script.content | b64decode"
+          - "'docker image inspect' in cleanup_script.content | b64decode"
+          - "'label!=' in cleanup_script.content | b64decode"
+          - "'docker system prune' not in cleanup_script.content | b64decode"
+          - "'critical)' in cleanup_script.content | b64decode"
+        fail_msg: "Cleanup script template is missing expected content"
+
+    - name: Read rendered runner config
+      ansible.builtin.slurp:
+        src: "{{ gitea_runner_config_dir }}/config.yaml"
+      register: runner_config
+
+    - name: Assert runner config declares capacity
+      ansible.builtin.assert:
+        that:
+          - "('capacity: ' ~ gitea_runner_capacity) in runner_config.content | b64decode"
+        fail_msg: "Runner config is missing capacity declaration"
+
     - name: Read rendered prune timer template
       ansible.builtin.slurp:
         src: "{{ gitea_runner_home }}/.config/systemd/user/docker-prune.timer"
@@ -108,12 +133,13 @@
           - "'systemctl --user restart gitea-runner.service' in healthcheck_script.content | b64decode"
           - "'docker rm -f' in healthcheck_script.content | b64decode"
           - "'GITEA-ACTIONS-TASK' in healthcheck_script.content | b64decode"
-          - "'docker system prune -af' in healthcheck_script.content | b64decode"
-          - "'docker network prune' in healthcheck_script.content | b64decode"
+          - "'--tier critical' in healthcheck_script.content | b64decode"
+          - "'--tier pressure' in healthcheck_script.content | b64decode"
+          - "'disk-admission-block' in healthcheck_script.content | b64decode"
+          - "'systemctl --user stop gitea-runner.service' in healthcheck_script.content | b64decode"
+          - "'docker system prune' not in healthcheck_script.content | b64decode"
           - "'status=removing' in healthcheck_script.content | b64decode"
           - "'status=stopping' in healthcheck_script.content | b64decode"
-          - "'status=exited' in healthcheck_script.content | b64decode"
-          - "'status=dead' in healthcheck_script.content | b64decode"
           - "gitea_runner_healthcheck_disk_threshold | string in healthcheck_script.content | b64decode"
           - "gitea_runner_healthcheck_disk_critical | string in healthcheck_script.content | b64decode"
         fail_msg: "Healthcheck script template is missing expected content"

ansible/roles/gitea_runner/tasks/main.yml

@@ -1,4 +1,23 @@
 ---
+# Implements: REQ-6 (GRM-173) — a CI runner must never be installed on a
+# production host (production workloads must not share hardware with
+# arbitrary CI jobs, and runner cleanup logic assumes a dedicated host).
+- name: Check for production-host marker
+  ansible.builtin.stat:
+    path: "{{ gitea_runner_production_marker_path }}"
+  register: gitea_runner_production_marker
+
+- name: Fail on production hosts
+  ansible.builtin.fail:
+    msg: >-
+      Refusing to install a CI runner on a production host
+      (marker: {{ gitea_runner_production_marker_path }} present or
+      gitea_runner_on_production_host=true). Set
+      gitea_runner_allow_production_host=true to override.
+  when:
+    - not gitea_runner_allow_production_host
+    - gitea_runner_on_production_host or gitea_runner_production_marker.stat.exists
+
 - name: Include systemd availability check
   ansible.builtin.include_tasks: systemd_check.yml
 

ansible/roles/gitea_runner/tasks/prune.yml

@@ -1,4 +1,14 @@
 ---
+# Implements: REQ-2 (GRM-173) — shared scoped cleanup script used by both
+# the prune timer and the healthcheck disk-pressure tiers.
+- name: Create runner cleanup script
+  ansible.builtin.template:
+    src: runner-cleanup.sh.j2
+    dest: "{{ gitea_runner_cleanup_script_path }}"
+    owner: "{{ gitea_runner_service_user }}"
+    group: "{{ gitea_runner_service_user }}"
+    mode: "0755"
+
 - name: Create docker-prune user service file
   ansible.builtin.template:
     src: docker-prune.service.j2

ansible/roles/gitea_runner/templates/docker-prune.service.j2

@@ -5,21 +5,10 @@ Description=Docker prune for Gitea runner resources
 Type=oneshot
 Environment=DOCKER_HOST=unix:///run/user/{{ gitea_runner_uid }}/docker.sock
 Environment=XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
-# Force-remove stale *stopped* containers left behind by failed molecule tests.
-# Implements: REQ-1 (GRM-166) — only containers with status=exited are
-# eligible. RunningFor measures creation time, so a stale molecule instance
-# (e.g. ubuntu-2604) that a new run restarts still looks ">1h old"; removing
-# running containers kills active converges with "No such container"
-# (infra nightly run 5710). Running leftovers are instead reused or destroyed
-# by the next molecule create/destroy cycle.
-# Exclude CI job containers (name starts with GITEA-ACTIONS-TASK) — removing
-# them kills the active CI job and causes "RWLayer is unexpectedly nil" errors.
-# Only remove containers older than 1 hour (grep for "hour/day/week/month/year
-# ago" in RunningFor) to avoid removing containers a job just created.
-ExecStart=/bin/sh -c 'docker ps -a --filter "status=exited" --format "{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}" 2>/dev/null | grep -v "GITEA-ACTIONS-TASK" | grep -E "(hour|day|week|month|year)s? ago" | awk "{print $1}" | xargs -r docker rm -f 2>/dev/null || true'
-ExecStart=/usr/bin/docker system prune -af --filter "until={{ gitea_runner_prune_until }}" --volumes
-# Prune networks older than the prune-until threshold to avoid removing
-# networks that molecule tests are actively creating (e.g. 'traefik' network
-# created during molecule create phase before containers are attached).
-ExecStart=/usr/bin/docker network prune -f --filter "until={{ gitea_runner_prune_until }}"
-ExecStart=/usr/bin/docker builder prune -f
+# Implements: REQ-1/REQ-2 (GRM-173) — all cleanup goes through the shared
+# scoped cleanup script: only stopped containers, CI job containers excluded
+# (GITEA-ACTIONS-TASK prefix), valid `org.oblachno.lease-until` leases never
+# removed, keep-images retained. The historical inline logic here killed
+# active molecule converges ("No such container", infra nightly run 5710)
+# and CI jobs ("RWLayer is unexpectedly nil").
+ExecStart={{ gitea_runner_cleanup_script_path }} --tier routine

ansible/roles/gitea_runner/templates/gitea-runner-config.yaml.j2

@@ -3,6 +3,9 @@ log:
 
 runner:
   file: "{{ gitea_runner_file }}"
+  # Implements: REQ-5 (GRM-173) — physical-host admission: declared capacity
+  # limits parallel tasks instead of relying on labels alone.
+  capacity: {{ gitea_runner_capacity }}
   fetch_timeout: 50s
   fetch_interval: 2s
 

ansible/roles/gitea_runner/templates/runner-cleanup.sh.j2

@@ -0,0 +1,125 @@
+#!/bin/bash
+# Scoped Docker cleanup for gitea-runner hosts.
+# Implements: REQ-1..REQ-3 (GRM-173) — ownership leases, tiered watermarks,
+# keep-images. Single entry point shared by docker-prune.service (routine)
+# and runner-healthcheck.sh (pressure/critical).
+# No `set -e`: a failing prune must not abort the remaining cleanup.
+set -uo pipefail
+
+DOCKER_HOST="unix:///run/user/{{ gitea_runner_uid }}/docker.sock"
+XDG_RUNTIME_DIR="/run/user/{{ gitea_runner_uid }}"
+export DOCKER_HOST XDG_RUNTIME_DIR
+
+# REQ-1 label contract: `org.oblachno.lease-until` (epoch) protects an object
+# while in the future; `org.oblachno.owner` records the owning run.
+TIER="${1:-routine}"
+LEASE_UNTIL_LABEL="org.oblachno.lease-until"
+KEEP_IMAGES_RE="{{ gitea_runner_keep_images | join('|') }}"
+now_epoch=$(date +%s)
+
+# Implements: REQ-1 — a lease whose `lease-until` epoch lies in the future
+# protects its object from every removal path in this script.
+lease_active() {
+    local until="$1"
+    [[ -n "$until" && "$until" =~ ^[0-9]+$ && "$until" -gt "$now_epoch" ]]
+}
+
+# Remove stopped containers. $1 = "aged" (only >1h, RunningFor heuristic)
+# or "all". CI job containers and valid leases are never removed.
+remove_stopped_containers() {
+    local mode="$1"
+    # Implements: REQ-1/REQ-3 — pipe-separated fields; RunningFor contains
+    # spaces, so whitespace-splitting would break the age gate.
+    { timeout 30 docker ps -a --filter "status=exited" --filter "status=dead" \
+        --format '{% raw %}{{.ID}}|{{.Names}}|{{.RunningFor}}|{{.Label "org.oblachno.lease-until"}}{% endraw %}' \
+        2>/dev/null || true; } \
+    | while IFS='|' read -r cid cname running_for lease_until; do
+        [[ -z "$cid" ]] && continue
+        case "$cname" in GITEA-ACTIONS-TASK*) continue ;; esac
+        lease_active "$lease_until" && continue
+        if [[ "$mode" != "all" ]] \
+            && ! grep -qE '(hour|day|week|month|year)s? ago' <<<"$running_for"; then
+            continue
+        fi
+        docker rm -f "$cid" >/dev/null 2>&1 || true
+    done
+}
+
+# Remove unused images older than $1 ("all" = no age limit). The keep-list
+# (warm base layers) and leased images are never removed; images referenced
+# by any container are refused by the daemon anyway. `docker images` has no
+# label formatter, so the lease is checked via inspect per candidate.
+remove_old_images() {
+    local until="$1"
+    docker image prune -f >/dev/null 2>&1 || true
+    local filters=(--filter "dangling=false")
+    [[ "$until" != "all" ]] && filters+=(--filter "until=${until}")
+    { timeout 30 docker images "${filters[@]}" \
+        --format '{% raw %}{{.ID}}|{{.Repository}}:{{.Tag}}{% endraw %}' \
+        2>/dev/null || true; } \
+    | while IFS='|' read -r iid ref; do
+        [[ -z "$iid" || "$ref" == *"<none>"* ]] && continue
+        [[ -n "$KEEP_IMAGES_RE" && "$ref" =~ $KEEP_IMAGES_RE ]] && continue
+        local lease_until
+        lease_until=$(docker image inspect "$iid" \
+            --format '{% raw %}{{index .Config.Labels "org.oblachno.lease-until"}}{% endraw %}' \
+            2>/dev/null || true)
+        lease_active "$lease_until" && continue
+        docker image rm "$iid" >/dev/null 2>&1 || true
+    done
+}
+
+# Reclaim volumes/networks whose lease expired. The prune filters below
+# skip every leased object (label!=); this pass removes the expired ones.
+reclaim_expired_leases() {
+    timeout 20 docker volume ls -q --filter "label=${LEASE_UNTIL_LABEL}" 2>/dev/null \
+    | while read -r vol; do
+        lease_until=$(docker volume inspect "$vol" \
+            --format '{% raw %}{{index .Labels "org.oblachno.lease-until"}}{% endraw %}' \
+            2>/dev/null || true)
+        lease_active "$lease_until" || docker volume rm "$vol" >/dev/null 2>&1 || true
+    done
+    timeout 20 docker network ls -q --filter "label=${LEASE_UNTIL_LABEL}" 2>/dev/null \
+    | while read -r net; do
+        lease_until=$(docker network inspect "$net" \
+            --format '{% raw %}{{index .Labels "org.oblachno.lease-until"}}{% endraw %}' \
+            2>/dev/null || true)
+        lease_active "$lease_until" || docker network rm "$net" >/dev/null 2>&1 || true
+    done
+}
+
+# Anonymous volumes only at routine/pressure tiers — a named volume may
+# belong to a job between create/attach steps. Critical removes all unused.
+prune_volumes_networks() {
+    local vol_all="$1" net_until="$2"
+    docker volume prune ${vol_all:+$vol_all} -f --filter "label!=${LEASE_UNTIL_LABEL}" >/dev/null 2>&1 || true
+    docker network prune -f --filter "label!=${LEASE_UNTIL_LABEL}" ${net_until:+--filter until=${net_until}} >/dev/null 2>&1 || true
+}
+
+case "$TIER" in
+    routine)
+        remove_stopped_containers aged
+        remove_old_images "{{ gitea_runner_prune_until }}"
+        prune_volumes_networks "" "{{ gitea_runner_prune_until }}"
+        docker builder prune -f --filter "until=24h" >/dev/null 2>&1 || true
+        ;;
+    pressure)
+        remove_stopped_containers aged
+        remove_old_images "1h"
+        prune_volumes_networks "" "1h"
+        docker builder prune -f --filter "until=24h" >/dev/null 2>&1 || true
+        ;;
+    critical)
+        # Implements: REQ-3 — age limits dropped, ownership still honored.
+        remove_stopped_containers all
+        remove_old_images all
+        prune_volumes_networks "-a" ""
+        docker builder prune -af >/dev/null 2>&1 || true
+        ;;
+    *)
+        echo "ERROR: unknown cleanup tier '$TIER' (expected routine|pressure|critical)" >&2
+        exit 2
+        ;;
+esac
+
+reclaim_expired_leases

ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2

@@ -265,54 +265,56 @@ except Exception:
 {% endif %}
 fi
 
-# 3. Check disk space — prune aggressively if below threshold
+# 3. Check disk space — scoped tiered cleanup via the shared cleanup script.
+# Implements: REQ-2/REQ-3 (GRM-173) — honors org.oblachno.lease-until leases,
+# keep-images, and the GITEA-ACTIONS-TASK exclusion; never removes running
+# containers. No unfiltered prune remains (the old `system prune -af
+# --volumes` could wipe a job's freshly created volumes mid-run).
 disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
+ADMISSION_MARKER="{{ gitea_runner_config_dir }}/disk-admission-block"
+CLEANUP_SCRIPT="{{ gitea_runner_cleanup_script_path }}"
+
 if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_critical }} ]]; then
-    echo "CRITICAL: Disk usage at ${disk_pct}% (>= {{ gitea_runner_healthcheck_disk_critical }}%), full prune"
-    # Critical level: remove ALL stopped containers (no age filter) and ALL
-    # unused images/volumes. The until=1h gentle prune is insufficient here.
-    # Implements: REQ-1 (GRM-167) — only exited/dead containers are removed.
-    # Running molecule instances are never killed: RunningFor counts creation
-    # time, so an adopted stale instance looks old; and a running container's
-    # writable layer is tiny — images/volumes are what actually fills the disk.
-    docker ps -a --filter "status=exited" --filter "status=dead" \
-        --format '{% raw %}{{.ID}} {{.Names}}{% endraw %}' 2>/dev/null \
-        | grep -v 'GITEA-ACTIONS-TASK' \
-        | awk '{print $1}' \
-        | xargs -r docker rm -f 2>/dev/null || true
-    docker system prune -af --volumes || true
-    docker network prune -f || true
-    docker builder prune -af || true
+    echo "CRITICAL: Disk usage at ${disk_pct}% (>= {{ gitea_runner_healthcheck_disk_critical }}%), critical cleanup"
+    "$CLEANUP_SCRIPT" --tier critical || true
     disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
-    echo "INFO: Disk usage after full prune: ${disk_pct}%"
+    echo "INFO: Disk usage after critical cleanup: ${disk_pct}%"
+
+    # Implements: REQ-4 — stop admitting new jobs while critically full,
+    # but only when no CI job is in flight (stopping the runner service
+    # mid-job would kill it). A later healthcheck resumes the service once
+    # disk drops below the warn threshold.
+{% if gitea_runner_disk_admission_enabled %}
+    in_flight=$(timeout 15 docker ps --filter "name=GITEA-ACTIONS-TASK" \
+        --format '{% raw %}{{.ID}}{% endraw %}' 2>/dev/null | wc -l || echo 0)
+    if [[ "$in_flight" -eq 0 ]] \
+        && systemctl --user is-active --quiet gitea-runner.service; then
+        echo "ADMISSION: disk critical, no jobs in flight — stopping runner service"
+        date +%s > "$ADMISSION_MARKER" 2>/dev/null || true
+        systemctl --user stop gitea-runner.service || true
+    elif [[ "$in_flight" -gt 0 ]]; then
+        echo "ADMISSION: disk critical but ${in_flight} job(s) in flight — runner left running"
+    fi
+{% endif %}
 elif [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
-    echo "WARN: Disk usage at ${disk_pct}%, pruning runner resources (until=1h)"
-    # Force-remove stale stopped containers older than 1 hour.
-    # Implements: REQ-1 (GRM-167) — only exited/dead containers are removed.
-    # A running molecule instance must never be janitor-killed: RunningFor
-    # measures creation time, so a stale instance restarted by an active run
-    # looks ">1h old" and would die mid-converge ("No such container",
-    # infra nightly run 5710). Running leftovers are reused or destroyed by
-    # the next molecule create/destroy cycle.
-    # Exclude CI job containers (name starts with GITEA-ACTIONS-TASK).
-    docker ps -a --filter "status=exited" --filter "status=dead" \
-        --format '{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}' 2>/dev/null \
-        | grep -v 'GITEA-ACTIONS-TASK' \
-        | grep -E '(hour|day|week|month|year)s? ago' \
-        | awk '{print $1}' \
-        | xargs -r docker rm -f 2>/dev/null || true
-    # Prune images and containers older than 1h (until filter is NOT
-    # supported with --volumes, so prune volumes separately without a filter).
-    docker image prune -af --filter "until=1h" 2>/dev/null || true
-    docker container prune -f --filter "until=1h" 2>/dev/null || true
-    docker volume prune -f 2>/dev/null || true
-    # Prune networks older than 1 hour to avoid removing networks that
-    # molecule tests are actively creating (e.g. 'traefik' network created
-    # during molecule create phase before containers are attached).
-    docker network prune -f --filter "until=1h" || true
+    echo "WARN: Disk usage at ${disk_pct}%, pressure cleanup (until=1h)"
+    "$CLEANUP_SCRIPT" --tier pressure || true
     disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
-    echo "INFO: Disk usage after prune: ${disk_pct}%"
+    echo "INFO: Disk usage after cleanup: ${disk_pct}%"
 fi
 
+# Implements: REQ-4 — resume admission once pressure has cleared.
+{% if gitea_runner_disk_admission_enabled %}
+if [[ -f "$ADMISSION_MARKER" ]]; then
+    if [[ "$disk_pct" -lt {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
+        echo "ADMISSION: disk recovered to ${disk_pct}% — resuming runner service"
+        rm -f "$ADMISSION_MARKER" 2>/dev/null || true
+        systemctl --user start gitea-runner.service || true
+    else
+        echo "ADMISSION: still blocked (disk ${disk_pct}%), runner stays stopped"
+    fi
+fi
+{% endif %}
+
 echo "OK: runner healthy, disk at ${disk_pct}%"
 exit 0

docs/index.md

@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
 
 [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
 [![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
-[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
-[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
-[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/python.svg)](https://www.python.org/downloads/)
+[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
+[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
+[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/65f88b7e01ee5e9bcdfaa3bdf2708ab68bca80e9/python.svg)](https://www.python.org/downloads/)
 
 ## Overview
 

docs/specs/GRM-171-kireto-token-historical.md

@@ -0,0 +1,28 @@
+# GRM-171: Use kireto token for auto-merge approval review
+
+## Problem
+The auto-merge workflow posts approval reviews with
+`REVIEWER_GITEA_API_TOKEN` (emil), but emil is also the PR creator.
+Gitea ignores self-approvals, so the merge fails with HTTP 405
+`Does not have enough approvals`.
+
+## Approach
+REQ-1: Change the approval review step in `.gitea/workflows/ci.yml` to use
+  `DEVELOPER_GITEA_API_TOKEN` (kireto) instead of
+  `REVIEWER_GITEA_API_TOKEN` (emil), since kireto is a different user
+  than the PR creator.
+
+## Test Plan
+- `make lint-all` passes (workflow-lint validates the YAML)
+- Next auto-merge PR succeeds (approval posted by kireto, merge completes)
+
+## Deploy Plan
+- Merge to master
+
+## Rollback Plan
+- Revert the merge commit
+
+## Acceptance Criteria
+- [x] REQ-1: Change the approval review step in `.gitea/workflows/ci.yml`
+  to use `DEVELOPER_GITEA_API_TOKEN` (kireto) instead of
+  `REVIEWER_GITEA_API_TOKEN` (emil)

docs/specs/GRM-171.md

@@ -1,28 +1,51 @@
-# GRM-171: Use kireto token for auto-merge approval review
+# GRM-171: Add runner-ops, molecule-testing, vikunja-tasks skills, fix create-task docs
 
 ## Problem
-The auto-merge workflow posts approval reviews with
-`REVIEWER_GITEA_API_TOKEN` (emil), but emil is also the PR creator.
-Gitea ignores self-approvals, so the merge fails with HTTP 405
-`Does not have enough approvals`.
+
+The OBL-INFRA-548 programme audit found grm lacks skills for runner
+fleet operations (needed for S08: leases, admission, watermarks),
+molecule scenario authoring, and Vikunja task lifecycle.
+`devx-workflow` and `AGENTS.md` document `make create-task -- --title`,
+which fails because `devx-create-task` forwards no arguments.
 
 ## Approach
-REQ-1: Change the approval review step in `.gitea/workflows/ci.yml` to use
-  `DEVELOPER_GITEA_API_TOKEN` (kireto) instead of
-  `REVIEWER_GITEA_API_TOKEN` (emil), since kireto is a different user
-  than the PR creator.
+
+REQ-1: Add `runner-ops` skill: RunnerManager/AnsibleExecutor model,
+  stale-runner cleanup, image pruning, molecule container lifecycle,
+  safe-debugging rules.
+REQ-2: Add `molecule-testing` skill: gitea_runner scenario layout,
+  platform overrides, isolation flags, debugging.
+REQ-3: Add `vikunja-tasks` skill: create via module call, query,
+  close, spec-collision convention.
+REQ-4: Fix broken `make create-task -- --title` documentation in
+  `devx-workflow` skill and `AGENTS.md`.
+REQ-5: Add skill validation tests (`tests/unit/test_skills_validation.py`)
+  + fix stale make-target refs and missing sections in existing skills.
+
+Preserve the colliding spec as
+[GRM-171-kireto-token-historical](GRM-171-kireto-token-historical.md).
 
 ## Test Plan
-- `make lint-all` passes (workflow-lint validates the YAML)
-- Next auto-merge PR succeeds (approval posted by kireto, merge completes)
+
+- `pytest tests/unit/test_skills_validation.py` passes (12 tests).
 
 ## Deploy Plan
-- Merge to master
+
+Documentation/skills only — auto-merge to master; no runtime deploy.
 
 ## Rollback Plan
-- Revert the merge commit
+
+Revert the squash-merge commit; skills are inert documentation.
 
 ## Acceptance Criteria
-- [x] REQ-1: Change the approval review step in `.gitea/workflows/ci.yml`
-  to use `DEVELOPER_GITEA_API_TOKEN` (kireto) instead of
-  `REVIEWER_GITEA_API_TOKEN` (emil)
+
+- [x] REQ-1: `runner-ops` skill exists.
+- [x] REQ-2: `molecule-testing` skill exists.
+- [x] REQ-3: `vikunja-tasks` skill exists.
+- [x] REQ-4: create-task docs corrected.
+- [x] REQ-5: Skill validation tests added and passing.
+
+## Out of Scope
+
+- Runner lease/admission implementation (S08 scope).
+- Fixing `devx-create-task` argument forwarding (devx repo, S11).

docs/specs/GRM-172.md

@@ -1,38 +1,43 @@
-# GRM-172: Audit and document pre-pull image usage guidelines
+# GRM-172: CI hygiene — docs fast-path, failure-notify scoping, post-merge cancel
 
 ## Problem
-The grm repo contains a runner-level `pre_pull_images.yml` task file that
-pre-pulls Docker images to avoid repeated pulls on every CI run. However,
-there was no audit confirming that molecule `prepare.yml` files are not
-also redundantly pre-pulling images that the runner setup already caches.
-Wasteful pre-pulling wastes CI time and disk space.
+
+GRM CI has the same inefficiencies fixed in infra (OBL-INFRA-613/615/616):
+docs-only PRs run the full quality suite, CI failures auto-create issues
+(noise — issues are for deploy failures only), and post-merge
+`cancel-in-progress: true` can kill a release mid-publish.
 
 ## Approach
-Audit all molecule `prepare.yml` files in the grm repo for pre-pull tasks.
-The audit found NO molecule prepare.yml files contain pre-pull tasks, so no
-code removal is needed. Document the audit findings in a spec and add a
-comment to the runner-level `pre_pull_images.yml` task file clarifying that
-it should not be used for images that molecule tests pull themselves (to
-avoid redundant pulls).
 
-REQ-1: Audit all molecule prepare.yml files for pre-pull tasks and confirm none exist
-REQ-2: Add documentation comment to pre_pull_images.yml stating it should not be used for CI runner container images (already cached by runner setup) or images molecule tests pull themselves
-REQ-3: Confirm gitea_runner_pre_pull_images default remains empty ([]) which is correct
+REQ-1: Docs-only PRs skip heavy validate steps (lint-all, unit tests,
+translation check, test-speed, security scan, workflow dry-run). Docs
+gate, spec validation, PR size, and auto-merge preconditions still run.
+Restricted to pull_request events.
+
+REQ-2: Remove the failure-issue step from `ci.yml` validate job.
+Post-merge keeps failure notification (release/publish failures are
+deploy-pipeline events).
+
+REQ-3: post-merge `cancel-in-progress: false` — queue instead of killing
+an in-flight release/publish.
 
 ## Test Plan
-- Grep all molecule prepare.yml files for pre-pull patterns confirms zero matches
-- Verify pre_pull_images.yml comment is present and accurate
-- Verify gitea_runner_pre_pull_images default is [] in defaults/main.yml
-- Run make lint-ci to confirm no lint regressions
+
+- `make workflow-lint` passes.
+- Docs-only PR: quality steps skipped, gates still run.
+- Non-docs PR: unchanged behavior.
 
 ## Deploy Plan
-- Merge to master via auto-merge workflow
-- No runtime changes; documentation-only
+
+Workflow-only change; takes effect on merge. No release needed.
 
 ## Rollback Plan
-- Revert the merge commit; comments are removed, no functional impact
+
+Revert the commit.
 
 ## Acceptance Criteria
-- [x] REQ-1: No molecule prepare.yml files in the grm repo contain pre-pull tasks (audit confirmed via grep)
-- [x] REQ-2: pre_pull_images.yml contains a comment documenting it should not be used for CI runner container images or images molecule tests pull themselves
-- [x] REQ-3: gitea_runner_pre_pull_images default remains empty ([]) in defaults/main.yml
+
+- [x] REQ-1 implemented — early docs-only step + step-level `if` gates
+- [x] REQ-2 implemented — notify step removed from ci.yml only
+- [x] REQ-3 implemented — post-merge concurrency flipped
+- [x] `make workflow-lint` passes

docs/specs/GRM-173-skills-historical.md

@@ -0,0 +1,46 @@
+# GRM-173: Add dependency-graph, deployment-coordination, and skill-creation skills
+
+## Problem
+Agents working across the oblachno ecosystem lack shared, written context
+for three recurring struggles: (1) knowing which repo produces what and
+the correct order for cross-repo changes, (2) coordinating grm releases
+with the downstream infra dependency PR, and (3) creating and validating
+new Devin skills consistently. Without these skills, agents repeatedly
+make mistakes such as deploying infra before the grm dependency PR is
+merged, or writing skills that fail the validator.
+
+## Approach
+Add three skill files under `.devin/skills/`. Two are shared skills
+(`dependency-graph`, `skill-creation`) that must be identical across
+repos; one is grm-specific (`deployment-coordination`). All three
+follow the standard skill structure (H1 title, When to Invoke,
+Prerequisites, core content) and reference real make targets, file
+paths, and API endpoints.
+
+REQ-1: Add `.devin/skills/dependency-graph/SKILL.md` — shared skill mapping the oblachno ecosystem (repos, produces/consumers, dependency chain, correct change order, state verification)
+REQ-2: Add `.devin/skills/deployment-coordination/SKILL.md` — grm-specific skill covering release flow, downstream consumer, coordinating a grm change, and common mistakes
+REQ-3: Add `.devin/skills/skill-creation/SKILL.md` — shared skill for creating, validating, and maintaining skills (structure, quality standards, scope rules, automated validation, checklist)
+
+## Files Affected
+- `.devin/skills/dependency-graph/SKILL.md` (new)
+- `.devin/skills/deployment-coordination/SKILL.md` (new)
+- `.devin/skills/skill-creation/SKILL.md` (new)
+- `docs/specs/GRM-173.md` (new)
+
+## Test Plan
+- Verify all three SKILL.md files follow the required structure (H1, When to Invoke, Prerequisites)
+- Verify referenced make targets and file paths are accurate
+- Run `make pytest-cov` to confirm no test regressions (skills are docs-only, no code changes)
+- Confirm shared skills (`dependency-graph`, `skill-creation`) are ready for cross-repo sync
+
+## Deploy Plan
+- Merge to master via auto-merge workflow
+- No runtime changes; documentation-only (`.devin/**` is infrastructure path, no release triggered)
+
+## Rollback Plan
+- Revert the merge commit; skill files are removed, no functional impact
+
+## Acceptance Criteria
+- [x] REQ-1: `.devin/skills/dependency-graph/SKILL.md` exists with ecosystem map, dependency chain, correct change order, and state verification sections
+- [x] REQ-2: `.devin/skills/deployment-coordination/SKILL.md` exists with release flow, downstream consumer table, coordination steps, and common mistakes
+- [x] REQ-3: `.devin/skills/skill-creation/SKILL.md` exists with skill structure template, quality standards, scope rules, automated validation, and creation checklist

docs/specs/GRM-173.md

@@ -1,46 +1,88 @@
-# GRM-173: Add dependency-graph, deployment-coordination, and skill-creation skills
+# GRM-173: Runner resource leases and scoped disk cleanup
 
 ## Problem
-Agents working across the oblachno ecosystem lack shared, written context
-for three recurring struggles: (1) knowing which repo produces what and
-the correct order for cross-repo changes, (2) coordinating grm releases
-with the downstream infra dependency PR, and (3) creating and validating
-new Devin skills consistently. Without these skills, agents repeatedly
-make mistakes such as deploying infra before the grm dependency PR is
-merged, or writing skills that fail the validator.
+
+Runner hosts protect in-flight work only via name-prefix and age heuristics:
+
+- The healthcheck critical tier runs unfiltered `docker system prune -af --volumes`
+  and `docker volume prune -f` — a job's momentarily unused volume/network can be
+  wiped mid-run, and warm base images are destroyed exactly when needed most.
+- Molecule containers owned by a live job are protected only by the
+  `GITEA-ACTIONS-TASK` naming convention, not by an ownership claim.
+- Cleanup logic is duplicated between `docker-prune.service` and the healthcheck.
+- No admission control: under disk pressure the runner keeps accepting jobs
+  while cleanup races in-flight work.
+- `capacity` is never declared; nothing prevents installing on a production host.
 
 ## Approach
-Add three skill files under `.devin/skills/`. Two are shared skills
-(`dependency-graph`, `skill-creation`) that must be identical across
-repos; one is grm-specific (`deployment-coordination`). All three
-follow the standard skill structure (H1 title, When to Invoke,
-Prerequisites, core content) and reference real make targets, file
-paths, and API endpoints.
 
-REQ-1: Add `.devin/skills/dependency-graph/SKILL.md` — shared skill mapping the oblachno ecosystem (repos, produces/consumers, dependency chain, correct change order, state verification)
-REQ-2: Add `.devin/skills/deployment-coordination/SKILL.md` — grm-specific skill covering release flow, downstream consumer, coordinating a grm change, and common mistakes
-REQ-3: Add `.devin/skills/skill-creation/SKILL.md` — shared skill for creating, validating, and maintaining skills (structure, quality standards, scope rules, automated validation, checklist)
+REQ-1: Define an ownership-lease label contract. Producers tag containers,
+images, volumes and networks with `org.oblachno.lease-until` (epoch seconds)
+and `org.oblachno.owner` (free-form run/job id). All cleanup paths must never
+remove an object whose `lease-until` is in the future; expired leases are
+reclaimable. Existing `GITEA-ACTIONS-TASK` name-prefix and `status=exited`
+guards are retained for unlabeled objects.
 
-## Files Affected
-- `.devin/skills/dependency-graph/SKILL.md` (new)
-- `.devin/skills/deployment-coordination/SKILL.md` (new)
-- `.devin/skills/skill-creation/SKILL.md` (new)
-- `docs/specs/GRM-173.md` (new)
+REQ-2: Introduce a single shared cleanup script (`runner-cleanup.sh`,
+templated next to the healthcheck script) invoked with
+`--tier routine|pressure|critical`, replacing all inline prune logic in
+`docker-prune.service` and the healthcheck. Every prune is scoped (leases,
+`until=` where supported); images matching `gitea_runner_keep_images` are
+never removed, so warm base layers survive critical pressure.
+
+REQ-3: Watermark-tiered behavior: `routine` (timer) prunes aged resources;
+`pressure` (disk >= warn) prunes unowned resources older than 1h; `critical`
+(disk >= critical) drops age limits but still honors leases, keep-images, and
+never removes running or `GITEA-ACTIONS-TASK` containers.
+
+REQ-4: Admission control under disk pressure. When disk is >= critical and no
+`GITEA-ACTIONS-TASK` container is running, the healthcheck writes a marker file
+and stops `gitea-runner.service` (the runner stops fetching jobs). A later
+healthcheck restarts it once disk drops below warn. In-flight jobs are never
+killed. Controlled by `gitea_runner_disk_admission_enabled`.
+
+REQ-5: Declare physical-host capacity explicitly: `runner.capacity:
+{{ gitea_runner_capacity }}` in the act_runner config (default 1 = upstream).
+
+REQ-6: Production-host exclusion. The role fails early when the target carries
+the marker file `/etc/oblachno/production-host` or
+`gitea_runner_on_production_host` is true, unless
+`gitea_runner_allow_production_host` overrides. Infra-side marker provisioning
+is a follow-up task.
+
+Historical spec for the colliding task ID:
+[GRM-173-skills-historical](GRM-173-skills-historical.md).
 
 ## Test Plan
-- Verify all three SKILL.md files follow the required structure (H1, When to Invoke, Prerequisites)
-- Verify referenced make targets and file paths are accurate
-- Run `make pytest-cov` to confirm no test regressions (skills are docs-only, no code changes)
-- Confirm shared skills (`dependency-graph`, `skill-creation`) are ready for cross-repo sync
+
+- `template-content` molecule scenario: prune service calls `runner-cleanup.sh`;
+  lease filters, keep-images and `capacity:` render correctly.
+- `default` scenario: cleanup script installed and executable.
+- `bash -n` on rendered templates; `make lint-all`, `make pytest-cov`, fast
+  molecule for the changed role.
 
 ## Deploy Plan
-- Merge to master via auto-merge workflow
-- No runtime changes; documentation-only (`.devin/**` is infrastructure path, no release triggered)
+
+- Merge via auto-merge; post-merge publishes the package and auto-creates the
+  infra dependency-bump PR. Runner hosts pick up the change on the next
+  `grm install`/update run — no manual host action.
+- Producer-side lease emission (molecule/CI jobs) is a separate devx change;
+  until then the guards degrade to the existing name-prefix/age behavior.
 
 ## Rollback Plan
-- Revert the merge commit; skill files are removed, no functional impact
+
+- Revert the merge commit and re-run `grm install` to redeploy the previous
+  prune/healthcheck units. No persistent state or migration.
 
 ## Acceptance Criteria
-- [x] REQ-1: `.devin/skills/dependency-graph/SKILL.md` exists with ecosystem map, dependency chain, correct change order, and state verification sections
-- [x] REQ-2: `.devin/skills/deployment-coordination/SKILL.md` exists with release flow, downstream consumer table, coordination steps, and common mistakes
-- [x] REQ-3: `.devin/skills/skill-creation/SKILL.md` exists with skill structure template, quality standards, scope rules, automated validation, and creation checklist
+
+- [x] REQ-1: `org.oblachno.lease-until`/`org.oblachno.owner` labels are honored
+  by every cleanup path; valid leases are never removed, expired leases are.
+- [x] REQ-2: single shared `runner-cleanup.sh` used by prune service and
+  healthcheck; no unfiltered `system prune --volumes`, `volume prune`, or
+  `network prune` remains; `gitea_runner_keep_images` never removed.
+- [x] REQ-3: three tiers behave as specified (routine/pressure/critical).
+- [x] REQ-4: critical pressure with zero in-flight job containers stops
+  admission via marker + service stop; recovery resumes automatically.
+- [x] REQ-5: `runner.capacity` rendered in `config.yaml`.
+- [x] REQ-6: role fails on production-marked hosts unless explicitly allowed.

src/grm/__init__.py

@@ -1,3 +1,3 @@
 """Gitea Runner Manager — lean CLI for managing Gitea Actions runners."""
 
-__version__ = "0.23.3"
+__version__ = "0.24.0"

tests/unit/test_skills_validation.py

@@ -0,0 +1,120 @@
+"""Pytest tests for Devin skill validation.
+
+Validates that all skills in .devin/skills/ are well-formed: H1 title,
+"when to invoke" section, prerequisites when commands are referenced,
+make-target references that exist, and file references that exist.
+
+Run with: make pytest TEST=tests/test_skills_validation.py
+"""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+
+# Sections required for every skill
+REQUIRED_SECTIONS = ["when to invoke"]
+
+# Sections required only for skills that reference commands/tools
+COMMAND_REQUIRED_SECTIONS = ["prerequisites"]
+
+# Markers indicating a skill references commands/tools
+COMMAND_MARKERS = ("`make ", "```bash", "```sh", "curl ", "python ", "python3 ", "ssh ")
+
+EXPECTED_SKILLS = [
+    "dependency-graph",
+    "deployment-coordination",
+    "devx-workflow",
+    "molecule-testing",
+    "pr-review",
+    "runner-ops",
+    "skill-creation",
+    "spec-driven-development",
+    "testing-and-debugging",
+    "vikunja-tasks",
+]
+
+
+def _find_skills() -> dict[str, Path]:
+    skills_dir = REPO_ROOT / ".devin" / "skills"
+    assert skills_dir.exists(), ".devin/skills/ directory not found"
+    return {d.name: d / "SKILL.md" for d in skills_dir.iterdir() if d.is_dir() and (d / "SKILL.md").exists()}
+
+
+# Skills shared with other repos — file-path references are only checked
+# in the owning repo (infra), where the referenced files live.
+SHARED_SKILLS = {"cross-repo-sync", "branch-hygiene", "dependency-graph", "skill-creation"}
+
+
+def _make_targets() -> set[str]:
+    """Collect make targets from Makefile plus included devx .mak files."""
+    targets: set[str] = set()
+    makefile = REPO_ROOT / "Makefile"
+    if makefile.exists():
+        targets.update(re.findall(r"^([a-zA-Z][a-zA-Z0-9_-]*):", makefile.read_text(), re.MULTILINE))
+    for mak in REPO_ROOT.glob(".venv/lib/python*/site-packages/devx/make/*.mak"):
+        targets.update(re.findall(r"^([a-zA-Z][a-zA-Z0-9_-]*):", mak.read_text(), re.MULTILINE))
+    return targets
+
+
+def _validate_skill(skill_name: str, skill_path: Path, make_targets: set[str]) -> list[str]:
+    """Validate a single skill file. Returns list of error messages."""
+    errors: list[str] = []
+    content = skill_path.read_text()
+
+    if not re.search(r"^# ", content, re.MULTILINE):
+        errors.append(f"{skill_name}: missing H1 title")
+
+    lower = content.lower()
+    for section in REQUIRED_SECTIONS:
+        if f"## {section}" not in lower:
+            errors.append(f"{skill_name}: missing '## {section.title()}' section")
+
+    references_commands = any(marker in content for marker in COMMAND_MARKERS)
+    if references_commands:
+        for section in COMMAND_REQUIRED_SECTIONS:
+            if f"## {section}" not in lower:
+                errors.append(
+                    f"{skill_name}: missing '## {section.title()}' section "
+                    "(required because skill references commands/tools)"
+                )
+
+    for target in re.findall(r"`make ([a-zA-Z][a-zA-Z0-9_-]*)`", content):
+        if target not in make_targets:
+            errors.append(f"{skill_name}: references `make {target}` but target does not exist")
+
+    # File-path checks: skip shared skills (checked in infra) and
+    # placeholder paths containing <...> templates.
+    if skill_name not in SHARED_SKILLS:
+        for match in re.findall(r"`((?:scripts|src|ansible|docs|tests|environments)/[^`\s]+)`", content):
+            if "<" in match:
+                continue
+            if not (REPO_ROOT / match).exists():
+                errors.append(f"{skill_name}: references `{match}` but file does not exist")
+
+    return errors
+
+
+@pytest.mark.parametrize("skill_name", EXPECTED_SKILLS)
+def test_skill_exists(skill_name: str) -> None:
+    """Each expected skill must have a SKILL.md."""
+    skill = REPO_ROOT / ".devin" / "skills" / skill_name / "SKILL.md"
+    assert skill.exists(), f"{skill_name}/SKILL.md not found"
+
+
+def test_minimum_skill_count() -> None:
+    """The repo should carry a working set of skills, not a stub."""
+    assert len(_find_skills()) >= 8, "expected >=10 skills"
+
+
+def test_all_skills_validate() -> None:
+    """All skills must pass structure/reference validation."""
+    make_targets = _make_targets()
+    errors: list[str] = []
+    for skill_name, skill_path in _find_skills().items():
+        errors.extend(_validate_skill(skill_name, skill_path, make_targets))
+    assert not errors, "Skill validation failed:\n" + "\n".join(f"  - {e}" for e in errors)