release: v0.23.3 [skip ci]

Co-authored-by: emo <emo@oblachno.com>

.devin/skills/dependency-graph/SKILL.md

@@ -0,0 +1,135 @@
+# dependency-graph
+
+Map of the oblachno ecosystem. Knows which repo produces what, which
+repos depend on which, and the correct order for cross-repo changes.
+
+## When to Invoke
+
+Invoke this skill when:
+- Changes span multiple repos
+- A change in one repo requires version bumps in downstream repos
+- Deploying infrastructure that depends on published packages/images
+- Verifying the ecosystem is in a consistent state before deployment
+- Determining which repos to update and in what order
+
+## Prerequisites
+
+- All repos cloned under `/home/emo/dev/ideas/oblachno/`
+- `.env` with `DEVELOPER_GITEA_API_TOKEN` in each repo
+
+## Ecosystem Map
+
+```
+                    devx (PyPI package)
+                   /    |    \
+                  /     |     \
+            grm   sso-bridge   infra
+           (PyPI)  (PyPI+Docker) (deploys all)
+              |        |           |
+              v        v           v
+         infra bump  infra bump   staging
+         (auto PR)   (auto PR)   production
+                                      |
+                          mattermost-oidc (Docker image)
+                          (infra pulls :latest at deploy)
+```
+
+## Repositories
+
+| Repo | Produces | Consumers | Release Trigger |
+|------|----------|-----------|-----------------|
+| `devx` | PyPI package `devx` | grm, sso-bridge, infra | User-facing changes to `src/devx/**` |
+| `grm` | PyPI package `grm` | infra | User-facing changes to `src/grm/**` or `ansible/**` |
+| `sso-bridge` | PyPI package `sso_bridge` + Docker image | infra | User-facing changes to `src/sso_bridge/**` or `ansible/**` |
+| `infra` | Staging/production deployment | (end users) | User-facing changes + nightly gate |
+| `mattermost-oidc` | Docker image `mattermost-oidc` | infra (pulls at deploy) | `Dockerfile` or `build.yml` changes |
+
+## Dependency Chain
+
+### devx → all repos
+
+devx publishes to the Gitea PyPI registry. grm, sso-bridge, and infra
+pin devx in `pyproject.toml`:
+```toml
+"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@vX.Y.Z"
+```
+
+When devx publishes a new version:
+1. grm, sso-bridge, and infra must bump their pinned devx version
+2. This is currently manual — no auto-dependency-PR from devx
+3. Each repo must `make setup` to pick up the new version
+
+### grm → infra
+
+grm publishes to PyPI. Its post-merge workflow auto-creates an infra
+dependency PR via `devx.ci.create_dependency_pr --repo oblachno/infra
+--package grm`. The PR bumps the pinned grm version in infra's
+`pyproject.toml`.
+
+### sso-bridge → infra
+
+sso-bridge publishes to PyPI AND builds a Docker image. Its post-merge
+workflow auto-creates an infra dependency PR via
+`devx.ci.create_dependency_pr --repo oblachno/infra --package
+sso_bridge`. The PR bumps the pinned sso_bridge version.
+
+The Docker image is pulled by infra at deploy time (`sso-bridge:latest`).
+
+### mattermost-oidc → infra
+
+mattermost-oidc builds a Docker image tagged `:latest` and `:MM_VERSION`.
+infra pulls `mattermost-oidc:latest` at deploy time. There is no
+auto-dependency-PR — infra simply pulls the latest image.
+
+### infra → staging/production
+
+infra deploys to staging and production. The deployment:
+1. Provisions VMs from golden images
+2. Runs Ansible roles (including grm and sso-bridge roles)
+3. Pulls Docker images (sso-bridge, mattermost-oidc)
+4. Configures services
+
+## Correct Order for Cross-Repo Changes
+
+When a change spans multiple repos, follow this order:
+
+1. **devx first** — if the change starts in devx, merge and publish devx
+   first. Wait for the PyPI publish job to complete.
+2. **Bump devx in consumers** — in grm/sso-bridge/infra, bump the pinned
+   devx version, run `make setup`, verify tests pass, merge.
+3. **grm/sso-bridge second** — merge and publish grm/sso-bridge. Wait
+   for the PyPI publish + Docker image build to complete.
+4. **Auto-dependency-PRs** — grm/sso-bridge post-merge auto-creates infra
+   PRs to bump pinned versions. Wait for these PRs to appear.
+5. **Merge infra dependency PRs** — review and merge the auto-created
+   infra PRs.
+6. **infra last** — deploy to staging, validate, promote to production.
+
+## State Verification Before Deployment
+
+Before deploying infra, verify:
+
+1. **devx version consistent** — all repos pin the same devx version
+2. **grm published** — latest grm tag exists in PyPI
+3. **sso-bridge published** — latest sso_bridge tag exists in PyPI
+4. **sso-bridge image built** — latest sso-bridge Docker image exists
+5. **mattermost-oidc image built** — latest mattermost-oidc image exists
+6. **infra pins match published versions** — no stale pins
+7. **Nightly gate green** — `NIGHTLY_STATUS` is not `failed`
+
+## Quick Check Commands
+
+```bash
+# Check latest devx version
+curl -sS https://git.oblachno.oblachno.fyi/api/v1/repos/oblachno-oss/devx/releases/latest | python3 -c "import json,sys; print(json.load(sys.stdin).get('tag_name','?'))"
+
+# Check pinned devx version in each repo
+for repo in grm sso-bridge infra; do
+  echo -n "$repo: "; grep 'devx @' /home/emo/dev/ideas/oblachno/$repo/pyproject.toml | grep -oP 'v[\d.]+'
+done
+
+# Check latest sso-bridge image build
+curl -sS -H "Authorization: token $DEVELOPER_GITEA_API_TOKEN" \
+  "https://git.oblachno.oblachno.fyi/api/v1/repos/oblachno/sso-bridge/actions/runs?per_page=5" \
+  | python3 -c "import json,sys; [print(r['id'],r['status'],r['conclusion']) for r in json.load(sys.stdin).get('workflow_runs',[]) if r.get('event')=='push']"
+```

.devin/skills/deployment-coordination/SKILL.md

@@ -0,0 +1,78 @@
+# deployment-coordination
+
+How grm releases propagate to infra. grm publishes a PyPI package and
+auto-creates an infra dependency PR. Coordination ensures the PR is
+merged before infra deploys.
+
+## When to Invoke
+
+Invoke this skill when:
+- Changes to grm affect infra deployments
+- Preparing a grm release that infra depends on
+- Verifying infra has bumped to the latest grm version
+- Coordinating a multi-repo change that includes grm
+
+## Prerequisites
+
+- grm repo at `/home/emo/dev/ideas/oblachno/grm`
+- `.env` with `DEVELOPER_GITEA_API_TOKEN`
+- See `dependency-graph` skill for the full ecosystem map
+
+## What grm Produces
+
+grm publishes a Python package to the Gitea PyPI registry. infra pins it:
+```toml
+"grm @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/grm.git@vX.Y.Z"
+```
+
+## Release Flow
+
+1. PR merged to master
+2. Post-merge workflow runs `devx.ci.release` — classifies changes
+3. If user-facing changes: git-cliff bumps version, creates tag, pushes
+4. `devx.ci.publish` builds and publishes to Gitea PyPI registry
+5. `devx.ci.create_dependency_pr --repo oblachno/infra --package grm`
+   auto-creates an infra PR to bump the pinned grm version
+
+## Downstream Consumer
+
+| Repo | Pin location | Auto-bump? |
+|------|-------------|------------|
+| infra | `pyproject.toml` | Yes — auto PR created by post-merge |
+
+## Coordinating a grm Change
+
+1. **Merge grm PR** — wait for post-merge publish + dependency-PR creation
+2. **Verify publish** — check the new tag:
+   ```bash
+   curl -sS -H "Authorization: token $DEVELOPER_GITEA_API_TOKEN" \
+     https://git.oblachno.oblachno.fyi/api/v1/repos/oblachno-oss/grm/releases/latest \
+     | python3 -c "import json,sys; print(json.load(sys.stdin).get('tag_name','?'))"
+   ```
+3. **Find the auto-created infra PR** — check infra for open PRs with
+   `dependency` label or title containing `bump grm`
+4. **Review and merge the infra dependency PR** — verify the version bump,
+   run `make pytest-cov` in infra, add `ready-to-merge`
+5. **Verify infra staging deploy** — after infra merges, staging deploy
+   picks up the new grm version
+
+## State Verification
+
+```bash
+# Current grm version
+grep '__version__' /home/emo/dev/ideas/oblachno/grm/src/grm/__init__.py
+
+# What infra pins
+grep 'grm @' /home/emo/dev/ideas/oblachno/infra/pyproject.toml | grep -oP 'v[\d.]+'
+
+# Check for open infra dependency PRs
+curl -sS -H "Authorization: token $DEVELOPER_GITEA_API_TOKEN" \
+  "https://git.oblachno.oblachno.fyi/api/v1/repos/oblachno/infra/pulls?state=open" \
+  | python3 -c "import json,sys; [print(p['number'],p['title']) for p in json.load(sys.stdin) if 'grm' in p.get('title','').lower()]"
+```
+
+## Common Mistakes
+
+- Merging the grm PR but ignoring the auto-created infra dependency PR
+- Deploying infra before the dependency PR is merged — stale grm version
+- Forgetting that grm also has an Ansible role used by infra at deploy time

.devin/skills/skill-creation/SKILL.md

@@ -0,0 +1,142 @@
+# skill-creation
+
+How to create, validate, and maintain Devin skills. Skills must be
+clear, succinct, and actionable — no AI slop.
+
+## When to Invoke
+
+Invoke this skill when:
+- Creating a new skill
+- Amending an existing skill
+- Evaluating whether a skill is needed
+- Reviewing a PR that adds or modifies skills
+
+## Prerequisites
+
+- Skill directory: `.devin/skills/<skill-name>/SKILL.md`
+- Validator: `tests/test_skills.py` (infra) or reference to it
+- Tests: `tests/unit/test_skills_validation.py` (infra)
+
+## When to Create a Skill
+
+Create a skill when:
+- An agent struggles with a task repeatedly (branch hygiene, PR order)
+- A workflow has non-obvious ordering constraints (deployment coordination)
+- A task requires specific tool usage over raw commands (CI monitoring)
+- Multiple agents need shared context (dependency graph)
+
+Do NOT create a skill for:
+- One-off tasks (use a spec instead)
+- Tasks already covered by AGENTS.md
+- Tasks that are obvious from the Makefile or README
+- Tasks that change frequently (skills should be stable)
+
+## Skill Structure
+
+Every skill MUST have:
+
+```markdown
+# <skill-name>
+
+One-line description of what the skill does.
+
+## When to Invoke
+
+2-4 bullet points describing when to use this skill.
+
+## Prerequisites
+
+What must exist before using the skill (venv, .env, tools).
+
+## <Core Content>
+
+The actual guidance. Keep it actionable.
+
+## Verification (if applicable)
+
+How to verify the skill's guidance works.
+
+## Common Mistakes (if applicable)
+
+What agents get wrong without this skill.
+```
+
+## Quality Standards
+
+### Do
+- **Be specific.** Reference exact make targets, file paths, commands.
+- **Be concise.** Each section should be scannable in under 30 seconds.
+- **Be actionable.** Every paragraph should tell the agent what to DO.
+- **Use tables** for command reference, mappings, and comparisons.
+- **Use code blocks** for commands the agent should run.
+- **Link to other skills** when related (e.g., "See `dependency-graph` skill").
+
+### Don't
+- **No preamble.** Don't start with "This skill helps agents..." — just state what it does.
+- **No filler.** Don't repeat information from AGENTS.md or other skills.
+- **No vague advice.** "Be careful with branches" is useless. "Run `git branch --show-current` before every commit" is useful.
+- **No AI slop.** Don't write "In this comprehensive guide, we will explore..." — just give the guidance.
+- **No redundant sections.** If "Common Mistakes" would repeat "When to Invoke", skip it.
+- **No marketing.** Don't describe the skill as "powerful" or "comprehensive".
+
+## Scope Rules
+
+- **One skill per concern.** Don't mix branch hygiene with CI monitoring.
+- **Project-specific, not generic.** Skills reference this repo's make targets, file paths, and conventions — not abstract advice.
+- **Shared skills must be identical across repos.** Use `SHARED_SKILLS` in the validator to enforce this.
+- **Per-repo skills must reflect that repo's reality.** Don't copy infra-specific targets to sso-bridge.
+
+## Automated Validation
+
+Every skill must pass the validator (`tests/test_skills.py`). The validator checks:
+
+1. **Structure** — H1 title, "When to Invoke" section, "Prerequisites" section
+2. **Commands** — referenced `make <target>` commands exist in Makefile or devx.mak
+3. **Paths** — referenced file paths exist in the repo
+4. **Shared skills** — identical content across repos (SHA-256 comparison)
+5. **No drift** — no references to nonexistent commands or files
+
+Run the validator:
+```bash
+python3 tests/test_skills.py --repo infra --repo sso-bridge
+```
+
+## Effectiveness Evaluation
+
+### Static Checks (automated, CI)
+
+The validator runs in CI as part of `make pytest-cov`. A failing skill
+test blocks the PR. This catches:
+- Missing sections
+- Invalid commands
+- Broken file references
+- Cross-repo drift
+
+### Runtime Metrics (manual, periodic)
+
+Track these signals to evaluate skill effectiveness:
+- **Skill invocation frequency** — how often agents invoke the skill
+- **Success rate when invoked** — did the skill prevent the mistake it targets?
+- **Feedback issues** — agents create Gitea issues with `feedback` label when a skill is unclear or wrong
+- **Mistake recurrence** — if agents still make the mistake the skill targets, the skill needs improvement
+
+### Retrospective Review
+
+Periodically (monthly or after major incidents) review skills:
+1. List all skills and their last-modified dates
+2. Check for feedback issues tagged `skill-improvement`
+3. Verify referenced commands still exist (run validator)
+4. Remove skills that are no longer relevant
+5. Update skills where mistakes still recur
+6. Document lessons in this skill's "Common Mistakes" section
+
+## Creating a New Skill — Checklist
+
+- [ ] Identify the repeated struggle or non-obvious workflow
+- [ ] Check no existing skill covers it
+- [ ] Write the skill following the structure above
+- [ ] Run `python3 tests/test_skills.py` — must pass
+- [ ] Run `make pytest-cov` — must pass with 100% coverage
+- [ ] If shared across repos, copy identical content to each repo
+- [ ] Add the skill to `SHARED_SKILLS` in the validator if shared
+- [ ] Create PR, verify CI passes, merge

.gitea/workflows/ci.yml

@@ -283,7 +283,7 @@ jobs:
       needs.validate.result == 'success'
     runs-on: docker
     container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
-    timeout-minutes: 10
+    timeout-minutes: 50
     defaults:
       run:
         shell: bash
@@ -321,7 +321,7 @@ jobs:
         run: |
           . .venv/bin/activate 2>/dev/null || true
           # Poll commit status until all required checks pass or fail
-          MAX_WAIT=600  # 10 minutes
+          MAX_WAIT=2400  # 40 minutes — covers the ~25-min molecule suite
           ELAPSED=0
           while [ $ELAPSED -lt $MAX_WAIT ]; do
             STATUS=$(curl -s -H "Authorization: token $CI_GITEA_API_TOKEN" \

CHANGELOG.md

@@ -2,6 +2,31 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.23.3] - 2026-09-18
+
+### Bug Fixes
+
+- Raise auto-merge molecule wait to cover suite duration
+- Harden stall-detection enumeration and timestamp parsing
+
+## [0.23.2] - 2026-09-15
+
+### Bug Fixes
+
+- Restrict healthcheck disk cleanup to exited/dead containers
+
+## [0.23.1] - 2026-09-15
+
+### Bug Fixes
+
+- Restrict docker-prune to exited containers
+
+## [0.23.0] - 2026-08-28
+
+### Features
+
+- Add pre-cache timer, force_pull, and Docker socket options to runner config
+
 ## [0.22.1] - 2026-08-26
 
 ### Bug Fixes

README.md

@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
 
 [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
 [![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
-[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
-[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
-[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/python.svg)](https://www.python.org/downloads/)
+[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
+[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
+[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/python.svg)](https://www.python.org/downloads/)
 
 ## Why GRM?
 

ansible/roles/gitea_runner/defaults/main.yml

@@ -50,6 +50,11 @@ gitea_runner_healthcheck_disk_threshold: 70
 gitea_runner_healthcheck_disk_critical: 75
 gitea_runner_healthcheck_script_path: "{{ gitea_runner_config_dir }}/healthcheck.sh"
 
+# CI job containers older than this many minutes get an exec-responsiveness
+# probe; a timeout writes one diagnostics bundle per container for
+# post-mortem analysis of recurring ~20min exec/archive stalls (GRM-168).
+gitea_runner_stall_minutes: 15
+
 # Auto-recovery: when the healthcheck detects an unregistered runner, it
 # can automatically re-register if a Gitea API token is provided.
 # The token needs admin or org-level access to fetch registration tokens.
@@ -90,6 +95,22 @@ gitea_runner_remove_user: true
 gitea_runner_log_level: "info"
 gitea_runner_container_label: "gitea-runner=true"
 gitea_runner_file: ".runner"
+# force_pull: when false (default), the runner reuses locally cached images
+# instead of pulling on every job. Pre-cached images (via the pre-cache timer
+# or pre_pull_images task) eliminate registry thundering-herd when all runners
+# start jobs simultaneously.
+gitea_runner_force_pull: false
+# Container options passed to `docker run` for CI job containers.
+# Mounts the host rootless Docker socket as /run/host-docker.sock so
+# start_docker.py inside the container can detect and use the host daemon
+# (full disk, no nested DinD) instead of starting an inner dockerd.
+gitea_runner_container_options: "-v /run/user/{{ gitea_runner_uid }}/docker.sock:/run/host-docker.sock"
+# Volumes allowed in CI job containers (validated by the runner against
+# container.options and job-level volumes). Must include the host Docker
+# socket mount target.
+gitea_runner_valid_volumes:
+  - "/run/host-docker.sock"
+  - "/run/user/{{ gitea_runner_uid }}/docker.sock"
 
 # Containerd version pinning — Docker 28.x vendors containerd v2.1.x internally.
 # containerd.io >= 2.3 ships a shim that returns a protobuf BootstrapResult which
@@ -135,3 +156,11 @@ gitea_runner_docker_ipv6_cidr: "fd00:dead:beef::/48"
 # disk-space prune only removes dangling images, so pre-pulled tagged images persist.
 # Set to [] to skip pre-pulling. Images are pulled as the runner user via rootless Docker.
 gitea_runner_pre_pull_images: []
+
+# Pre-cache timer: periodically pulls the runner container image so it stays
+# fresh in the local Docker cache. This prevents thundering-herd registry
+# timeouts when all runners start CI jobs simultaneously with empty caches.
+# Runs every 6 hours (aligned with prune schedule). Set to empty string to
+# disable the timer.
+gitea_runner_pre_cache_schedule: "*-*-* 00/6:30:00"
+gitea_runner_pre_cache_images: "{{ gitea_runner_pre_pull_images }}"

ansible/roles/gitea_runner/molecule/template-content/verify.yml

@@ -48,6 +48,7 @@
         that:
           - "'Type=oneshot' in prune_service.content | b64decode"
           - "'docker rm -f' in prune_service.content | b64decode"
+          - "'status=exited' in prune_service.content | b64decode"
           - "'GITEA-ACTIONS-TASK' in prune_service.content | b64decode"
           - "'docker system prune -af' in prune_service.content | b64decode"
           - "'docker network prune' in prune_service.content | b64decode"
@@ -111,6 +112,8 @@
           - "'docker network prune' in healthcheck_script.content | b64decode"
           - "'status=removing' in healthcheck_script.content | b64decode"
           - "'status=stopping' in healthcheck_script.content | b64decode"
+          - "'status=exited' in healthcheck_script.content | b64decode"
+          - "'status=dead' in healthcheck_script.content | b64decode"
           - "gitea_runner_healthcheck_disk_threshold | string in healthcheck_script.content | b64decode"
           - "gitea_runner_healthcheck_disk_critical | string in healthcheck_script.content | b64decode"
         fail_msg: "Healthcheck script template is missing expected content"

ansible/roles/gitea_runner/tasks/main.yml

@@ -20,6 +20,9 @@
 - name: Include pre-pull images
   ansible.builtin.include_tasks: pre_pull_images.yml
 
+- name: Include pre-cache timer
+  ansible.builtin.include_tasks: pre_cache.yml
+
 - name: Include integration test
   ansible.builtin.include_tasks: integration_test.yml
   when: not gitea_runner_skip_registration

ansible/roles/gitea_runner/tasks/pre_cache.yml

@@ -0,0 +1,67 @@
+---
+# Periodic timer that pre-pulls CI runner images into the local Docker cache.
+# Prevents thundering-herd registry timeouts when all runners start jobs
+# simultaneously with empty/stale caches. Runs every 6 hours (configurable).
+# The prune timer removes dangling images but NOT tagged ones, so pre-pulled
+# images persist between runs.
+
+- name: Create docker-pull-images user service file
+  ansible.builtin.template:
+    src: docker-pull-images.service.j2
+    dest: "{{ gitea_runner_home }}/.config/systemd/user/docker-pull-images.service"
+    owner: "{{ gitea_runner_service_user }}"
+    group: "{{ gitea_runner_service_user }}"
+    mode: "0644"
+  register: gitea_runner_pre_cache_service
+
+- name: Create docker-pull-images user timer file
+  ansible.builtin.template:
+    src: docker-pull-images.timer.j2
+    dest: "{{ gitea_runner_home }}/.config/systemd/user/docker-pull-images.timer"
+    owner: "{{ gitea_runner_service_user }}"
+    group: "{{ gitea_runner_service_user }}"
+    mode: "0644"
+  register: gitea_runner_pre_cache_timer
+
+- name: Reload systemd user daemon for pre-cache timer
+  ansible.builtin.command: systemctl --user daemon-reload
+  become: true
+  become_user: "{{ gitea_runner_service_user }}"
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
+  changed_when: true
+  when:
+    - gitea_runner_systemd_available.stat.exists
+    - gitea_runner_docker_rootless_setup
+    - gitea_runner_pre_cache_service is changed or gitea_runner_pre_cache_timer is changed
+    - gitea_runner_pre_cache_schedule | length > 0
+    - gitea_runner_pre_cache_images | length > 0
+
+- name: Enable and start docker-pull-images user timer
+  ansible.builtin.command: systemctl --user enable --now docker-pull-images.timer
+  become: true
+  become_user: "{{ gitea_runner_service_user }}"
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
+  changed_when: true
+  when:
+    - gitea_runner_systemd_available.stat.exists
+    - gitea_runner_docker_rootless_setup
+    - gitea_runner_pre_cache_schedule | length > 0
+    - gitea_runner_pre_cache_images | length > 0
+
+- name: Disable and stop docker-pull-images timer (no images or schedule)
+  ansible.builtin.command: systemctl --user disable --now docker-pull-images.timer
+  become: true
+  become_user: "{{ gitea_runner_service_user }}"
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+    DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
+  changed_when: true
+  failed_when: false
+  when:
+    - gitea_runner_systemd_available.stat.exists
+    - gitea_runner_docker_rootless_setup
+    - gitea_runner_pre_cache_schedule | length == 0 or gitea_runner_pre_cache_images | length == 0

ansible/roles/gitea_runner/tasks/pre_pull_images.yml

@@ -8,6 +8,15 @@
 #
 # Set gitea_runner_pre_pull_images to a list of image refs to pull, or
 # empty list to skip pre-pulling.
+#
+# IMPORTANT: Do NOT use this mechanism for:
+#   - CI runner container images (e.g. ci-full) — these are already
+#     cached by the runner setup task and pulling them here is redundant.
+#   - Images that molecule tests pull themselves — molecule prepare/converge
+#     steps handle their own image pulls; pre-pulling them here wastes time
+#     and disk space.
+# This mechanism is intended only for images that are needed by the runner
+# itself but not pulled by any molecule scenario or runner setup step.
 
 - name: Pre-pull Docker images for CI runner
   ansible.builtin.command: "docker pull {{ item }}"

ansible/roles/gitea_runner/templates/docker-prune.service.j2

@@ -5,16 +5,18 @@ Description=Docker prune for Gitea runner resources
 Type=oneshot
 Environment=DOCKER_HOST=unix:///run/user/{{ gitea_runner_uid }}/docker.sock
 Environment=XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
-# Force-remove stale containers (including running ones) left behind by failed
-# molecule tests. "docker container prune -f" only removes stopped containers,
-# so running containers from crashed/interrupted CI jobs accumulate indefinitely,
-# consuming disk and memory. We stop+rm everything first, then prune the rest.
+# Force-remove stale *stopped* containers left behind by failed molecule tests.
+# Implements: REQ-1 (GRM-166) — only containers with status=exited are
+# eligible. RunningFor measures creation time, so a stale molecule instance
+# (e.g. ubuntu-2604) that a new run restarts still looks ">1h old"; removing
+# running containers kills active converges with "No such container"
+# (infra nightly run 5710). Running leftovers are instead reused or destroyed
+# by the next molecule create/destroy cycle.
 # Exclude CI job containers (name starts with GITEA-ACTIONS-TASK) — removing
 # them kills the active CI job and causes "RWLayer is unexpectedly nil" errors.
 # Only remove containers older than 1 hour (grep for "hour/day/week/month/year
-# ago" in RunningFor) to avoid killing molecule test containers that CI jobs
-# are actively using.
-ExecStart=/bin/sh -c 'docker ps -a --format "{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}" 2>/dev/null | grep -v "GITEA-ACTIONS-TASK" | grep -E "(hour|day|week|month|year)s? ago" | awk "{print $1}" | xargs -r docker rm -f 2>/dev/null || true'
+# ago" in RunningFor) to avoid removing containers a job just created.
+ExecStart=/bin/sh -c 'docker ps -a --filter "status=exited" --format "{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}" 2>/dev/null | grep -v "GITEA-ACTIONS-TASK" | grep -E "(hour|day|week|month|year)s? ago" | awk "{print $1}" | xargs -r docker rm -f 2>/dev/null || true'
 ExecStart=/usr/bin/docker system prune -af --filter "until={{ gitea_runner_prune_until }}" --volumes
 # Prune networks older than the prune-until threshold to avoid removing
 # networks that molecule tests are actively creating (e.g. 'traefik' network

ansible/roles/gitea_runner/templates/docker-pull-images.service.j2

@@ -0,0 +1,14 @@
+[Unit]
+Description=Pre-pull Docker images for CI runner cache
+After=docker.service
+Wants=docker.service
+
+[Service]
+Type=oneshot
+Environment=DOCKER_HOST=unix:///run/user/{{ gitea_runner_uid }}/docker.sock
+Environment=XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
+# Pull each image quietly. docker pull exits 0 if image is already up-to-date,
+# so this is idempotent. Errors are non-fatal (image may already be cached).
+{% for image in gitea_runner_pre_cache_images %}
+ExecStart=/usr/bin/docker pull -q {{ image }}
+{% endfor %}

ansible/roles/gitea_runner/templates/docker-pull-images.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Periodic Docker image pre-cache for CI runner
+
+[Timer]
+OnCalendar={{ gitea_runner_pre_cache_schedule }}
+Persistent=true
+RandomizedDelaySec=300
+
+[Install]
+WantedBy=timers.target

ansible/roles/gitea_runner/templates/gitea-runner-config.yaml.j2

@@ -9,3 +9,13 @@ runner:
 container:
   label: "{{ gitea_runner_container_label }}"
   docker_host: "unix:///run/user/{{ gitea_runner_uid }}/docker.sock"
+  force_pull: {{ gitea_runner_force_pull | lower }}
+{% if gitea_runner_container_options | length > 0 %}
+  options: "{{ gitea_runner_container_options }}"
+{% endif %}
+{% if gitea_runner_valid_volumes | length > 0 %}
+  valid_volumes:
+{% for volume in gitea_runner_valid_volumes %}
+    - "{{ volume }}"
+{% endfor %}
+{% endif %}

ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2

@@ -30,6 +30,50 @@ if [[ -n "$stuck_containers" ]]; then
     echo "$stuck_containers" | xargs -r docker rm -f 2>/dev/null || true
 fi
 
+# 1c. Detect stalled CI job containers — Implements: REQ-1..REQ-4 (GRM-168)
+#     act_runner exec/archive calls into long-running job containers have
+#     repeatedly timed out ~20min into jobs while the daemon stayed up.
+#     Probe exec responsiveness on aged job containers and, on timeout,
+#     write one diagnostics bundle per container for post-mortem analysis.
+STALL_MINUTES={{ gitea_runner_stall_minutes }}
+DIAG_DIR="{{ gitea_runner_config_dir }}"
+now_epoch=$(date +%s)
+# Implements: REQ-1 — guard the enumeration: a slow/dead daemon must not
+# abort the healthcheck under pipefail; an empty list just skips probing.
+# Implements: REQ-2 — pipe-separate fields: CreatedAt contains spaces, so
+# whitespace-splitting `read` only captured the date and broke the age gate.
+{ timeout 15 docker ps --filter "name=GITEA-ACTIONS-TASK" \
+    --format '{% raw %}{{.ID}}|{{.Names}}|{{.CreatedAt}}{% endraw %}' 2>/dev/null || true; } \
+    | while IFS='|' read -r cid cname ccreated _rest; do
+    # GNU date rejects the redundant " +0000 UTC" suffix — drop it.
+    created_epoch=$(date -d "${ccreated% UTC}" +%s 2>/dev/null || echo 0)
+    age_min=$(( (now_epoch - created_epoch) / 60 ))
+    [[ "$age_min" -lt "$STALL_MINUTES" ]] && continue
+    marker="$DIAG_DIR/.stall-diag-$cid"
+    [[ -f "$marker" ]] && continue
+    if ! timeout 10 docker exec "$cid" true 2>/dev/null; then
+        diag="$DIAG_DIR/stall-diag-$cname-$(date +%Y%m%dT%H%M%S).log"
+        {
+            echo "=== stall diagnostics for $cname ($cid), age ${age_min}m ==="
+            echo "--- exec probe: TIMEOUT (>10s) ---"
+            echo "--- docker inspect ---"
+            # Implements: REQ-3 — full inspect, but redact the Env block:
+            # job containers carry CI tokens in env vars; the bundle must
+            # not become a secret-material artifact.
+            timeout 15 docker inspect "$cid" 2>/dev/null \
+                | sed -E 's/("[^"]*(TOKEN|PASSWORD|SECRET|KEY)[^=]*=)[^",]*/\1<redacted>/Ig'
+            echo "--- docker top ---"
+            timeout 15 docker top "$cid" 2>/dev/null
+            echo "--- docker stats --no-stream ---"
+            timeout 15 docker stats --no-stream "$cid" 2>/dev/null
+            echo "--- docker events --since 30m ---"
+            timeout 15 docker events --since 30m --until 0s 2>/dev/null | tail -50
+        } > "$diag" 2>&1 || true
+        touch "$marker"
+        echo "WARN: job container $cname unresponsive to exec (${age_min}m old) — diagnostics at $diag"
+    fi
+done
+
 # 2. Check gitea-runner service is active
 runner_state=$(systemctl --user is-active gitea-runner.service 2>/dev/null || true)
 if [[ "$runner_state" != "active" ]]; then
@@ -227,9 +271,12 @@ if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_critical }} ]]; then
     echo "CRITICAL: Disk usage at ${disk_pct}% (>= {{ gitea_runner_healthcheck_disk_critical }}%), full prune"
     # Critical level: remove ALL stopped containers (no age filter) and ALL
     # unused images/volumes. The until=1h gentle prune is insufficient here.
-    # Stop+rm stale non-CI containers regardless of age (failed molecule tests
-    # from the last 59 minutes also consume disk).
-    docker ps -a --format '{% raw %}{{.ID}} {{.Names}}{% endraw %}' 2>/dev/null \
+    # Implements: REQ-1 (GRM-167) — only exited/dead containers are removed.
+    # Running molecule instances are never killed: RunningFor counts creation
+    # time, so an adopted stale instance looks old; and a running container's
+    # writable layer is tiny — images/volumes are what actually fills the disk.
+    docker ps -a --filter "status=exited" --filter "status=dead" \
+        --format '{% raw %}{{.ID}} {{.Names}}{% endraw %}' 2>/dev/null \
         | grep -v 'GITEA-ACTIONS-TASK' \
         | awk '{print $1}' \
         | xargs -r docker rm -f 2>/dev/null || true
@@ -240,13 +287,16 @@ if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_critical }} ]]; then
     echo "INFO: Disk usage after full prune: ${disk_pct}%"
 elif [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
     echo "WARN: Disk usage at ${disk_pct}%, pruning runner resources (until=1h)"
-    # Force-remove stale containers (including running ones from failed molecule tests)
-    # that are older than 1 hour. "docker container prune -f" only removes stopped
-    # containers, so running containers from crashed CI jobs accumulate and consume
-    # disk/memory. Exclude CI job containers (name starts with GITEA-ACTIONS-TASK).
-    # Only remove containers older than 1 hour to avoid killing molecule test
-    # containers that CI jobs are actively using.
-    docker ps -a --format '{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}' 2>/dev/null \
+    # Force-remove stale stopped containers older than 1 hour.
+    # Implements: REQ-1 (GRM-167) — only exited/dead containers are removed.
+    # A running molecule instance must never be janitor-killed: RunningFor
+    # measures creation time, so a stale instance restarted by an active run
+    # looks ">1h old" and would die mid-converge ("No such container",
+    # infra nightly run 5710). Running leftovers are reused or destroyed by
+    # the next molecule create/destroy cycle.
+    # Exclude CI job containers (name starts with GITEA-ACTIONS-TASK).
+    docker ps -a --filter "status=exited" --filter "status=dead" \
+        --format '{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}' 2>/dev/null \
         | grep -v 'GITEA-ACTIONS-TASK' \
         | grep -E '(hour|day|week|month|year)s? ago' \
         | awk '{print $1}' \

docs/index.md

@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
 
 [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
 [![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
-[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
-[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
-[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
-[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/5ba6b90dffc24876035267268dd3780f7089264c/python.svg)](https://www.python.org/downloads/)
+[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
+[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
+[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
+[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/raw/commit/a0d7a6dd9b449a8135aadcc0547fca7397f56359/python.svg)](https://www.python.org/downloads/)
 
 ## Overview
 

docs/specs/GRM-162.md

@@ -0,0 +1,47 @@
+# GRM-162: Add pre-cache timer, force_pull, and Docker socket options to runner config
+
+## Problem
+CI containers were not using the host's rootless Docker daemon, leading to
+"no space left on device" errors. The runner config template was missing
+`force_pull`, `options` (host Docker socket mount), and `valid_volumes`
+fields. Additionally, no pre-cache timer existed to prevent thundering-herd
+registry timeouts when all runners pull images simultaneously.
+
+## Approach
+REQ-1: Add `force_pull: false` to runner config template (explicit default
+so the runner reuses locally cached images instead of pulling on every job)
+REQ-2: Add `options` field to mount host rootless Docker socket as
+`/run/host-docker.sock` so `start_docker.py` inside CI containers can detect
+and use the host daemon (full disk, no nested DinD)
+REQ-3: Add `valid_volumes` list for the socket mount targets (validated by
+the runner against `container.options` and job-level volumes)
+REQ-4: Add `pre_cache.yml` task with a systemd user timer that pre-pulls CI
+images every 6 hours (configurable via `gitea_runner_pre_cache_schedule`)
+REQ-5: Add `docker-pull-images.service.j2` and `docker-pull-images.timer.j2`
+templates for the pre-cache timer
+REQ-6: Timer is disabled when `gitea_runner_pre_cache_schedule` is empty or
+`gitea_runner_pre_cache_images` is empty (graceful degradation)
+
+## Test Plan
+- `make lint-ci` passes (ansible-lint on new task/template files)
+- `make molecule` converges successfully with the new pre-cache tasks
+- Verify the runner config template renders correctly with and without
+  container options/valid_volumes
+
+## Deploy Plan
+- Merge to master → post-merge auto-publishes package
+- Infra dependency PR auto-created to bump pinned grm version
+- Runners pick up the new config on next `make setup` or ansible apply
+
+## Rollback Plan
+- Revert the merge commit
+- Set `gitea_runner_pre_cache_schedule: ""` to disable the timer without
+  reverting
+
+## Acceptance Criteria
+- [x] REQ-1: `force_pull: false` in runner config template
+- [x] REQ-2: `options` field mounts host Docker socket as `/run/host-docker.sock`
+- [x] REQ-3: `valid_volumes` list includes both socket mount targets
+- [x] REQ-4: `pre_cache.yml` task creates and manages systemd user timer
+- [x] REQ-5: Service and timer templates created
+- [x] REQ-6: Timer disabled gracefully when schedule or images empty

docs/specs/GRM-166.md

@@ -0,0 +1,43 @@
+# GRM-166: Fix docker-prune killing running molecule containers
+
+## Problem
+`docker-prune.service` force-removes containers older than 1h by
+**creation** time (`RunningFor`). Molecule instance containers
+(`ubuntu-2604` etc.) run on the runner's shared rootless daemon and are
+not name-excluded (only `GITEA-ACTIONS-TASK` jobs are). A stale molecule
+container left by a crashed run is *restarted/reused* by the next run's
+create phase — it then shows `RunningFor` >1h and gets `docker rm -f`'d
+mid-converge: `No such container: ubuntu-2604`. Observed on infra nightly
+run 5710 across multiple runners (incl. a 32 GB host), causing
+shard failures.
+
+## Approach
+REQ-1: Change the container cleanup `ExecStart` in
+  `ansible/roles/gitea_runner/templates/docker-prune.service.j2` to only
+  target containers with `status=exited` (add
+  `--filter "status=exited"`). Running containers — including stale
+  molecule instances adopted by an active run — are never force-removed.
+REQ-2: Update the template comment to document the race and why only
+  stopped containers are removed.
+REQ-3: Extend the `template-content` molecule verify assertions to
+  require the `status=exited` filter in the rendered unit.
+
+## Test Plan
+- `make molecule` (template-content scenario) verifies the rendered unit.
+- `make pytest-cov` and `make lint-all` pass.
+
+## Deploy Plan
+- Merge to master → package publishes; runner hosts pick up the role on
+  their next grm install/upgrade cycle (or manual re-run of the role on
+  affected runners).
+- Until then, nightly molecule reruns remain exposed to the race —
+  mitigated by the fact that swept daemons have no stale containers left.
+
+## Rollback Plan
+- Revert the merge commit. Running stale leftovers would then again be
+  force-removed (the pre-existing risky behaviour).
+
+## Acceptance Criteria
+- [x] REQ-1: prune rm step filtered to `status=exited`
+- [x] REQ-2: comment documents the creation-time vs running race
+- [x] REQ-3: template-content verify asserts the exited filter

docs/specs/GRM-167-devx-bump-historical.md

@@ -0,0 +1,21 @@
+# GRM-167: Bump devx to v0.51.0
+
+## Problem
+devx v0.51.0 released with role defaults path support for create_dependency_pr. grm pins v0.50.2.
+
+## Approach
+Bump devx pin in pyproject.toml.
+
+REQ-1: Bump devx from v0.50.2 to v0.51.0 in pyproject.toml
+
+## Test Plan
+- Verify CI passes with new devx version
+
+## Deploy Plan
+- Merge to master, auto-release
+
+## Rollback Plan
+- Revert the merge commit
+
+## Acceptance Criteria
+- [x] REQ-1: devx pinned to v0.51.0 in pyproject.toml

docs/specs/GRM-167.md

@@ -1,21 +1,51 @@
-# GRM-167: Bump devx to v0.51.0
+# GRM-167: Fix runner healthcheck force-removing active molecule containers
 
 ## Problem
-devx v0.51.0 released with role defaults path support for create_dependency_pr. grm pins v0.50.2.
+`runner-healthcheck.sh` runs every 2 minutes and force-removes molecule
+instance containers on two disk paths:
+
+- **Warn path (disk >= 70%)**: `docker rm -f` any non-`GITEA-ACTIONS-TASK`
+  container with `RunningFor` >= 1h — same creation-age race fixed in
+  GRM-166 for the prune service. A stale molecule instance restarted by a
+  new run still reads >1h old and is killed mid-converge.
+- **Critical path (disk >= 75%)**: `docker rm -f` ALL
+  non-`GITEA-ACTIONS-TASK` containers with no age or status filter —
+  running molecule instances die instantly. Under parallel DinD load the
+  runner disk crosses 75% routinely (infra nightly run 5710: `No such
+  container: ubuntu-2604` across three different runners).
 
 ## Approach
-Bump devx pin in pyproject.toml.
-
-REQ-1: Bump devx from v0.50.2 to v0.51.0 in pyproject.toml
+REQ-1: In `ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2`,
+  restrict both disk-pressure removal paths to containers that are not
+  running: add `--filter "status=exited" --filter "status=created"` is
+  NOT sufficient for the critical path since a just-created molecule
+  instance is in `created` state — use `status=exited` and
+  `status=dead` only. Running containers are never janitor-killed; the
+  image/volume/system prunes still reclaim the actual disk.
+REQ-2: Keep `GITEA-ACTIONS-TASK` exclusion, the 1h age gate on the warn
+  path, and all image/volume/network/builder prunes unchanged.
+REQ-3: Update comments documenting the running-container race.
+REQ-4: Extend molecule `template-content` verify assertions for the
+  rendered healthcheck script (`status=exited` present in both paths).
 
 ## Test Plan
-- Verify CI passes with new devx version
+- `make molecule` template-content + default scenarios pass.
+- `make pytest-cov`, `make lint-all` pass.
 
 ## Deploy Plan
-- Merge to master, auto-release
+- Merge to master → package publishes. Runner hosts apply the role on
+  their next grm install/upgrade run; affected runners may need a manual
+  role re-run for immediate relief.
+- Vikunja counter reused GRM-167; prior spec preserved in
+  [GRM-167-devx-bump-historical](GRM-167-devx-bump-historical.md).
 
 ## Rollback Plan
-- Revert the merge commit
+- Revert the merge commit — restores the aggressive cleanup that kills
+  active CI jobs.
 
 ## Acceptance Criteria
-- [x] REQ-1: devx pinned to v0.51.0 in pyproject.toml
+- [x] REQ-1: both disk paths only remove `status=exited`/`status=dead`
+  containers
+- [x] REQ-2: exclusions, age gate, and non-container prunes unchanged
+- [x] REQ-3: comments updated
+- [x] REQ-4: template-content verify covers the filters

docs/specs/GRM-168.md

@@ -1,21 +1,84 @@
-# GRM-168: Bump devx to v0.51.9
+# GRM-168: Capture dockerd diagnostics when a CI job container stalls
 
 ## Problem
-grm pins devx@v0.51.0 which rejects `deps:` as a conventional commit type,
-causing post-merge CI failures on dependency bump commits.
+
+Recurring CI failures (5+ times on 2026-09-17/18, infra runs 5913, 5925,
+5943, 5955 notify-sso-bridge): ~20 min into a long-running job,
+act_runner's API calls into the job container (`docker exec`, archive
+fetch of `/var/run/act/workflow/*.txt`) time out with
+`docker daemon ping during version negotiation failed /
+context deadline exceeded` — killing the job.
+
+Established facts:
+
+- Host rootless dockerd never restarted (all daemons up since Sep 14);
+  the healthcheck's 10 s `docker info` never timed out — the daemon API
+  stayed responsive at daemon level.
+- No OOM, disk, inode, or load pressure on the host.
+- The wedge is therefore per-container (shim/exec path), most consistent
+  with attach-stdio backpressure or a containerd-shim event stall — but
+  cannot be confirmed post-mortem because job containers and their
+  dockerd goroutine state are gone by the time anyone looks.
+
+A `SIGUSR1` dockerd dump is not useful here: it lands in the user
+journal, which runner users cannot read (2026-08-08 journal-permission
+incident documented in this file's header comments).
 
 ## Approach
-REQ-1: Bump devx from v0.51.0 to v0.51.9 in pyproject.toml
+
+Extend `runner-healthcheck.sh.j2` with a stall-detection section that
+runs after the daemon liveness check. On every healthcheck tick (2 min):
+
+REQ-1: For each running `GITEA-ACTIONS-TASK-*` container older than
+`gitea_runner_stall_minutes` (default 15), probe exec responsiveness
+with `timeout 10 docker exec <id> true`.
+
+REQ-2: If the probe times out, write a diagnostics bundle to
+`{{ gitea_runner_config_dir }}/stall-diag-<container>-<timestamp>.log`
+containing: probe result, `docker inspect` output (State, OOMKilled,
+Pid, finished/started times), `docker top` output, `docker stats
+--no-stream` for the container, and `docker events --since 30m` output.
+Each line prefixed with the container name for grepability.
+
+REQ-3: Cooldown per container — write at most one diagnostics bundle
+per container id (marker file under the same dir), so a 2-minute
+healthcheck does not spam dumps on a persistent stall.
+
+REQ-4: Do not kill or restart anything — diagnostics only. The job may
+recover on its own; if it does not, the captured evidence isolates
+shim-vs-daemon and stream-vs-exec for the follow-up fix.
+
+## Files Affected
+
+- `ansible/roles/gitea_runner/templates/runner-healthcheck.sh.j2` (extend)
+- `ansible/roles/gitea_runner/defaults/main.yml` (add `gitea_runner_stall_minutes`)
+- `docs/specs/GRM-168.md` (new)
 
 ## Test Plan
-- `make lint-all` passes
-- `make pytest-cov` passes
+
+- `make lint-all` (ansible-lint + shellcheck-adjacent linters) passes.
+- Molecule fast-converge on the gitea_runner role scenario that deploys
+  the healthcheck template (template renders without error).
+- Manual trace: the new section only touches containers matching
+  `GITEA-ACTIONS-TASK-*` older than the threshold; a stalled exec probe
+  writes exactly one bundle per container.
 
 ## Deploy Plan
-- Merge to master
+
+Merge via auto-merge → GRM release → infra picks up the new version via
+the automated dependency PR. Runner hosts get the updated healthcheck on
+the next `gitea_runner` role apply (nightly or manual run).
 
 ## Rollback Plan
-- Revert the merge commit
+
+Revert the template change — the healthcheck returns to the previous
+probe set. The diagnostics path is additive; removing it risks nothing.
 
 ## Acceptance Criteria
-- [x] REQ-1: Bump devx from v0.51.0 to v0.51.9 in pyproject.toml
+
+- [x] Stalled job containers probed via `timeout docker exec`.
+- [x] One diagnostics bundle per stalled container, written to the
+      runner config dir (readable without journal access).
+- [x] Per-container cooldown prevents dump spam.
+- [x] Nothing is killed/restarted — diagnostics only.
+- [x] `make lint-all` passes.

docs/specs/GRM-169.md

@@ -0,0 +1,48 @@
+# GRM-169: Fix auto-merge timeout — molecule wait exceeds 10-min job cap
+
+## Problem
+
+The `auto-merge` job in `ci.yml` polls molecule-tests status with
+`MAX_WAIT=600` (10 minutes) inside a job capped at
+`timeout-minutes: 10`. The molecule suite takes ~25 minutes under the
+4-runner distribution. Result: every `pull_request` synchronize run of
+auto-merge exhausts MAX_WAIT, prints `Timed out waiting for molecule
+tests`, and fails — observed on PR #275 (run 5988) where all molecule
+jobs were green but auto-merge died before they finished. The merge
+only completed via a manual rerun-failed-jobs call after molecule was
+already green.
+
+## Approach
+
+REQ-1: Raise the molecule wait budget in `.gitea/workflows/ci.yml` so it
+exceeds the observed suite duration: `MAX_WAIT=2400` (40 minutes — ~1.6x
+the observed 25-minute suite) and the job `timeout-minutes` to `50`
+(wait budget plus setup/post overhead).
+
+REQ-2: No other behavior changes — the wait loop, success/failure/skipped
+classification, and merge semantics stay identical. The job still runs
+on every pull_request event; it simply no longer aborts early.
+
+## Test Plan
+
+- `make workflow-lint` (actionlint) passes on the edited file.
+- `make workflow-dryrun` where available.
+- Next PR's auto-merge run waits past the 10-minute mark and merges
+  after molecule turns green (verified on a subsequent PR).
+
+## Deploy Plan
+
+Merge via auto-merge — ironically exercised by this very PR's auto-merge
+run: it must wait for this PR's own molecule jobs, demonstrating the fix
+in production immediately.
+
+## Rollback Plan
+
+Revert the two changed lines. Risk of keeping the fix: none — a longer
+wait can only extend a job that was previously guaranteed to fail.
+
+## Acceptance Criteria
+
+- [x] `MAX_WAIT` raised to 2400 in the auto-merge wait loop.
+- [x] `timeout-minutes` raised to 50 on the auto-merge job.
+- [x] `make workflow-lint` passes.

docs/specs/GRM-170.md

@@ -0,0 +1,73 @@
+# GRM-170: Fix stall-detection robustness bugs in runner healthcheck
+
+## Problem
+
+Post-merge review of the GRM-168 stall-detection block in
+`runner-healthcheck.sh.j2` found three defects:
+
+1. **Missing `|| true` on the container enumeration.** `timeout 15
+   docker ps … | while …` runs under `set -euo pipefail`. If the daemon
+   is unresponsive — precisely the condition the section exists to
+   diagnose — `docker ps` exits nonzero, pipefail propagates it, and the
+   healthcheck dies mid-run before reaching the runner-service check.
+   Every other docker call in the script is guarded; this one is not.
+
+2. **CreatedAt split bug.** `docker ps --format '{{.ID}} {{.Names}}
+   {{.CreatedAt}}'` emits a timestamp containing spaces
+   (`2026-09-18 10:30:00 +0000 UTC`), but `read -r cid cname ccreated
+   _rest` only captures `2026-09-18` — the date part. `date -d` then
+   computes age from midnight: containers created today always appear
+   ≥N hours old, so the 15-minute gate effectively never filters.
+
+3. **`head -200` truncates `docker inspect`.** Inspect output is ~300+
+   lines and the `State` block (OOMKilled, Pid, times) the spec requires
+   can be cut off.
+
+## Approach
+
+REQ-1: Wrap the enumeration so a failed `docker ps` yields empty input
+instead of aborting the script: `{ timeout 15 docker ps … || true; } |
+while …`.
+
+REQ-2: Emit fields separated by `|` (`{{.ID}}|{{.Names}}|{{.CreatedAt}}`)
+and parse with `IFS='|' read -r cid cname ccreated _rest` so the full
+timestamp reaches `date -d`; also strip the redundant ` UTC` suffix
+because GNU date rejects `+0000 UTC` together. The age gate then
+compares real minutes.
+
+REQ-3: Remove the `head -200` truncation on `docker inspect` output so
+the full State block is captured — but pipe through a `sed` filter that
+redacts the value of any env entry whose name contains TOKEN, PASSWORD,
+SECRET, or KEY. Job containers carry CI tokens in their Env block; the
+diagnostics bundle must not become a secret-material artifact
+(OBL-INFRA-548 S02).
+
+REQ-4: Diagnostics-only constraint unchanged — no kills, no restarts.
+
+## Test Plan
+
+- Render the template and run `bash -n` on the output.
+- Shell-simulate: feed a fake `docker ps` line with spaced CreatedAt and
+  verify `date -d` computes minutes correctly (manual check).
+- `make lint-all` (ansible-lint, actionlint, ruff) passes.
+- Molecule gitea_runner scenario converges with the template change.
+
+## Deploy Plan
+
+Merge via auto-merge → release (fix: commit bumps patch) → infra
+dependency-bump PR picks up the new role version → runner role applied
+on next infra run. This PR also carries the merged-but-unreleased
+GRM-168 healthcheck into the release.
+
+## Rollback Plan
+
+Revert the three-line change set; the section degrades to the GRM-168
+behavior (still diagnostics-only, just less robust).
+
+## Acceptance Criteria
+
+- [x] `docker ps` enumeration guarded against nonzero exit.
+- [x] Full CreatedAt timestamp parsed via `|` separator.
+- [x] `docker inspect` captured without truncation.
+- [x] Rendered script passes `bash -n`.
+- [x] `make lint-all` passes.

docs/specs/GRM-172.md

@@ -0,0 +1,38 @@
+# GRM-172: Audit and document pre-pull image usage guidelines
+
+## Problem
+The grm repo contains a runner-level `pre_pull_images.yml` task file that
+pre-pulls Docker images to avoid repeated pulls on every CI run. However,
+there was no audit confirming that molecule `prepare.yml` files are not
+also redundantly pre-pulling images that the runner setup already caches.
+Wasteful pre-pulling wastes CI time and disk space.
+
+## Approach
+Audit all molecule `prepare.yml` files in the grm repo for pre-pull tasks.
+The audit found NO molecule prepare.yml files contain pre-pull tasks, so no
+code removal is needed. Document the audit findings in a spec and add a
+comment to the runner-level `pre_pull_images.yml` task file clarifying that
+it should not be used for images that molecule tests pull themselves (to
+avoid redundant pulls).
+
+REQ-1: Audit all molecule prepare.yml files for pre-pull tasks and confirm none exist
+REQ-2: Add documentation comment to pre_pull_images.yml stating it should not be used for CI runner container images (already cached by runner setup) or images molecule tests pull themselves
+REQ-3: Confirm gitea_runner_pre_pull_images default remains empty ([]) which is correct
+
+## Test Plan
+- Grep all molecule prepare.yml files for pre-pull patterns confirms zero matches
+- Verify pre_pull_images.yml comment is present and accurate
+- Verify gitea_runner_pre_pull_images default is [] in defaults/main.yml
+- Run make lint-ci to confirm no lint regressions
+
+## Deploy Plan
+- Merge to master via auto-merge workflow
+- No runtime changes; documentation-only
+
+## Rollback Plan
+- Revert the merge commit; comments are removed, no functional impact
+
+## Acceptance Criteria
+- [x] REQ-1: No molecule prepare.yml files in the grm repo contain pre-pull tasks (audit confirmed via grep)
+- [x] REQ-2: pre_pull_images.yml contains a comment documenting it should not be used for CI runner container images or images molecule tests pull themselves
+- [x] REQ-3: gitea_runner_pre_pull_images default remains empty ([]) in defaults/main.yml

docs/specs/GRM-173.md

@@ -0,0 +1,46 @@
+# GRM-173: Add dependency-graph, deployment-coordination, and skill-creation skills
+
+## Problem
+Agents working across the oblachno ecosystem lack shared, written context
+for three recurring struggles: (1) knowing which repo produces what and
+the correct order for cross-repo changes, (2) coordinating grm releases
+with the downstream infra dependency PR, and (3) creating and validating
+new Devin skills consistently. Without these skills, agents repeatedly
+make mistakes such as deploying infra before the grm dependency PR is
+merged, or writing skills that fail the validator.
+
+## Approach
+Add three skill files under `.devin/skills/`. Two are shared skills
+(`dependency-graph`, `skill-creation`) that must be identical across
+repos; one is grm-specific (`deployment-coordination`). All three
+follow the standard skill structure (H1 title, When to Invoke,
+Prerequisites, core content) and reference real make targets, file
+paths, and API endpoints.
+
+REQ-1: Add `.devin/skills/dependency-graph/SKILL.md` — shared skill mapping the oblachno ecosystem (repos, produces/consumers, dependency chain, correct change order, state verification)
+REQ-2: Add `.devin/skills/deployment-coordination/SKILL.md` — grm-specific skill covering release flow, downstream consumer, coordinating a grm change, and common mistakes
+REQ-3: Add `.devin/skills/skill-creation/SKILL.md` — shared skill for creating, validating, and maintaining skills (structure, quality standards, scope rules, automated validation, checklist)
+
+## Files Affected
+- `.devin/skills/dependency-graph/SKILL.md` (new)
+- `.devin/skills/deployment-coordination/SKILL.md` (new)
+- `.devin/skills/skill-creation/SKILL.md` (new)
+- `docs/specs/GRM-173.md` (new)
+
+## Test Plan
+- Verify all three SKILL.md files follow the required structure (H1, When to Invoke, Prerequisites)
+- Verify referenced make targets and file paths are accurate
+- Run `make pytest-cov` to confirm no test regressions (skills are docs-only, no code changes)
+- Confirm shared skills (`dependency-graph`, `skill-creation`) are ready for cross-repo sync
+
+## Deploy Plan
+- Merge to master via auto-merge workflow
+- No runtime changes; documentation-only (`.devin/**` is infrastructure path, no release triggered)
+
+## Rollback Plan
+- Revert the merge commit; skill files are removed, no functional impact
+
+## Acceptance Criteria
+- [x] REQ-1: `.devin/skills/dependency-graph/SKILL.md` exists with ecosystem map, dependency chain, correct change order, and state verification sections
+- [x] REQ-2: `.devin/skills/deployment-coordination/SKILL.md` exists with release flow, downstream consumer table, coordination steps, and common mistakes
+- [x] REQ-3: `.devin/skills/skill-creation/SKILL.md` exists with skill structure template, quality standards, scope rules, automated validation, and creation checklist

src/grm/__init__.py

@@ -1,3 +1,3 @@
 """Gitea Runner Manager — lean CLI for managing Gitea Actions runners."""
 
-__version__ = "0.22.1"
+__version__ = "0.23.3"