Compare commits
17
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3ba2ea8268 | ||
|
|
13aab5539a | ||
|
|
998e438270 | ||
|
|
eab452ec04 | ||
|
|
9609ef2505 | ||
|
|
97e7687e25 | ||
|
|
f672da1753 | ||
|
|
d6549de2e0 | ||
|
|
dc4bb0936d | ||
|
|
52477e558a | ||
|
|
28214de583 | ||
|
|
7fd023d192 | ||
|
|
2ffedaa793 | ||
|
|
e7b2d4e6af | ||
|
|
bda2af91bc | ||
|
|
c72dc97f63 | ||
|
|
db9fb6f162 |
@@ -12,7 +12,6 @@ Quick reference for devx tools when working on this repo.
|
||||
| Check CI status | `make devx-pr-status` or `make devx-pr-status PR=42 WAIT=1` |
|
||||
| Fetch CI failure logs | `make devx-pr-logs` or `make devx-pr-logs PR=42 JOB=quality TAIL=50` |
|
||||
| Add ready-to-merge label | `make devx-pr-label` or `make devx-pr-label PR=42` |
|
||||
| Post PR review | `make devx-pr-review PR=42 EVENT=APPROVE BODY="..." CHECKLIST=1,2,3,4,5,6,7,8,9,10,11,12,13` |
|
||||
| Rebase current branch | `make rebase` |
|
||||
| Rebase PR via API | `make pr-rebase` or `make pr-rebase PR=42` |
|
||||
|
||||
@@ -30,6 +29,25 @@ CI runs a `pre-merge-check` job early (after quality + detect-changes)
|
||||
that validates branch format, PR title, and Vikunja task match.
|
||||
This fails fast before expensive molecule tests run.
|
||||
|
||||
## Spec-Driven CI Gates (Pre-merge)
|
||||
|
||||
Every PR must pass these gates before merge:
|
||||
|
||||
| Gate | Module | What it checks |
|
||||
|------|--------|----------------|
|
||||
| Spec validation | `devx.ci.validate_spec` | Spec file exists at `docs/specs/<TASK-ID>.md`, has REQ-IDs, all ACs checked |
|
||||
| PR size | `devx.ci.check_pr_size` | Max 500 lines / 10 files (excludes CHANGELOG, badges, locks) |
|
||||
|
||||
Full molecule tests still run on every PR (6 scenarios, all platforms).
|
||||
|
||||
## Post-merge Auto-publish + Dependency PR
|
||||
|
||||
After merge to master, `post-merge.yml`:
|
||||
1. Runs release (git-cliff semver, tags, publishes to Gitea PyPI)
|
||||
2. Auto-creates an infra dependency PR (`devx.ci.create_dependency_pr`)
|
||||
to bump the pinned grm version in `infra/pyproject.toml`
|
||||
3. Syncs wiki, updates Vikunja task, pushes badges
|
||||
|
||||
## Key Rules
|
||||
|
||||
- Never manually merge via API — always use auto-merge with `ready-to-merge` label
|
||||
|
||||
@@ -0,0 +1,272 @@
|
||||
# pr-review
|
||||
|
||||
Deep, critical PR review with auto-fix. This skill guides the agent
|
||||
through a thorough review of a pull request, posting inline comments
|
||||
for each issue found, auto-fixing them, resolving the discussion threads,
|
||||
and marking the PR as ready-to-merge when no blocking issues remain.
|
||||
|
||||
## When to Invoke
|
||||
|
||||
Invoke this skill when asked to review a PR, or when a PR is open and
|
||||
needs review before merge. Do NOT invoke automatically on every PR —
|
||||
this is an on-demand deep review, not a CI gate.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The PR must be open in a Gitea repo
|
||||
- The agent needs Gitea MCP access (gitea server)
|
||||
- The agent needs git push access to the PR's head branch
|
||||
- The PR should have passed CI (validate job) before deep review
|
||||
|
||||
## Review Categories
|
||||
|
||||
Review every PR against these 8 categories. For each issue found, post
|
||||
an inline comment on the specific line, then auto-fix it.
|
||||
|
||||
### 1. Functional Correctness
|
||||
|
||||
- Does the code actually do what the spec/PR title claims?
|
||||
- Are edge cases handled? (empty input, null, boundary values, concurrent access)
|
||||
- Are error paths tested? Not just happy path.
|
||||
- Does the code handle all return values? (ignored errors, unchecked None)
|
||||
- Are there off-by-one errors, wrong comparisons, inverted conditions?
|
||||
- Do loops terminate correctly? (no infinite loops, correct break/continue)
|
||||
- Are regex patterns correct? (anchored, escaped, non-greedy where needed)
|
||||
- Are API responses validated before use? (status codes, response shape)
|
||||
|
||||
### 2. Completeness
|
||||
|
||||
- Are all requirements from the spec implemented? (check each REQ-ID)
|
||||
- Are all acceptance criteria in the spec checked off?
|
||||
- Are tests written for all new code paths?
|
||||
- Are error messages user-facing (wrapped in `_()`)?
|
||||
- Are new CLI commands documented in `docs/user/cli-commands.md`?
|
||||
- Are new modules added to architecture docs?
|
||||
- Are CHANGELOG entries added for user-facing changes?
|
||||
- Are translations added for new user-facing strings?
|
||||
|
||||
### 3. Architecture
|
||||
|
||||
- Does the code follow the repo's layer separation? (no business logic in CLI, no direct subprocess in CLI)
|
||||
- Are new dependencies justified? (no unnecessary new packages)
|
||||
- Is configuration via env vars / config.py, not hardcoded?
|
||||
- Are new modules placed in the correct directory? (ci/ vs tools/ vs molecule/)
|
||||
- Does the code reuse existing utilities? (no reimplemented helpers)
|
||||
- Are imports circular? (check import chains)
|
||||
- Is the code testable? (injectable dependencies, no hidden global state)
|
||||
- Does the code follow existing patterns in the codebase?
|
||||
|
||||
### 4. Reliability
|
||||
|
||||
- Are external API calls retried with backoff?
|
||||
- Are timeouts set on all network operations?
|
||||
- Are file operations atomic? (write to temp, rename)
|
||||
- Are database operations transactional where needed?
|
||||
- Are there race conditions? (check shared mutable state)
|
||||
- Are resources cleaned up in all paths? (finally blocks, context managers)
|
||||
- Can the code handle partial failures? (one service down, others up)
|
||||
- Are idempotency guarantees maintained? (safe to retry)
|
||||
|
||||
### 5. Robustness
|
||||
|
||||
- Does the code fail gracefully? (meaningful error messages, not stack traces)
|
||||
- Are unexpected inputs handled? (type checking, validation)
|
||||
- Are there any crash-on-bad-input paths?
|
||||
- Does the code degrade under load? (backpressure, queue limits)
|
||||
- Are there resource leaks? (file handles, connections, memory)
|
||||
- Does the code survive network partitions? (retry, circuit breaker)
|
||||
- Are there any unhandled exceptions that could crash the process?
|
||||
- Is logging sufficient to diagnose production issues?
|
||||
|
||||
### 6. Security
|
||||
|
||||
- Are there hardcoded secrets, tokens, or passwords?
|
||||
- Is `shell=True` used with user input? (command injection)
|
||||
- Is `eval()` or `exec()` used? (code injection)
|
||||
- Are SQL queries parameterized? (no string concatenation)
|
||||
- Are file paths validated? (no path traversal)
|
||||
- Are user inputs sanitized before display? (XSS in web contexts)
|
||||
- Are SSL/TLS verifications disabled without justification?
|
||||
- Are secrets logged in error messages or debug output?
|
||||
- Are permissions checked before privileged operations?
|
||||
- Is sensitive data in memory longer than necessary?
|
||||
|
||||
### 7. Technical Excellence
|
||||
|
||||
- Are functions under 50 lines? (refactor if longer)
|
||||
- Is cyclomatic complexity reasonable? (no deeply nested if/else chains)
|
||||
- Are names meaningful? (no single-letter vars, no misleading names)
|
||||
- Is dead code removed? (no commented-out blocks, no unused imports)
|
||||
- Are comments explaining WHY, not WHAT?
|
||||
- Is the code DRY? (no copy-pasted blocks that should be shared)
|
||||
- Is the code SOLID? (single responsibility, open/closed)
|
||||
- Are magic numbers extracted to named constants?
|
||||
- Is the code formatted per the repo's linter config?
|
||||
- Are type hints present on all function signatures?
|
||||
|
||||
### 8. Test Quality
|
||||
|
||||
- Do tests actually test the behavior? (not just that code runs)
|
||||
- Are tests independent? (no shared mutable state, no order dependency)
|
||||
- Are tests fast? (no real sleeps, no real network calls, mocked)
|
||||
- Are edge cases tested? (empty, None, boundary, error paths)
|
||||
- Are test names descriptive? (test_what_condition_expected_result)
|
||||
- Are mocks set up correctly? (mocking the right object, not too broad)
|
||||
- Is coverage 100% for new code? (every branch, every line)
|
||||
- Are integration tests added for cross-module changes?
|
||||
- Do tests clean up after themselves? (tmp_path, fixtures)
|
||||
|
||||
## Review Procedure
|
||||
|
||||
### Step 1: Gather Context
|
||||
|
||||
```
|
||||
1. Read the PR spec (if exists): docs/specs/<TASK-ID>.md
|
||||
2. Fetch PR details via Gitea MCP: pull_request_read (get_pr, list_pr_files)
|
||||
3. Read the full diff: git diff origin/master...HEAD
|
||||
4. Read the PR description and any existing review comments
|
||||
5. Identify the repo's task prefix (OBL-INFRA, GRM, SSO, DEVX)
|
||||
```
|
||||
|
||||
### Step 2: Review Each File
|
||||
|
||||
For each changed file in the PR:
|
||||
|
||||
1. Read the full file (not just the diff) to understand context
|
||||
2. Go through all 8 review categories
|
||||
3. For each issue found, note: file path, line number, category, severity, description, suggested fix
|
||||
|
||||
### Step 3: Post Inline Comments
|
||||
|
||||
For each issue found, post an inline review comment using the Gitea MCP:
|
||||
|
||||
```
|
||||
mcp_call_tool: gitea / pull_request_review_write
|
||||
method: create
|
||||
owner: <owner>
|
||||
repo: <repo>
|
||||
pull_number: <PR number>
|
||||
state: PENDING (accumulate comments before submitting)
|
||||
body: "" (empty for now, summary added on submit)
|
||||
comments: [
|
||||
{
|
||||
path: "<file path>",
|
||||
new_line_num: <line number>,
|
||||
body: "**[<category>] [<severity>]** <description>\n\n**Suggested fix:**\n```<lang>\n<fixed code>\n```"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Comment format:
|
||||
```
|
||||
**[Security] [error]** `shell=True` used with user input — command injection risk.
|
||||
|
||||
**Suggested fix:**
|
||||
```python
|
||||
subprocess.run(["git", "log", commit], check=True)
|
||||
```
|
||||
```
|
||||
|
||||
Severity levels:
|
||||
- `error` — must fix before merge (security, correctness, crash)
|
||||
- `warning` — should fix before merge (reliability, best practice)
|
||||
- `info` — consider fixing (style, minor improvement)
|
||||
|
||||
### Step 4: Auto-Fix Issues
|
||||
|
||||
For each issue that can be safely auto-fixed:
|
||||
|
||||
1. Edit the file using the `edit` tool
|
||||
2. Commit with message: `fix: address review comment — <short description>`
|
||||
3. Push to the PR's head branch: `git push origin HEAD`
|
||||
4. Wait for CI to re-run on the push
|
||||
|
||||
Auto-fix ALL issues unless:
|
||||
- The fix requires an architectural decision (ask the user)
|
||||
- The fix changes public API behavior (ask the user)
|
||||
- The fix is ambiguous (multiple valid approaches, ask the user)
|
||||
|
||||
### Step 5: Resolve Discussion Threads
|
||||
|
||||
After auto-fixing an issue and CI passes:
|
||||
|
||||
1. Find the review comment thread for that issue
|
||||
2. Post a reply: `Fixed in <commit-sha>. Closing this thread.`
|
||||
3. Resolve the discussion (if Gitea supports it via API)
|
||||
4. If resolving via API is not available, the reply comment serves as resolution
|
||||
|
||||
### Step 6: Submit Final Review
|
||||
|
||||
After all issues are addressed (fixed or discussed):
|
||||
|
||||
```
|
||||
mcp_call_tool: gitea / pull_request_review_write
|
||||
method: submit
|
||||
owner: <owner>
|
||||
repo: <repo>
|
||||
pull_number: <PR number>
|
||||
review_id: <from step 3 create>
|
||||
state: COMMENT (or APPROVED if no blocking issues remain)
|
||||
body: <summary — see below>
|
||||
```
|
||||
|
||||
### Step 7: Post Summary
|
||||
|
||||
Post a brief summary as a PR comment (via `issue_write / add_comment`):
|
||||
|
||||
```
|
||||
## Deep Review Summary
|
||||
|
||||
- **Files reviewed:** N
|
||||
- **Issues found:** N (N auto-fixed, N require attention)
|
||||
- **Categories:** security (N), correctness (N), architecture (N), ...
|
||||
|
||||
**Outcome:** ✅ Ready to merge — all issues addressed.
|
||||
**OR**
|
||||
**Outcome:** ⚠️ N blocking issue(s) remain — see inline comments.
|
||||
```
|
||||
|
||||
Keep the summary to 5-10 bullet points. Do not paste the full review.
|
||||
|
||||
### Step 8: Mark PR Ready
|
||||
|
||||
If all issues are addressed and no blocking issues remain:
|
||||
|
||||
```
|
||||
mcp_call_tool: gitea / issue_write
|
||||
method: add_labels
|
||||
owner: <owner>
|
||||
repo: <repo>
|
||||
issue_number: <PR number>
|
||||
labels: [<label_id for "ready-to-merge">]
|
||||
```
|
||||
|
||||
If blocking issues remain, do NOT add the label. Post a comment
|
||||
explaining what needs to be resolved before the PR can merge.
|
||||
|
||||
## Gitea MCP Tools Reference
|
||||
|
||||
| Action | MCP tool | Method |
|
||||
|--------|----------|--------|
|
||||
| Get PR details | `pull_request_read` | `get_pr` |
|
||||
| List PR files | `pull_request_read` | `list_pr_files` |
|
||||
| Get PR diff | `pull_request_read` | `get_pr_diff` |
|
||||
| Create review (pending) | `pull_request_review_write` | `create` (state: PENDING) |
|
||||
| Submit review | `pull_request_review_write` | `submit` (state: APPROVED/COMMENT/REQUEST_CHANGES) |
|
||||
| Post PR comment | `issue_write` | `add_comment` |
|
||||
| Add label | `issue_write` | `add_labels` |
|
||||
| List labels | `label_read` | `list_repo_labels` |
|
||||
| Merge PR | `pull_request_write` | `merge` (do NOT use — auto-merge handles this) |
|
||||
|
||||
## Important Rules
|
||||
|
||||
- **Never merge the PR yourself.** Add the `ready-to-merge` label and let
|
||||
the auto-merge workflow handle it. This ensures CI passes and the
|
||||
commit message follows the `<PREFIX>-N: <conventional>` format.
|
||||
- **Never approve your own PR.** If the agent created the PR, post
|
||||
COMMENT state, not APPROVED.
|
||||
- **Always push fixes to the PR branch**, not directly to master.
|
||||
- **Wait for CI after each push** before resolving the discussion thread.
|
||||
- **Post one review with all comments**, not multiple reviews.
|
||||
- **The summary must be brief** — 5-10 bullet points max.
|
||||
- **Severity matters**: only `error` severity blocks the `ready-to-merge` label.
|
||||
@@ -0,0 +1,130 @@
|
||||
# Spec-Driven Development
|
||||
|
||||
## Overview
|
||||
|
||||
Every change starts with a spec. No spec, no code. No code, no PR.
|
||||
|
||||
The spec is a markdown file at `docs/specs/<TASK-ID>.md` in the repo.
|
||||
It contains structured requirements (REQ-IDs) and acceptance criteria
|
||||
(AC checklist) that CI validates before merge.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Create Vikunja task** — `make create-task -- --title "Title" --description "..."`
|
||||
2. **Write spec** — Create `docs/specs/<TASK-ID>.md` (see template below)
|
||||
3. **Create branch** — `git checkout -b <PREFIX>-N-short-description`
|
||||
4. **Implement** — Write code with `# Implements: REQ-N` comments
|
||||
5. **Check ACs** — Tick all acceptance criteria checkboxes in the spec
|
||||
6. **Push and create PR** — `make push-with-pr`
|
||||
7. **CI validates** — Spec validation, PR size check, fast molecule, lint, tests
|
||||
8. **Auto-merge** — Add `ready-to-merge` label after review
|
||||
9. **Auto-deploy** — Post-merge deploys to staging (if nightly gate is green)
|
||||
|
||||
## Spec Template
|
||||
|
||||
```markdown
|
||||
# <TASK-ID>: <Title>
|
||||
|
||||
## Problem
|
||||
<What is broken or missing? Why does this change exist?>
|
||||
|
||||
## Approach
|
||||
<How will you solve it? What are the key design decisions?>
|
||||
|
||||
REQ-1: <First requirement description>
|
||||
REQ-2: <Second requirement description>
|
||||
REQ-3: <Third requirement description>
|
||||
|
||||
## Test Plan
|
||||
- <How will you verify each REQ is implemented correctly?>
|
||||
- <Include unit tests, molecule scenarios, integration tests>
|
||||
|
||||
## Deploy Plan
|
||||
- <How will this change be deployed?>
|
||||
- <What order do components need to deploy in?>
|
||||
- <Are there migrations or one-time operations?>
|
||||
|
||||
## Rollback Plan
|
||||
- <How do you revert if something goes wrong?>
|
||||
- <What data/state changes are irreversible?>
|
||||
|
||||
## Acceptance Criteria
|
||||
- [ ] REQ-1: <criterion that proves REQ-1 is done>
|
||||
- [ ] REQ-2: <criterion that proves REQ-2 is done>
|
||||
- [ ] REQ-3: <criterion that proves REQ-3 is done>
|
||||
```
|
||||
|
||||
## CI Validation
|
||||
|
||||
The `devx.ci.validate_spec` module checks:
|
||||
|
||||
1. **Spec file exists** at `docs/specs/<TASK-ID>.md` (TASK-ID from branch name)
|
||||
2. **Required sections present**: Problem, Approach, Test Plan, Deploy Plan, Rollback Plan, Acceptance Criteria
|
||||
3. **At least one REQ-ID** line (format: `REQ-N: <description>`)
|
||||
4. **All AC checkboxes checked** (`- [x]`, not `- [ ]`)
|
||||
|
||||
If any check fails, CI blocks the PR before expensive jobs run.
|
||||
|
||||
## PR Size Limits
|
||||
|
||||
CI enforces max 500 lines / 10 files changed (excluding CHANGELOG.md,
|
||||
README.md, badges, lock files). Oversized PRs are rejected. Split your
|
||||
work into smaller PRs.
|
||||
|
||||
## Code-to-Spec Linking
|
||||
|
||||
Each function, task, or template that implements a requirement should
|
||||
have a comment:
|
||||
|
||||
```python
|
||||
# Implements: REQ-1
|
||||
def install_sso_bridge():
|
||||
...
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Implements: REQ-2
|
||||
- name: Clone infra repo
|
||||
git:
|
||||
...
|
||||
```
|
||||
|
||||
## Fast Molecule (Pre-merge)
|
||||
|
||||
CI runs molecule only for **changed roles** (detected via git diff),
|
||||
with converge + verify only, single platform. This gives quick feedback
|
||||
(~5-10 min) without the full molecule suite.
|
||||
|
||||
## Full Molecule (Nightly)
|
||||
|
||||
The complete molecule suite (all scenarios, all platforms) runs nightly
|
||||
at 02:00 CET on master. If it fails:
|
||||
- A Gitea issue is created with the `feedback` label
|
||||
- The `NIGHTLY_STATUS` repo variable is set to `failed:<run_id>`
|
||||
- All staging deploys are blocked until nightly passes again
|
||||
|
||||
## Auto-Deploy on Merge
|
||||
|
||||
Every merged PR auto-deploys to staging (if nightly gate is green).
|
||||
No manual trigger needed. The deploy runs the full pipeline:
|
||||
provision → deploy-observability → deploy-customer → configure-oidc.
|
||||
|
||||
For grm/sso-bridge: post-merge publishes the package, then auto-creates
|
||||
an infra PR to bump the pinned version. That infra PR auto-deploys when
|
||||
merged.
|
||||
|
||||
## Key Commands
|
||||
|
||||
```bash
|
||||
# Validate spec locally (before pushing)
|
||||
python -m devx.ci.validate_spec --branch <PREFIX>-N-description
|
||||
|
||||
# Check PR size locally
|
||||
python -m devx.ci.check_pr_size --base origin/master --head HEAD
|
||||
|
||||
# See which roles need fast molecule
|
||||
python -m devx.ci.fast_molecule --base origin/master --head HEAD
|
||||
|
||||
# Check nightly gate status
|
||||
python -m devx.ci.nightly_gate --repo oblachno/infra --action check
|
||||
```
|
||||
@@ -35,6 +35,12 @@ produces false failures (missing dependencies, wrong Python version).
|
||||
| All platforms | `make molecule-all` | All 6 scenarios on all 4 OSes |
|
||||
| Parallel | `make molecule-all-parallel` | MOLECULE_JOBS=4 |
|
||||
|
||||
### Spec-Driven Workflow
|
||||
|
||||
Every PR requires a spec file at `docs/specs/<TASK-ID>.md`. See the
|
||||
`spec-driven-development` skill for the full workflow and template.
|
||||
CI validates the spec before running expensive jobs.
|
||||
|
||||
## Pre-Push Verification
|
||||
|
||||
**Before pushing any branch:**
|
||||
|
||||
+137
-32
@@ -110,14 +110,30 @@ jobs:
|
||||
--pr-title "$PR_TITLE" \
|
||||
--repo "$REPOSITORY" \
|
||||
--pr-number "$PR_NUMBER"
|
||||
- name: Run automated PR review
|
||||
- name: Validate spec file
|
||||
if: github.event_name == 'pull_request'
|
||||
env:
|
||||
DEVX_TASK_PREFIX: GRM
|
||||
PYTHONPATH: ${{ env.PYTHONPATH }}
|
||||
HEAD_REF: ${{ github.head_ref }}
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
set -euo pipefail
|
||||
python3 -m devx.ci.pr_review \
|
||||
"${{ github.event.number }}" \
|
||||
"${{ github.repository }}"
|
||||
python3 -m devx.ci.validate_spec \
|
||||
--branch "$HEAD_REF" \
|
||||
--github-output
|
||||
- name: Check PR size
|
||||
if: github.event_name == 'pull_request'
|
||||
env:
|
||||
PYTHONPATH: ${{ env.PYTHONPATH }}
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
python3 -m devx.ci.check_pr_size \
|
||||
--base "origin/master" \
|
||||
--head "${{ github.event.pull_request.head.sha || github.sha }}" \
|
||||
--repo "${{ github.repository }}" \
|
||||
--pr-number "${{ github.event.number }}" \
|
||||
--github-output
|
||||
# --- release-dry-run step (conditional) ---
|
||||
- name: Release dry-run validation
|
||||
if: steps.detect.outputs.user-facing-changed == 'true'
|
||||
@@ -160,10 +176,10 @@ jobs:
|
||||
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
|
||||
timeout-minutes: 15
|
||||
strategy:
|
||||
fail-fast: true
|
||||
max-parallel: 6
|
||||
fail-fast: false
|
||||
max-parallel: 4
|
||||
matrix:
|
||||
runner-index: [1, 2, 3, 4, 5, 6]
|
||||
runner-index: [1, 2, 3, 4]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up environment
|
||||
@@ -178,25 +194,34 @@ jobs:
|
||||
- name: Discover assigned test pairs
|
||||
env:
|
||||
RUNNER_INDEX: ${{ matrix.runner-index }}
|
||||
MAX_RUNNERS: 6
|
||||
MAX_RUNNERS: 4
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
python3 -m devx.molecule.distribute_molecule \
|
||||
--runner-index "$RUNNER_INDEX" \
|
||||
--max-runners "$MAX_RUNNERS" \
|
||||
--github-env
|
||||
- name: Run molecule tests
|
||||
- name: Prune stale Docker data
|
||||
id: prune
|
||||
if: env.SKIP != 'true'
|
||||
run: |
|
||||
docker system prune -af --volumes 2>/dev/null || true
|
||||
disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
|
||||
echo "Disk usage after prune: ${disk_pct}%"
|
||||
if [ "$disk_pct" -ge 85 ]; then
|
||||
echo "should-run=false" >> "$GITHUB_OUTPUT"
|
||||
echo "::warning::Disk usage at ${disk_pct}% after prune — skipping molecule tests to avoid ENOSPC failures"
|
||||
else
|
||||
echo "should-run=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
- name: Run molecule tests
|
||||
if: env.SKIP != 'true' && steps.prune.outputs.should-run != 'false'
|
||||
shell: bash
|
||||
env:
|
||||
GITEA_URL: ${{ github.server_url }}
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
|
||||
RUN_ID: ${{ github.run_id }}
|
||||
ANSIBLE_INJECT_INVOCATION: "1"
|
||||
JOB_NAME: ${{ github.job }}
|
||||
MATRIX_INDEX: ${{ matrix.runner-index }}
|
||||
GITEA_REPOSITORY: ${{ github.repository }}
|
||||
DOCKER_HOST: unix:///var/run/docker.sock
|
||||
ANSIBLE_INJECT_INVOCATION: "1"
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
if [ -z "$TEST_PAIRS" ]; then exit 0; fi
|
||||
@@ -207,19 +232,55 @@ jobs:
|
||||
_TOKEN="$CI_GITEA_API_TOKEN"; [ -z "$_TOKEN" ] && _TOKEN="$CI_GITEA_TOKEN"
|
||||
[ -z "$_TOKEN" ] && { echo "Gitea API token not set — skipping Docker login"; exit 0; }
|
||||
echo "$_TOKEN" | docker login git.oblachno.oblachno.fyi -u "$CI_GITEA_USERNAME" --password-stdin
|
||||
# shellcheck disable=SC2086 # intentional word splitting for argument expansion
|
||||
python3 -m devx.molecule.molecule_ci_guard $TEST_PAIRS
|
||||
# Run each molecule test pair sequentially.
|
||||
# Pairs are 4-part: scenario|platform_name|platform_image|platform_command
|
||||
# Spaces in platform_command are encoded as __SPACE__.
|
||||
role_dir="ansible/roles/gitea_runner"
|
||||
# shellcheck disable=SC2086 # intentional word splitting for pair list
|
||||
for pair in $TEST_PAIRS; do
|
||||
IFS='|' read -r scenario platform_name platform_image platform_command <<< "$pair"
|
||||
platform_command="${platform_command//__SPACE__/ }"
|
||||
export MOLECULE_PLATFORM_NAME="$platform_name"
|
||||
export MOLECULE_PLATFORM_IMAGE="$platform_image"
|
||||
if [ -n "$platform_command" ]; then
|
||||
export MOLECULE_PLATFORM_COMMAND="$platform_command"
|
||||
else
|
||||
unset MOLECULE_PLATFORM_COMMAND
|
||||
fi
|
||||
export ANSIBLE_ALLOW_BROKEN_CONDITIONALS=true
|
||||
echo "--- Running: $scenario on $platform_name ---"
|
||||
pushd "$role_dir" >/dev/null
|
||||
if [ "$scenario" = "default" ]; then
|
||||
molecule test || {
|
||||
echo "FAILED: $pair — running molecule destroy"
|
||||
molecule destroy 2>/dev/null || true
|
||||
popd >/dev/null
|
||||
exit 1
|
||||
}
|
||||
else
|
||||
molecule test -s "$scenario" || {
|
||||
echo "FAILED: $pair — running molecule destroy"
|
||||
molecule destroy -s "$scenario" 2>/dev/null || true
|
||||
popd >/dev/null
|
||||
exit 1
|
||||
}
|
||||
fi
|
||||
popd >/dev/null
|
||||
echo "PASSED: $pair"
|
||||
docker system prune -af --volumes 2>/dev/null || true
|
||||
done
|
||||
echo "All molecule tests passed."
|
||||
|
||||
auto-merge:
|
||||
# Auto-merge runs after validate + molecule-tests pass (or molecule is skipped).
|
||||
# Uses always() so it evaluates even when molecule-tests is skipped
|
||||
# (Gitea Actions skips dependent jobs of skipped jobs by default).
|
||||
needs: [validate, molecule-tests]
|
||||
# Auto-merge runs after validate passes. molecule-tests is NOT in needs
|
||||
# because Gitea Actions skips dependent jobs of skipped jobs without
|
||||
# evaluating if: conditions — having molecule-tests in needs would
|
||||
# cascade the skip to auto-merge when ansible-changed=false.
|
||||
needs: [validate]
|
||||
if: >-
|
||||
always() &&
|
||||
github.event_name == 'pull_request' &&
|
||||
needs.validate.result == 'success' &&
|
||||
(needs.molecule-tests.result == 'success' || needs.molecule-tests.result == 'skipped')
|
||||
needs.validate.result == 'success'
|
||||
runs-on: docker
|
||||
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
|
||||
timeout-minutes: 10
|
||||
@@ -240,16 +301,60 @@ jobs:
|
||||
env:
|
||||
REVIEWER_GITEA_API_TOKEN: ${{ secrets.REVIEWER_GITEA_API_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.number }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
python3 -m devx.ci.pr_review \
|
||||
"$PR_NUMBER" \
|
||||
"$REPOSITORY" \
|
||||
--event APPROVE \
|
||||
--checklist-confirmed \
|
||||
--checklist-categories 1,2,3,4,5,6,7,8,9,10,11,12,13 \
|
||||
--body "Auto-approved: all CI checks passed (validate, molecule-tests)."
|
||||
# Post APPROVE review via Gitea API to satisfy branch protection
|
||||
curl -s -X POST \
|
||||
"${GITHUB_SERVER_URL}/api/v1/repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/reviews" \
|
||||
-H "Authorization: token ${REVIEWER_GITEA_API_TOKEN}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"event":"APPROVED","body":"Auto-approved: all CI checks passed (validate, molecule-tests)."}' \
|
||||
|| echo "::warning::Failed to post approval review (best-effort)."
|
||||
- name: Wait for molecule tests to complete
|
||||
env:
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
# Poll commit status until all required checks pass or fail
|
||||
MAX_WAIT=600 # 10 minutes
|
||||
ELAPSED=0
|
||||
while [ $ELAPSED -lt $MAX_WAIT ]; do
|
||||
STATUS=$(curl -s -H "Authorization: token $CI_GITEA_API_TOKEN" \
|
||||
"https://git.oblachno.oblachno.fyi/api/v1/repos/${{ github.repository }}/commits/$HEAD_SHA/status" \
|
||||
| python3 -c "
|
||||
import sys,json
|
||||
d=json.load(sys.stdin)
|
||||
statuses={s['context']:s['status'] for s in d.get('statuses',[])}
|
||||
# Check if all molecule-tests contexts are terminal (success/failure/skipped)
|
||||
mol_contexts=[k for k in statuses if 'molecule-tests' in k]
|
||||
if not mol_contexts:
|
||||
print('skipped')
|
||||
elif all(statuses[k] in ('success','failure','skipped') for k in mol_contexts):
|
||||
if any(statuses[k]=='failure' for k in mol_contexts):
|
||||
print('failure')
|
||||
else:
|
||||
print('success')
|
||||
else:
|
||||
print('pending')
|
||||
")
|
||||
echo "Molecule tests status: $STATUS (elapsed: ${ELAPSED}s)"
|
||||
if [ "$STATUS" = "success" ] || [ "$STATUS" = "skipped" ]; then
|
||||
echo "All molecule tests passed (or skipped — no ansible changes)."
|
||||
break
|
||||
elif [ "$STATUS" = "failure" ]; then
|
||||
echo "ERROR: Molecule tests failed. Aborting auto-merge."
|
||||
exit 1
|
||||
fi
|
||||
sleep 30
|
||||
ELAPSED=$((ELAPSED + 30))
|
||||
done
|
||||
if [ $ELAPSED -ge $MAX_WAIT ]; then
|
||||
echo "ERROR: Timed out waiting for molecule tests."
|
||||
exit 1
|
||||
fi
|
||||
- name: Squash merge with task ID
|
||||
env:
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
|
||||
@@ -148,6 +148,26 @@ jobs:
|
||||
git fetch --tags
|
||||
git checkout "${{ steps.release-tag.outputs.tag }}"
|
||||
python3 -m devx.ci.publish "${{ steps.release-tag.outputs.tag }}" "${{ github.repository }}" --auto-login
|
||||
- name: Create infra dependency PR
|
||||
if: steps.release-tag.outputs.tag != ''
|
||||
env:
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
VIKUNJA_TOKEN: ${{ secrets.VIKUNJA_TOKEN }}
|
||||
PYTHONPATH: ${{ env.PYTHONPATH }}
|
||||
DEVX_TASK_PREFIX: GRM
|
||||
DEVX_VIKUNJA_PROJECT_ID: 6
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
# Extract version from the tag (strip leading 'v')
|
||||
TAG="${{ steps.release-tag.outputs.tag }}"
|
||||
VERSION="${TAG#v}"
|
||||
python3 -m devx.ci.create_dependency_pr \
|
||||
--repo oblachno/infra \
|
||||
--package grm \
|
||||
--new-version "$VERSION" \
|
||||
--source-repo "${{ github.repository }}" \
|
||||
--source-run-id "${{ github.run_id }}" || \
|
||||
echo "::warning::Failed to create infra dependency PR (best-effort)."
|
||||
# --- sync-wiki + vikunja (skip on automated/release commits) ---
|
||||
- name: Sync documentation to wiki
|
||||
if: needs.detect-and-configure.outputs.is-automated == 'false'
|
||||
|
||||
@@ -96,3 +96,43 @@ repos:
|
||||
types: [python]
|
||||
pass_filenames: false
|
||||
stages: [pre-push]
|
||||
|
||||
- id: check-ansible-no-log
|
||||
name: ansible no_log on secret tasks
|
||||
entry: make check-ansible-no-log
|
||||
language: system
|
||||
files: ^ansible/.*\.(yml|yaml)$
|
||||
pass_filenames: false
|
||||
stages: [pre-commit]
|
||||
|
||||
- id: check-ansible-no-state-absent-on-db
|
||||
name: no state absent on DB paths
|
||||
entry: make check-ansible-no-state-absent-on-db
|
||||
language: system
|
||||
files: ^ansible/.*\.(yml|yaml)$
|
||||
pass_filenames: false
|
||||
stages: [pre-commit]
|
||||
|
||||
- id: check-ansible-patterns
|
||||
name: ansible failure-masking patterns
|
||||
entry: make check-ansible-patterns
|
||||
language: system
|
||||
files: ^ansible/.*\.(yml|yaml)$
|
||||
pass_filenames: false
|
||||
stages: [pre-commit]
|
||||
|
||||
- id: check-jinja-expr
|
||||
name: jinja2 expression validation
|
||||
entry: make check-jinja-expr
|
||||
language: system
|
||||
files: ^ansible/.*\.(yml|yaml|j2)$
|
||||
pass_filenames: false
|
||||
stages: [pre-commit]
|
||||
|
||||
- id: check-ansible-set-fact-to-json
|
||||
name: set_fact to_json misuse check
|
||||
entry: make check-ansible-set-fact-to-json
|
||||
language: system
|
||||
files: ^ansible/.*\.(yml|yaml)$
|
||||
pass_filenames: false
|
||||
stages: [pre-commit]
|
||||
|
||||
@@ -61,8 +61,37 @@ CI also runs a best-effort `make workflow-dryrun` step (skipped if act_runner is
|
||||
- **devx package** (installed from git) — Reusable CI/CD tools: auto-merge, post-merge, release, publishing, molecule distribution, PR reviews, failure notifications
|
||||
- **Versioning** (`cliff.toml`) — git-cliff configuration for automated semver versioning from conventional commits
|
||||
|
||||
|
||||
## Spec-Driven Development
|
||||
|
||||
Every change starts with a spec. No spec, no code.
|
||||
|
||||
**Workflow:**
|
||||
1. Create Vikunja task → get `<PREFIX>-N` task ID
|
||||
2. Write spec at `docs/specs/<TASK-ID>.md` (see template in `.devin/skills/spec-driven-development/SKILL.md`)
|
||||
3. Create branch, implement with `# Implements: REQ-N` comments
|
||||
4. Tick all acceptance criteria checkboxes in spec
|
||||
5. Push and create PR — CI validates spec before expensive jobs
|
||||
|
||||
**CI gates (pre-merge):**
|
||||
- `devx.ci.validate_spec` — checks spec exists, has required sections, REQ-IDs, all ACs checked
|
||||
- `devx.ci.check_pr_size` — max 500 lines / 10 files (excludes CHANGELOG, badges, locks)
|
||||
- `devx.ci.fast_molecule` — converge+verify only for changed roles, single platform
|
||||
|
||||
**Nightly (infra only):**
|
||||
- Full molecule suite (all scenarios, all platforms) + staging deploy + integration tests
|
||||
- On failure: sets `NIGHTLY_STATUS=failed`, blocks staging deploys
|
||||
- Post-merge auto-deploy to staging checks this gate before deploying
|
||||
|
||||
**Post-merge:**
|
||||
- Infra: auto-deploys to staging (if nightly gate is green)
|
||||
- GRM/sso-bridge: auto-publishes package, auto-creates infra dependency PR to bump pinned version
|
||||
|
||||
**Skill:** `.devin/skills/spec-driven-development/SKILL.md` — full template and workflow details.
|
||||
|
||||
## PR Workflow (Mandatory)
|
||||
|
||||
|
||||
Every change to master goes through this workflow. No exceptions.
|
||||
|
||||
### Branch Protection (Required Gitea Settings)
|
||||
@@ -118,67 +147,40 @@ docs: update README
|
||||
|
||||
### 6. Review the PR (Mandatory — Before Adding ready-to-merge Label)
|
||||
|
||||
**Review checklist:** Every PR is reviewed against 13 categories covering
|
||||
architecture, code quality, security, i18n, testing, performance,
|
||||
UX, documentation, workflow compliance, maintainability, resource
|
||||
management, backwards compatibility, and logging.
|
||||
**Review checklist:** Every PR is reviewed against 8 categories covering
|
||||
functional correctness, completeness, architecture, reliability,
|
||||
robustness, security, technical excellence, and test quality.
|
||||
|
||||
**Automated review (CI `validate` job):** Every PR triggers an automated
|
||||
review via `python -m devx.ci.pr_review` as a step in the `validate` job.
|
||||
This posts a review with
|
||||
`COMMENT` (no issues) or `REQUEST_CHANGES` (issues found) based on
|
||||
the **[auto]** items in the checklist:
|
||||
**Deep review (agent-invoked `pr-review` skill):** The agent invokes
|
||||
the `pr-review` skill to perform a deep, critical review of the PR.
|
||||
The skill posts inline comments for each issue found via the Gitea MCP,
|
||||
auto-fixes them, pushes fixes to the PR branch, resolves discussion
|
||||
threads, and posts a brief summary. When no blocking issues remain,
|
||||
the PR is marked `ready-to-merge`.
|
||||
|
||||
- Architecture compliance (no subprocess in CLI, no hardcoded URLs)
|
||||
- Best practices (no `print()`, no bare `except`, no `TODO`/`FIXME`,
|
||||
no functions > 50 lines)
|
||||
- Security (no hardcoded secrets, no `shell=True`, no `eval`/`exec`)
|
||||
- i18n (no raw strings in `click.echo()` without `_()` wrapper)
|
||||
- Resource management (no `open()` without `with`, no `Popen()` without cleanup)
|
||||
- Documentation (source changes must include doc updates)
|
||||
- Test coverage (source changes must include test updates)
|
||||
- Commit conventions (conventional commit format on PR commits)
|
||||
|
||||
The automated review posts inline comments on specific lines and
|
||||
includes a summary of the checklist categories. The agent **must** address all
|
||||
`REQUEST_CHANGES` issues before proceeding.
|
||||
|
||||
**Manual review (agent):** After the automated review passes, the agent
|
||||
must go through **every category** listed above and verify
|
||||
the **[manual]** items by reviewing the full diff
|
||||
(`git diff master...HEAD`).
|
||||
|
||||
Post review comments using `devx.ci.pr_review` (run as `python -m devx.ci.pr_review`):
|
||||
```bash
|
||||
CI_GITEA_TOKEN=<token> python -m devx.ci.pr_review <pr_number> <owner/repo> \
|
||||
--event REQUEST_CHANGES \
|
||||
--body "Review summary"
|
||||
```
|
||||
See `.devin/skills/pr-review/SKILL.md` for the full review procedure,
|
||||
categories, and MCP tool reference.
|
||||
|
||||
### 7. Address Review Comments
|
||||
Fix each comment one by one, commit, and push. Re-review until satisfied.
|
||||
|
||||
### 8. Approve and Merge
|
||||
Once all checklist items are verified and comments are addressed, post
|
||||
an approval review with `--checklist-confirmed` and `--checklist-categories`:
|
||||
### 8. Mark Ready to Merge
|
||||
Once all issues are addressed, add the `ready-to-merge` label:
|
||||
```bash
|
||||
CI_GITEA_TOKEN=<token> python -m devx.ci.pr_review <pr_number> <owner/repo> \
|
||||
--event APPROVE --checklist-confirmed \
|
||||
--checklist-categories 1,2,3,4,5,6,7,8,9,10,11,12,13 \
|
||||
--body "All 13 checklist categories verified. Architecture: <summary>. Security: <summary>. Tests: <summary>. Docs: <summary>."
|
||||
make devx-pr-label
|
||||
```
|
||||
The auto-merge workflow posts an APPROVE review via the Gitea API
|
||||
and squash-merges with title `GRM-N: <conventional commit message>`.
|
||||
|
||||
The `--checklist-confirmed` flag is **required** for APPROVE events —
|
||||
it attests that the reviewer has gone through every checklist category.
|
||||
The `--checklist-categories` flag is also **required** — it must list at
|
||||
least 8 of the 13 category numbers, ensuring the reviewer actually
|
||||
checked each category rather than rubber-stamping. The review body must
|
||||
be substantive (> 50 characters) — perfunctory approvals like "LGTM" are
|
||||
rejected.
|
||||
> **IMPORTANT**: Never manually merge PRs via the API. Always use the auto-merge
|
||||
> workflow by adding the `ready-to-merge` label. Manual merges bypass the
|
||||
> `GRM-N: <conventional>` format enforcement, producing incorrectly named commits.
|
||||
> The auto-merge script validates the PR title matches the Vikunja task ID
|
||||
> and conventional commit format before merging.
|
||||
|
||||
Then add the `ready-to-merge` label. The auto-merge workflow will:
|
||||
The auto-merge workflow will:
|
||||
1. **Validate** PR title format (`GRM-N: <vikunja task title>`) and match against Vikunja task title
|
||||
2. **Check** that at least one substantive APPROVE review exists (body > 20 chars or has inline comments)
|
||||
2. **Post** an APPROVE review via the Gitea API (to satisfy branch protection)
|
||||
3. Wait for all CI checks to pass (including the `validate` job)
|
||||
4. Squash-merge with title: `GRM-N: <conventional commit message>`
|
||||
5. The post-merge workflow marks the Vikunja task as done
|
||||
@@ -190,12 +192,6 @@ a new CI run. The next auto-merge attempt will merge successfully.
|
||||
No manual rebase needed. To rebase manually: `make rebase` (local) or
|
||||
`make pr-rebase` (server-side via API).
|
||||
|
||||
> **IMPORTANT**: Never manually merge PRs via the API. Always use the auto-merge
|
||||
> workflow by adding the `ready-to-merge` label. Manual merges bypass the
|
||||
> `GRM-N: <conventional>` format enforcement, producing incorrectly named commits.
|
||||
> The auto-merge script validates the PR title matches the Vikunja task ID
|
||||
> and conventional commit format before merging.
|
||||
|
||||
### CI Path Filtering
|
||||
|
||||
The CI workflow's `validate` job includes a pre-merge validation step
|
||||
@@ -278,9 +274,9 @@ via `[tool.devx.classify]` in `pyproject.toml`.
|
||||
- Any new file type not in the allowlist
|
||||
|
||||
**devx module structure** (installed from git, not in this repo):
|
||||
- `devx.ci.*` — CI/CD automation (run by workflows): release, publish, auto_merge, classify_changes, detect_release_commit, push_badges, doc_coverage, sync_wiki, distribute_molecule, molecule_ci_guard, discover_runners, notify_failure, post_merge, pr_review, validate_commit_msg
|
||||
- `devx.ci.*` — CI/CD automation (run by workflows): release, publish, auto_merge, classify_changes, detect_release_commit, push_badges, doc_coverage, sync_wiki, distribute_molecule, discover_runners, notify_failure, post_merge, validate_commit_msg
|
||||
- `devx.tools.*` — Dev tools (run locally): check_test_speed, configure_repo, install_checkmake, install_tools, setup, generate_badges, create_task, create_pr, pr_status, pr_logs, pr_label, rebase, pr_rebase
|
||||
- `devx.molecule.*` — Molecule helpers: molecule_all, platforms, discover_runners, distribute_molecule, molecule_ci_guard
|
||||
- `devx.molecule.*` — Molecule helpers: molecule_all, platforms, discover_runners, distribute_molecule
|
||||
- `devx.gitea_cli` — Tea CLI wrapper
|
||||
- `devx.i18n` — i18n translation system
|
||||
- `devx.config` — Shared configuration (DEVX_* env vars)
|
||||
@@ -334,12 +330,10 @@ The `tea` Gitea CLI tool is used for Gitea API interactions in devx. It is insta
|
||||
- `devx.tools.configure_repo` — Creates labels via `tea labels create` (falls back to `GiteaClient` if tea fails; branch protection still uses `GiteaClient` since tea only supports basic protect/unprotect)
|
||||
|
||||
**Operations still using `GiteaClient` (not supported by tea):**
|
||||
- PR reviews (`devx.ci.pr_review`) — tea v0.14.1 only supports interactive reviews
|
||||
- Wiki page management (`devx.ci.sync_wiki`)
|
||||
- Commit status checks (`devx.ci.auto_merge`)
|
||||
- Runner discovery (`devx.molecule.discover_runners`)
|
||||
- Branch protection with detailed config (`devx.tools.configure_repo`)
|
||||
- PR file/commit listing (`devx.ci.pr_review`)
|
||||
|
||||
### PYTHONPATH Configuration
|
||||
|
||||
@@ -347,8 +341,8 @@ Since devx is installed as a package (via `pip install` from git), it is importa
|
||||
|
||||
| PYTHONPATH | When to use | Example modules |
|
||||
|------------|-------------|-----------------|
|
||||
| `src` | Module imports from `grm` | `devx.ci.auto_merge`, `devx.ci.pr_review`, `devx.ci.pr_review`, `devx.ci.sync_wiki`, `devx.ci.post_merge`, `devx.ci.classify_changes`, `devx.molecule.discover_runners`, `devx.ci.doc_coverage` |
|
||||
| (none) | Module has no GRM imports | `devx.ci.detect_release_commit`, `devx.molecule.distribute_molecule`, `devx.molecule.molecule_ci_guard`, `devx.ci.push_badges`, `devx.ci.validate_commit_msg` |
|
||||
| `src` | Module imports from `grm` | `devx.ci.auto_merge`, `devx.ci.sync_wiki`, `devx.ci.post_merge`, `devx.ci.classify_changes`, `devx.molecule.discover_runners`, `devx.ci.doc_coverage` |
|
||||
| (none) | Module has no GRM imports | `devx.ci.detect_release_commit`, `devx.molecule.distribute_molecule`, `devx.ci.push_badges`, `devx.ci.validate_commit_msg` |
|
||||
|
||||
**In workflows**, always use `env:` blocks (not inline `PYTHONPATH=value`):
|
||||
```yaml
|
||||
|
||||
@@ -2,12 +2,43 @@
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [0.22.0] - 2026-08-25
|
||||
|
||||
### Features
|
||||
|
||||
- Adopt spec-driven CI gates, create_dependency_pr, and pr-review skill
|
||||
|
||||
## [0.21.1] - 2026-08-24
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Use runuser for systemctl --user tasks in gitea_runner role
|
||||
|
||||
## [0.21.0] - 2026-08-09
|
||||
|
||||
### Features
|
||||
|
||||
- *(healthcheck)* Add two-tier disk prune with critical threshold
|
||||
|
||||
## [0.20.0] - 2026-08-09
|
||||
|
||||
### Features
|
||||
|
||||
- *(healthcheck)* Add two-tier disk prune with critical threshold
|
||||
|
||||
## [0.19.0] - 2026-08-08
|
||||
|
||||
### Features
|
||||
|
||||
- Use Gitea mirror for Ansible collection installs
|
||||
|
||||
## [0.18.8] - 2026-08-06
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Pin containerd.io to compatible version for Docker 28.x
|
||||
|
||||
|
||||
## [0.18.7] - 2026-08-06
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
@@ -175,11 +175,36 @@ makefile-lint:
|
||||
echo "checkmake not found, skipping Makefile lint"; \
|
||||
fi
|
||||
|
||||
lint-all: lint ansible-lint makefile-lint workflow-lint check-api-identity-checks
|
||||
lint-all: lint ansible-lint makefile-lint workflow-lint check-api-identity-checks check-ansible-no-log check-ansible-no-state-absent-on-db check-ansible-patterns check-jinja-expr check-ansible-set-fact-to-json
|
||||
|
||||
check-api-identity-checks:
|
||||
@$(BIN)/python -m devx.tools.check_api_identity_checks
|
||||
|
||||
check-ansible-no-log:
|
||||
@echo "[check-ansible-no-log] Checking Ansible tasks for missing no_log on secret-handling tasks..."
|
||||
@$(BIN)/python -m devx.tools.check_ansible_no_log
|
||||
@echo "[check-ansible-no-log] Passed."
|
||||
|
||||
check-ansible-no-state-absent-on-db:
|
||||
@echo "[check-ansible-no-state-absent-on-db] Checking for state: absent on DB data directories..."
|
||||
@$(BIN)/python -m devx.tools.check_ansible_no_state_absent_on_db
|
||||
@echo "[check-ansible-no-state-absent-on-db] Passed."
|
||||
|
||||
check-ansible-patterns:
|
||||
@echo "[check-ansible-patterns] Checking for dangerous failure-masking patterns..."
|
||||
@$(BIN)/python -m devx.tools.check_ansible_patterns
|
||||
@echo "[check-ansible-patterns] Passed."
|
||||
|
||||
check-jinja-expr:
|
||||
@echo "[check-jinja-expr] Validating Jinja2 expressions in Ansible files..."
|
||||
@$(BIN)/python -m devx.tools.check_jinja_expr
|
||||
@echo "[check-jinja-expr] Passed."
|
||||
|
||||
check-ansible-set-fact-to-json:
|
||||
@echo "[check-ansible-set-fact-to-json] Checking set_fact tasks for to_json misuse..."
|
||||
@$(BIN)/python -m devx.tools.check_ansible_set_fact_to_json
|
||||
@echo "[check-ansible-set-fact-to-json] Passed."
|
||||
|
||||
test-integration:
|
||||
$(BIN)/pytest tests/integration/ -v --no-cov
|
||||
|
||||
|
||||
@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
|
||||
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
|
||||
## Why GRM?
|
||||
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
---
|
||||
collections:
|
||||
- name: community.general
|
||||
version: "==13.1.0"
|
||||
type: url
|
||||
source: https://git.oblachno.oblachno.fyi/api/packages/emil/generic/ansible-collections/13.1.0/community-general-13.1.0.tar.gz
|
||||
- name: ansible.posix
|
||||
version: "==2.2.1"
|
||||
type: url
|
||||
source: https://git.oblachno.oblachno.fyi/api/packages/emil/generic/ansible-collections/2.2.1/ansible-posix-2.2.1.tar.gz
|
||||
- name: community.docker
|
||||
version: "==5.2.1"
|
||||
type: url
|
||||
source: https://git.oblachno.oblachno.fyi/api/packages/emil/generic/ansible-collections/5.2.1/community-docker-5.2.1.tar.gz
|
||||
|
||||
@@ -3,6 +3,13 @@ gitea_runner_version: "2.0.1"
|
||||
gitea_runner_labels: "docker,ubuntu-latest:docker://runner-images:ubuntu-26.04"
|
||||
gitea_runner_skip_registration: false
|
||||
|
||||
# Force re-registration even if .runner file exists.
|
||||
# Use this when Gitea no longer recognizes the runner (e.g., after a Gitea
|
||||
# server restore/reinstall or when the runner record was deleted from the
|
||||
# admin UI). The existing .runner file is removed and a new registration is
|
||||
# performed. Requires registration_token.
|
||||
gitea_runner_force_reregister: false
|
||||
|
||||
# Per-runner user (rootless isolation)
|
||||
gitea_runner_user_prefix: "grm-"
|
||||
gitea_runner_base_home: "/home"
|
||||
@@ -34,9 +41,27 @@ gitea_runner_service_restart_sec: "5"
|
||||
# jobs in the window between healthcheck runs.
|
||||
gitea_runner_healthcheck_interval: "2min"
|
||||
gitea_runner_healthcheck_boot_delay: "2min"
|
||||
gitea_runner_healthcheck_disk_threshold: 75
|
||||
gitea_runner_healthcheck_disk_threshold: 70
|
||||
# When disk reaches this level, prune EVERYTHING (no until-filter) — the
|
||||
# runner is dangerously full and the gentle until=1h prune isn't enough.
|
||||
# This removes all stopped containers and unused images regardless of age.
|
||||
# At 75%+, molecule containers fail with "container is not running" because
|
||||
# overlay2 runs out of space under parallel DinD load.
|
||||
gitea_runner_healthcheck_disk_critical: 75
|
||||
gitea_runner_healthcheck_script_path: "{{ gitea_runner_config_dir }}/healthcheck.sh"
|
||||
|
||||
# Auto-recovery: when the healthcheck detects an unregistered runner, it
|
||||
# can automatically re-register if a Gitea API token is provided.
|
||||
# The token needs admin or org-level access to fetch registration tokens.
|
||||
# Stored in a file readable by the runner user (mode 0400).
|
||||
# Set to empty string to disable auto-recovery (manual re-registration required).
|
||||
gitea_runner_auto_recover_api_token: ""
|
||||
|
||||
# Cooldown file to prevent auto-recovery loops (e.g., if Gitea is down).
|
||||
# The healthcheck writes a timestamp to this file after a re-registration
|
||||
# attempt and skips further attempts for the cooldown period.
|
||||
gitea_runner_auto_recover_cooldown_sec: 300
|
||||
|
||||
# Docker daemon resilience settings (applied to daemon.json).
|
||||
# live-restore: containers survive daemon restarts — prevents stuck container
|
||||
# states when the healthcheck restarts a hung daemon.
|
||||
|
||||
@@ -47,9 +47,9 @@
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- "'Type=oneshot' in prune_service.content | b64decode"
|
||||
- "'docker system prune' in prune_service.content | b64decode"
|
||||
- "'docker volume prune' in prune_service.content | b64decode"
|
||||
- "'docker container prune' in prune_service.content | b64decode"
|
||||
- "'docker rm -f' in prune_service.content | b64decode"
|
||||
- "'GITEA-ACTIONS-TASK' in prune_service.content | b64decode"
|
||||
- "'docker system prune -af' in prune_service.content | b64decode"
|
||||
- "'docker network prune' in prune_service.content | b64decode"
|
||||
- "'docker builder prune' in prune_service.content | b64decode"
|
||||
fail_msg: "Prune service template is missing expected directives"
|
||||
@@ -105,23 +105,12 @@
|
||||
- "'timeout 10 docker info' in healthcheck_script.content | b64decode"
|
||||
- "'systemctl --user restart docker.service' in healthcheck_script.content | b64decode"
|
||||
- "'systemctl --user restart gitea-runner.service' in healthcheck_script.content | b64decode"
|
||||
- "'docker system prune' in healthcheck_script.content | b64decode"
|
||||
- "'docker container prune' in healthcheck_script.content | b64decode"
|
||||
- "'docker rm -f' in healthcheck_script.content | b64decode"
|
||||
- "'GITEA-ACTIONS-TASK' in healthcheck_script.content | b64decode"
|
||||
- "'docker system prune -af' in healthcheck_script.content | b64decode"
|
||||
- "'docker network prune' in healthcheck_script.content | b64decode"
|
||||
- "'status=removing' in healthcheck_script.content | b64decode"
|
||||
- "'status=stopping' in healthcheck_script.content | b64decode"
|
||||
- "'docker rm -f' in healthcheck_script.content | b64decode"
|
||||
- "gitea_runner_healthcheck_disk_threshold | string in healthcheck_script.content | b64decode"
|
||||
- "gitea_runner_healthcheck_disk_critical | string in healthcheck_script.content | b64decode"
|
||||
fail_msg: "Healthcheck script template is missing expected content"
|
||||
|
||||
- name: Assert healthcheck script does NOT use aggressive prune (-af)
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- "'prune -af' not in healthcheck_script.content | b64decode"
|
||||
- "'image prune -af' not in healthcheck_script.content | b64decode"
|
||||
- "'system prune -af' not in healthcheck_script.content | b64decode"
|
||||
- "'volume prune -af' not in healthcheck_script.content | b64decode"
|
||||
fail_msg: >-
|
||||
Healthcheck script uses 'prune -af' which removes ALL images
|
||||
(including tagged runner images like ci-full). Use 'prune -f'
|
||||
(dangling only) to preserve tagged images.
|
||||
|
||||
@@ -7,6 +7,22 @@
|
||||
group: "{{ gitea_runner_service_user }}"
|
||||
mode: "0755"
|
||||
|
||||
- name: Write auto-recovery API token file
|
||||
ansible.builtin.copy:
|
||||
content: "{{ gitea_runner_auto_recover_api_token }}"
|
||||
dest: "{{ gitea_runner_config_dir }}/auto-recover.token"
|
||||
owner: "{{ gitea_runner_service_user }}"
|
||||
group: "{{ gitea_runner_service_user }}"
|
||||
mode: "0400"
|
||||
no_log: true
|
||||
when: gitea_runner_auto_recover_api_token | length > 0
|
||||
|
||||
- name: Remove stale auto-recovery token file (if auto-recovery disabled)
|
||||
ansible.builtin.file:
|
||||
path: "{{ gitea_runner_config_dir }}/auto-recover.token"
|
||||
state: absent
|
||||
when: gitea_runner_auto_recover_api_token | length == 0
|
||||
|
||||
- name: Create healthcheck user service file
|
||||
ansible.builtin.template:
|
||||
src: runner-healthcheck.service.j2
|
||||
@@ -24,24 +40,22 @@
|
||||
mode: "0644"
|
||||
|
||||
- name: Reload systemd user daemon for healthcheck timer
|
||||
ansible.builtin.command: systemctl --user daemon-reload
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user daemon-reload
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_systemd_available.stat.exists
|
||||
- gitea_runner_docker_rootless_setup
|
||||
|
||||
- name: Enable and start healthcheck user timer
|
||||
ansible.builtin.command: systemctl --user enable --now runner-healthcheck.timer
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user enable --now runner-healthcheck.timer
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_systemd_available.stat.exists
|
||||
|
||||
@@ -13,9 +13,9 @@
|
||||
- name: Include validation
|
||||
ansible.builtin.include_tasks: validate.yml
|
||||
|
||||
- name: Include service setup
|
||||
ansible.builtin.include_tasks: service.yml
|
||||
|
||||
- name: Include registration
|
||||
ansible.builtin.include_tasks: register.yml
|
||||
when: not gitea_runner_skip_registration
|
||||
|
||||
- name: Include service setup
|
||||
ansible.builtin.include_tasks: service.yml
|
||||
|
||||
@@ -18,12 +18,11 @@
|
||||
register: gitea_runner_prune_timer
|
||||
|
||||
- name: Reload systemd user daemon for prune timer
|
||||
ansible.builtin.command: systemctl --user daemon-reload
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user daemon-reload
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_systemd_available.stat.exists
|
||||
@@ -31,12 +30,11 @@
|
||||
- gitea_runner_prune_service is changed or gitea_runner_prune_timer is changed
|
||||
|
||||
- name: Enable and start docker-prune user timer
|
||||
ansible.builtin.command: systemctl --user enable --now docker-prune.timer
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user enable --now docker-prune.timer
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_systemd_available.stat.exists
|
||||
|
||||
@@ -7,11 +7,20 @@
|
||||
group: "{{ gitea_runner_service_user }}"
|
||||
mode: "0755"
|
||||
|
||||
- name: Check if runner is already registered
|
||||
- name: Check if runner registration file exists
|
||||
ansible.builtin.stat:
|
||||
path: "{{ gitea_runner_data_dir }}/.runner"
|
||||
register: gitea_runner_registered
|
||||
|
||||
- name: Remove stale runner registration file
|
||||
ansible.builtin.file:
|
||||
path: "{{ gitea_runner_data_dir }}/.runner"
|
||||
state: absent
|
||||
when:
|
||||
- gitea_runner_registered.stat.exists
|
||||
- gitea_runner_force_reregister | bool
|
||||
register: gitea_runner_registration_removed
|
||||
|
||||
- name: Register runner with Gitea
|
||||
ansible.builtin.command: >
|
||||
{{ gitea_runner_binary_path }} register
|
||||
@@ -28,7 +37,26 @@
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid | default(0) }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
|
||||
DOCKER_HOST: "unix:///run/user/{{ gitea_runner_uid | default(0) }}/docker.sock"
|
||||
when: not gitea_runner_registered.stat.exists
|
||||
when: not gitea_runner_registered.stat.exists or gitea_runner_force_reregister | bool
|
||||
register: gitea_runner_register_output
|
||||
changed_when: "'already exists' not in gitea_runner_register_output.stdout | default('')"
|
||||
changed_when: >-
|
||||
gitea_runner_register_output.rc == 0 and
|
||||
('already exists' not in gitea_runner_register_output.stdout | default(''))
|
||||
timeout: 60
|
||||
|
||||
# Note: service start is handled by service.yml (included after register.yml
|
||||
# in install_runner.yml). Starting here fails because the systemd unit file
|
||||
# has not been created yet.
|
||||
|
||||
- name: Restart runner service after (re-)registration
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user restart gitea-runner
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_systemd_available.stat.exists
|
||||
- gitea_runner_docker_rootless_setup
|
||||
- gitea_runner_register_output is defined
|
||||
- gitea_runner_register_output.rc | default(1) == 0
|
||||
|
||||
@@ -53,72 +53,6 @@
|
||||
register: gitea_runner_docker_install
|
||||
when: ansible_facts['os_family'] == 'Debian'
|
||||
|
||||
# Docker 28.x vendors containerd v2.1.x. containerd.io >= 2.3 ships a shim that
|
||||
# returns a protobuf BootstrapResult the vendored 2.1.x code cannot parse, causing
|
||||
# "failed to create TTRPC connection: unsupported protocol" on every container start.
|
||||
# Detect the mismatch and downgrade containerd.io to the latest compatible 2.2.x.
|
||||
- name: Check installed Docker CE version (Debian/Ubuntu)
|
||||
ansible.builtin.command: dpkg-query -W -f='${Version}' docker-ce
|
||||
register: gitea_runner_docker_version_check
|
||||
changed_when: false
|
||||
when: ansible_facts['os_family'] == 'Debian'
|
||||
|
||||
- name: Check installed containerd.io version (Debian/Ubuntu)
|
||||
ansible.builtin.shell: "set -o pipefail; dpkg-query -W -f='${Version}' containerd.io 2>/dev/null | cut -d: -f2 | cut -d- -f1"
|
||||
args:
|
||||
executable: /bin/bash
|
||||
register: gitea_runner_containerd_version_check
|
||||
changed_when: false
|
||||
when: ansible_facts['os_family'] == 'Debian'
|
||||
|
||||
- name: Determine if containerd.io is incompatible with installed Docker
|
||||
ansible.builtin.set_fact:
|
||||
gitea_runner_containerd_needs_downgrade: >-
|
||||
{{
|
||||
gitea_runner_containerd_version_check.stdout.split('.')[0] | int > gitea_runner_containerd_max_compatible_major
|
||||
or (
|
||||
gitea_runner_containerd_version_check.stdout.split('.')[0] | int == gitea_runner_containerd_max_compatible_major
|
||||
and gitea_runner_containerd_version_check.stdout.split('.')[1] | int > gitea_runner_containerd_max_compatible_minor
|
||||
)
|
||||
}}
|
||||
gitea_runner_docker_major: "{{ gitea_runner_docker_version_check.stdout.split('.')[0] | default('0') | int }}"
|
||||
when: ansible_facts['os_family'] == 'Debian'
|
||||
|
||||
- name: Find latest compatible containerd.io version (Debian/Ubuntu)
|
||||
ansible.builtin.shell: |
|
||||
set -o pipefail
|
||||
apt-cache madison containerd.io \
|
||||
| awk -F'|' '{print $2}' \
|
||||
| tr -d ' ' \
|
||||
| grep -E '^{{ gitea_runner_containerd_max_compatible_major }}\.{{ gitea_runner_containerd_max_compatible_minor }}\.' \
|
||||
| head -1
|
||||
args:
|
||||
executable: /bin/bash
|
||||
register: gitea_runner_containerd_compatible_version
|
||||
changed_when: false
|
||||
when:
|
||||
- ansible_facts['os_family'] == 'Debian'
|
||||
- gitea_runner_containerd_needs_downgrade | default(false)
|
||||
- gitea_runner_docker_major | int < 29
|
||||
|
||||
- name: Downgrade containerd.io to compatible version (Debian/Ubuntu)
|
||||
ansible.builtin.apt:
|
||||
name: "containerd.io={{ gitea_runner_containerd_compatible_version.stdout }}"
|
||||
state: present
|
||||
allow_downgrades: true
|
||||
register: gitea_runner_containerd_downgrade
|
||||
when:
|
||||
- ansible_facts['os_family'] == 'Debian'
|
||||
- gitea_runner_containerd_needs_downgrade | default(false)
|
||||
- gitea_runner_docker_major | int < 29
|
||||
- gitea_runner_containerd_compatible_version.stdout | length > 0
|
||||
|
||||
- name: Hold containerd.io package to prevent auto-upgrade (Debian/Ubuntu)
|
||||
ansible.builtin.dpkg_selections:
|
||||
name: containerd.io
|
||||
selection: hold
|
||||
when: ansible_facts['os_family'] == 'Debian'
|
||||
|
||||
- name: Update pacman cache (Arch Linux)
|
||||
community.general.pacman:
|
||||
update_cache: true
|
||||
@@ -209,6 +143,10 @@
|
||||
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
|
||||
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
|
||||
{% endif %}
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=10
|
||||
mode: "0644"
|
||||
owner: "{{ gitea_runner_service_user }}"
|
||||
group: "{{ gitea_runner_service_user }}"
|
||||
@@ -278,22 +216,20 @@
|
||||
- not gitea_runner_rootless_docker_check.stat.exists
|
||||
|
||||
- name: Start rootless Docker daemon (systemd user service)
|
||||
ansible.builtin.command: systemctl --user start docker
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user start docker
|
||||
changed_when: true
|
||||
when: gitea_runner_docker_rootless_setup
|
||||
|
||||
- name: Enable rootless Docker daemon (systemd user service)
|
||||
ansible.builtin.command: systemctl --user enable docker
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user enable docker
|
||||
changed_when: true
|
||||
when: gitea_runner_docker_rootless_setup
|
||||
|
||||
@@ -316,6 +252,10 @@
|
||||
{% if gitea_runner_docker_rootless_net_driver == 'pasta' %}
|
||||
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
|
||||
{% endif %}
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=10
|
||||
mode: "0644"
|
||||
owner: "{{ gitea_runner_service_user }}"
|
||||
group: "{{ gitea_runner_service_user }}"
|
||||
@@ -323,12 +263,11 @@
|
||||
when: gitea_runner_docker_rootless_setup
|
||||
|
||||
- name: Reload systemd user daemon if network config changed
|
||||
ansible.builtin.command: systemctl --user daemon-reload
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user daemon-reload
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_docker_rootless_setup
|
||||
@@ -366,12 +305,11 @@
|
||||
when: gitea_runner_docker_rootless_setup
|
||||
|
||||
- name: Restart rootless Docker if config changed
|
||||
ansible.builtin.command: systemctl --user restart docker
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user restart docker
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_docker_rootless_setup
|
||||
|
||||
@@ -9,12 +9,11 @@
|
||||
register: gitea_runner_service_file
|
||||
|
||||
- name: Reload systemd user daemon
|
||||
ansible.builtin.command: systemctl --user daemon-reload
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user daemon-reload
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_systemd_available.stat.exists
|
||||
@@ -22,12 +21,11 @@
|
||||
- gitea_runner_service_file is changed
|
||||
|
||||
- name: Restart gitea-runner if service file changed
|
||||
ansible.builtin.command: systemctl --user restart gitea-runner
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user restart gitea-runner
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_systemd_available.stat.exists
|
||||
@@ -35,12 +33,11 @@
|
||||
- gitea_runner_service_file is changed
|
||||
|
||||
- name: Enable and start gitea-runner user service
|
||||
ansible.builtin.command: systemctl --user enable --now gitea-runner
|
||||
become: true
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
DBUS_SESSION_BUS_ADDRESS: "unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus"
|
||||
ansible.builtin.command: >
|
||||
runuser -u {{ gitea_runner_service_user }} --
|
||||
env XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{{ gitea_runner_uid | default(0) }}/bus
|
||||
systemctl --user enable --now gitea-runner
|
||||
changed_when: true
|
||||
when:
|
||||
- gitea_runner_systemd_available.stat.exists
|
||||
|
||||
@@ -5,8 +5,19 @@ Description=Docker prune for Gitea runner resources
|
||||
Type=oneshot
|
||||
Environment=DOCKER_HOST=unix:///run/user/{{ gitea_runner_uid }}/docker.sock
|
||||
Environment=XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}
|
||||
ExecStart=/usr/bin/docker system prune -f --filter "label={{ gitea_runner_prune_label }}" --filter "until={{ gitea_runner_prune_until }}"
|
||||
ExecStart=/usr/bin/docker volume prune -f --filter "label={{ gitea_runner_prune_label }}"
|
||||
ExecStart=/usr/bin/docker container prune -f
|
||||
ExecStart=/usr/bin/docker network prune -f
|
||||
# Force-remove stale containers (including running ones) left behind by failed
|
||||
# molecule tests. "docker container prune -f" only removes stopped containers,
|
||||
# so running containers from crashed/interrupted CI jobs accumulate indefinitely,
|
||||
# consuming disk and memory. We stop+rm everything first, then prune the rest.
|
||||
# Exclude CI job containers (name starts with GITEA-ACTIONS-TASK) — removing
|
||||
# them kills the active CI job and causes "RWLayer is unexpectedly nil" errors.
|
||||
# Only remove containers older than 1 hour (grep for "hour/day/week/month/year
|
||||
# ago" in RunningFor) to avoid killing molecule test containers that CI jobs
|
||||
# are actively using.
|
||||
ExecStart=/bin/sh -c 'docker ps -a --format "{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}" 2>/dev/null | grep -v "GITEA-ACTIONS-TASK" | grep -E "(hour|day|week|month|year)s? ago" | awk "{print $1}" | xargs -r docker rm -f 2>/dev/null || true'
|
||||
ExecStart=/usr/bin/docker system prune -af --filter "until={{ gitea_runner_prune_until }}" --volumes
|
||||
# Prune networks older than the prune-until threshold to avoid removing
|
||||
# networks that molecule tests are actively creating (e.g. 'traefik' network
|
||||
# created during molecule create phase before containers are attached).
|
||||
ExecStart=/usr/bin/docker network prune -f --filter "until={{ gitea_runner_prune_until }}"
|
||||
ExecStart=/usr/bin/docker builder prune -f
|
||||
|
||||
@@ -44,17 +44,222 @@ if [[ "$runner_state" != "active" ]]; then
|
||||
echo "RECOVERED: gitea-runner service restarted successfully"
|
||||
fi
|
||||
|
||||
# 2b. Detect unregistered runner state. When Gitea no longer recognizes the
|
||||
# runner (e.g., server restore, runner record deleted, Gitea restart with
|
||||
# token salt change), the runner logs "unregistered runner" every few seconds.
|
||||
# A service restart will not fix this; re-registration is required.
|
||||
#
|
||||
# Detection method: query the Gitea API to verify the runner's UUID still
|
||||
# exists. This is more reliable than parsing journal logs (which requires
|
||||
# journal access permissions that runner users may not have — see the
|
||||
# 2026-08-08 incident where journalctl --user returned "No journal files
|
||||
# were opened due to insufficient permissions" for all runner users,
|
||||
# causing the healthcheck to always report "OK: runner healthy" even
|
||||
# though all runners were unregistered).
|
||||
{% if gitea_runner_auto_recover_api_token %}
|
||||
# Auto-recovery is enabled: fetch a new registration token from the Gitea API
|
||||
# and re-register the runner automatically. A cooldown prevents infinite loops.
|
||||
GITEA_API_TOKEN_FILE="{{ gitea_runner_config_dir }}/auto-recover.token"
|
||||
COOLDOWN_FILE="{{ gitea_runner_data_dir }}/auto-recover.cooldown"
|
||||
COOLDOWN_SEC={{ gitea_runner_auto_recover_cooldown_sec }}
|
||||
GITEA_URL="{{ gitea_url }}"
|
||||
RUNNER_NAME="{{ gitea_runner_name }}"
|
||||
RUNNER_LABELS="{{ gitea_runner_labels }}"
|
||||
BINARY="{{ gitea_runner_binary_path }}"
|
||||
RUNNER_FILE="{{ gitea_runner_data_dir }}/.runner"
|
||||
{% endif %}
|
||||
runner_unregistered=0
|
||||
|
||||
# Primary detection: query the Gitea API to check if the runner's ID
|
||||
# still exists in Gitea's runner list. This works regardless of journal
|
||||
# permissions.
|
||||
{% if gitea_runner_auto_recover_api_token %}
|
||||
if [[ -f "$GITEA_API_TOKEN_FILE" && -f "$RUNNER_FILE" ]]; then
|
||||
API_TOKEN=$(cat "$GITEA_API_TOKEN_FILE" 2>/dev/null || true)
|
||||
RUNNER_ID=$(python3 -c "import json; print(json.load(open('$RUNNER_FILE')).get('id',''))" 2>/dev/null || true)
|
||||
if [[ -n "$API_TOKEN" && -n "$RUNNER_ID" ]]; then
|
||||
# List all runners and check if our ID is present
|
||||
runner_found=$(curl -sf --connect-timeout 5 --max-time 10 \
|
||||
-H "Authorization: token $API_TOKEN" \
|
||||
"${GITEA_URL}/api/v1/admin/actions/runners" 2>/dev/null \
|
||||
| python3 -c "
|
||||
import sys, json
|
||||
try:
|
||||
data = json.load(sys.stdin)
|
||||
runners = data if isinstance(data, list) else data.get('runners', [])
|
||||
ids = [str(r.get('id', '')) for r in runners]
|
||||
print('1' if '$RUNNER_ID' in ids else '0')
|
||||
except Exception:
|
||||
print('0')
|
||||
" 2>/dev/null || echo "0")
|
||||
if [[ "$runner_found" != "1" ]]; then
|
||||
runner_unregistered=1
|
||||
echo "CRITICAL: runner ID $RUNNER_ID not found in Gitea (unregistered)."
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
{% endif %}
|
||||
|
||||
# Fallback detection: check journal logs (if accessible)
|
||||
if [[ "$runner_unregistered" -eq 0 ]]; then
|
||||
recent_errors=$(journalctl --user -u gitea-runner.service --since "5 minutes ago" --no-pager -q 2>/dev/null | grep -c "unregistered runner" || true)
|
||||
if [[ "$recent_errors" -ge 3 ]]; then
|
||||
runner_unregistered=1
|
||||
echo "CRITICAL: runner is unregistered in Gitea (re-login failed $recent_errors times in 5 minutes)."
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$runner_unregistered" -ge 1 ]]; then
|
||||
{% if gitea_runner_auto_recover_api_token %}
|
||||
# Check cooldown — skip if we recently attempted recovery
|
||||
if [[ -f "$COOLDOWN_FILE" ]]; then
|
||||
last_attempt=$(cat "$COOLDOWN_FILE" 2>/dev/null || echo 0)
|
||||
now=$(date +%s)
|
||||
elapsed=$((now - last_attempt))
|
||||
if [[ "$elapsed" -lt "$COOLDOWN_SEC" ]]; then
|
||||
echo "SKIP: auto-recovery cooldown active (${elapsed}s < ${COOLDOWN_SEC}s). Will retry later."
|
||||
exit 3
|
||||
fi
|
||||
fi
|
||||
|
||||
# Mark attempt time BEFORE trying (so failures also get cooldown)
|
||||
date +%s > "$COOLDOWN_FILE" 2>/dev/null || true
|
||||
|
||||
# Read the API token
|
||||
if [[ ! -f "$GITEA_API_TOKEN_FILE" ]]; then
|
||||
echo "ERROR: auto-recover token file not found at $GITEA_API_TOKEN_FILE. Manual re-registration required."
|
||||
exit 3
|
||||
fi
|
||||
API_TOKEN=$(cat "$GITEA_API_TOKEN_FILE" 2>/dev/null || true)
|
||||
if [[ -z "$API_TOKEN" ]]; then
|
||||
echo "ERROR: auto-recover token file is empty. Manual re-registration required."
|
||||
exit 3
|
||||
fi
|
||||
|
||||
echo "ATTEMPT: auto-recovering by fetching new registration token and re-registering..."
|
||||
|
||||
# Fetch a new registration token from the Gitea API
|
||||
# Try org-level first (for org-scoped runners), then instance-level
|
||||
REG_TOKEN=""
|
||||
for endpoint in \
|
||||
"api/v1/orgs/{{ gitea_runner_org | default('oblachno') }}/actions/runners/registration-token" \
|
||||
"api/v1/admin/actions/runners/registration-token"; do
|
||||
REG_TOKEN=$(curl -sf --connect-timeout 5 --max-time 10 -X POST \
|
||||
-H "Authorization: token $API_TOKEN" \
|
||||
"${GITEA_URL}/${endpoint}" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('token',''))" 2>/dev/null || true)
|
||||
if [[ -n "$REG_TOKEN" ]]; then
|
||||
echo "INFO: fetched registration token from ${endpoint}"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -z "$REG_TOKEN" ]]; then
|
||||
echo "ERROR: failed to fetch registration token from Gitea API. Is Gitea reachable?"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Stop the runner service
|
||||
systemctl --user stop gitea-runner.service 2>/dev/null || true
|
||||
sleep 1
|
||||
|
||||
# Remove the stale .runner file
|
||||
rm -f "{{ gitea_runner_data_dir }}/.runner" 2>/dev/null || true
|
||||
|
||||
# Re-register
|
||||
cd "{{ gitea_runner_data_dir }}"
|
||||
if "$BINARY" register \
|
||||
--token "$REG_TOKEN" \
|
||||
--name "$RUNNER_NAME" \
|
||||
--instance "$GITEA_URL" \
|
||||
--labels "$RUNNER_LABELS" \
|
||||
--no-interactive 2>&1; then
|
||||
echo "RECOVERED: runner re-registered successfully"
|
||||
else
|
||||
echo "ERROR: re-registration failed. Manual intervention required."
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Start the runner service
|
||||
systemctl --user start gitea-runner.service
|
||||
sleep 3
|
||||
|
||||
# Verify recovery — query the Gitea API to confirm the new ID is registered
|
||||
NEW_ID=$(python3 -c "import json; print(json.load(open('$RUNNER_FILE')).get('id',''))" 2>/dev/null || true)
|
||||
if [[ -n "$NEW_ID" ]]; then
|
||||
new_found=$(curl -sf --connect-timeout 5 --max-time 10 \
|
||||
-H "Authorization: token $API_TOKEN" \
|
||||
"${GITEA_URL}/api/v1/admin/actions/runners" 2>/dev/null \
|
||||
| python3 -c "
|
||||
import sys, json
|
||||
try:
|
||||
data = json.load(sys.stdin)
|
||||
runners = data if isinstance(data, list) else data.get('runners', [])
|
||||
ids = [str(r.get('id', '')) for r in runners]
|
||||
print('1' if '$NEW_ID' in ids else '0')
|
||||
except Exception:
|
||||
print('0')
|
||||
" 2>/dev/null || echo "0")
|
||||
if [[ "$new_found" == "1" ]]; then
|
||||
echo "OK: runner recovered and registered with new ID $NEW_ID"
|
||||
# Clear cooldown on success
|
||||
rm -f "$COOLDOWN_FILE" 2>/dev/null || true
|
||||
else
|
||||
echo "WARN: runner re-registered but ID not found in Gitea API. Will retry after cooldown."
|
||||
exit 3
|
||||
fi
|
||||
else
|
||||
echo "WARN: could not read new .runner file after re-registration. Will retry after cooldown."
|
||||
exit 3
|
||||
fi
|
||||
{% else %}
|
||||
echo "Manual re-registration required: re-run gitea_runner role with gitea_runner_force_reregister=true."
|
||||
# Restart the service once in case it is a transient token refresh issue,
|
||||
# but this cannot recover an unregistered runner without re-registration.
|
||||
systemctl --user restart gitea-runner.service
|
||||
sleep 2
|
||||
exit 3
|
||||
{% endif %}
|
||||
fi
|
||||
|
||||
# 3. Check disk space — prune aggressively if below threshold
|
||||
disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
|
||||
if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
|
||||
echo "WARN: Disk usage at ${disk_pct}%, pruning all runner resources"
|
||||
docker system prune -f --filter "label={{ gitea_runner_prune_label }}" --filter "until=1h" || true
|
||||
docker volume prune -f --filter "label={{ gitea_runner_prune_label }}" || true
|
||||
# Only prune dangling (untagged) images — keep tagged runner images (ci-full, ci-quality)
|
||||
docker image prune -f || true
|
||||
# Clean up stopped containers and dangling networks that accumulate from failed jobs
|
||||
docker container prune -f || true
|
||||
if [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_critical }} ]]; then
|
||||
echo "CRITICAL: Disk usage at ${disk_pct}% (>= {{ gitea_runner_healthcheck_disk_critical }}%), full prune"
|
||||
# Critical level: remove ALL stopped containers (no age filter) and ALL
|
||||
# unused images/volumes. The until=1h gentle prune is insufficient here.
|
||||
# Stop+rm stale non-CI containers regardless of age (failed molecule tests
|
||||
# from the last 59 minutes also consume disk).
|
||||
docker ps -a --format '{% raw %}{{.ID}} {{.Names}}{% endraw %}' 2>/dev/null \
|
||||
| grep -v 'GITEA-ACTIONS-TASK' \
|
||||
| awk '{print $1}' \
|
||||
| xargs -r docker rm -f 2>/dev/null || true
|
||||
docker system prune -af --volumes || true
|
||||
docker network prune -f || true
|
||||
docker builder prune -af || true
|
||||
disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
|
||||
echo "INFO: Disk usage after full prune: ${disk_pct}%"
|
||||
elif [[ "$disk_pct" -ge {{ gitea_runner_healthcheck_disk_threshold }} ]]; then
|
||||
echo "WARN: Disk usage at ${disk_pct}%, pruning runner resources (until=1h)"
|
||||
# Force-remove stale containers (including running ones from failed molecule tests)
|
||||
# that are older than 1 hour. "docker container prune -f" only removes stopped
|
||||
# containers, so running containers from crashed CI jobs accumulate and consume
|
||||
# disk/memory. Exclude CI job containers (name starts with GITEA-ACTIONS-TASK).
|
||||
# Only remove containers older than 1 hour to avoid killing molecule test
|
||||
# containers that CI jobs are actively using.
|
||||
docker ps -a --format '{% raw %}{{.ID}} {{.Names}} {{.RunningFor}}{% endraw %}' 2>/dev/null \
|
||||
| grep -v 'GITEA-ACTIONS-TASK' \
|
||||
| grep -E '(hour|day|week|month|year)s? ago' \
|
||||
| awk '{print $1}' \
|
||||
| xargs -r docker rm -f 2>/dev/null || true
|
||||
# Prune images and containers older than 1h (until filter is NOT
|
||||
# supported with --volumes, so prune volumes separately without a filter).
|
||||
docker image prune -af --filter "until=1h" 2>/dev/null || true
|
||||
docker container prune -f --filter "until=1h" 2>/dev/null || true
|
||||
docker volume prune -f 2>/dev/null || true
|
||||
# Prune networks older than 1 hour to avoid removing networks that
|
||||
# molecule tests are actively creating (e.g. 'traefik' network created
|
||||
# during molecule create phase before containers are attached).
|
||||
docker network prune -f --filter "until=1h" || true
|
||||
disk_pct=$(df -P / | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
|
||||
echo "INFO: Disk usage after prune: ${disk_pct}%"
|
||||
fi
|
||||
|
||||
@@ -33,5 +33,5 @@
|
||||
become_user: "{{ gitea_runner_service_user }}"
|
||||
environment:
|
||||
XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
|
||||
when: systemd_available.stat.exists
|
||||
when: gitea_runner_systemd_available.stat.exists
|
||||
changed_when: true
|
||||
|
||||
+6
-6
@@ -8,12 +8,12 @@ Each runner runs in an isolated **rootless Docker** environment under a dedicate
|
||||
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
|
||||
## Overview
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# GRM-165: Adopt spec-driven CI gates and pr-review skill
|
||||
|
||||
## Problem
|
||||
grm uses the old `devx.ci.pr_review` CI step and lacks the new spec-driven
|
||||
CI gates (validate_spec, check_pr_size). It also needs a create_dependency_pr
|
||||
step in post-merge to auto-create infra PRs when grm publishes a new version.
|
||||
|
||||
## Approach
|
||||
Replace pr_review CI steps with validate_spec + check_pr_size + curl-based
|
||||
APPROVE. Add create_dependency_pr step to post-merge. Add the spec-driven-
|
||||
development and pr-review skills. Update AGENTS.md.
|
||||
|
||||
REQ-1: Replace pr_review CI steps with validate_spec + check_pr_size + curl-based APPROVE
|
||||
REQ-2: Add create_dependency_pr step to post-merge for auto-creating infra PR to bump grm version
|
||||
REQ-3: Add spec-driven-development and pr-review skills under `.devin/skills/`
|
||||
REQ-4: Update AGENTS.md to document spec-driven development workflow and pr-review skill
|
||||
|
||||
## Test Plan
|
||||
- Verify CI workflow YAML passes actionlint
|
||||
- Verify post-merge.yml includes create_dependency_pr step with correct DEVX_TASK_PREFIX (GRM)
|
||||
- Verify validate_spec and check_pr_size steps reference correct DEVX_TASK_PREFIX (GRM)
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master via auto-merge workflow
|
||||
- Post-merge workflow handles release + publish + dependency PR automatically
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit; CI reverts to pr_review-based workflow
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: CI workflow uses validate_spec + check_pr_size + curl APPROVE instead of pr_review
|
||||
- [x] REQ-2: post-merge.yml includes create_dependency_pr step targeting oblachno/infra with package grm
|
||||
- [x] REQ-3: `.devin/skills/spec-driven-development/SKILL.md` and `.devin/skills/pr-review/SKILL.md` exist
|
||||
- [x] REQ-4: AGENTS.md documents spec-driven development workflow and pr-review skill
|
||||
@@ -0,0 +1,21 @@
|
||||
# GRM-167: Bump devx to v0.51.0
|
||||
|
||||
## Problem
|
||||
devx v0.51.0 released with role defaults path support for create_dependency_pr. grm pins v0.50.2.
|
||||
|
||||
## Approach
|
||||
Bump devx pin in pyproject.toml.
|
||||
|
||||
REQ-1: Bump devx from v0.50.2 to v0.51.0 in pyproject.toml
|
||||
|
||||
## Test Plan
|
||||
- Verify CI passes with new devx version
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master, auto-release
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: devx pinned to v0.51.0 in pyproject.toml
|
||||
@@ -259,9 +259,9 @@ OS platform matrix (defined in `devx.molecule.platforms`), then splits
|
||||
the resulting test pairs evenly across the requested number of runners.
|
||||
Each pair is encoded as `scenario|platform_name|platform_image|platform_command`.
|
||||
|
||||
`devx.molecule.molecule_ci_guard` runs the actual molecule test for a
|
||||
given test pair, with CI context (Gitea URL, token, run ID) for
|
||||
reporting results back to the commit status API.
|
||||
The CI workflow runs each test pair sequentially via a shell loop that
|
||||
sets the appropriate `MOLECULE_PLATFORM_*` environment variables and
|
||||
invokes `molecule test` directly.
|
||||
|
||||
### Commit Message Validation
|
||||
|
||||
|
||||
@@ -91,7 +91,7 @@ The `molecule-tests` job uses `fromJSON()` to consume the dynamic matrix, and pa
|
||||
|
||||
`devx.molecule.distribute_molecule` discovers all molecule scenarios under `ansible/roles/*/molecule/` and crosses them with the supported OS platform matrix, then splits the resulting test pairs evenly across the requested number of runners. Each pair is encoded as `scenario|platform_name|platform_image|platform_command`.
|
||||
|
||||
`devx.molecule.molecule_ci_guard` runs the actual molecule test for a given test pair, with CI context (Gitea URL, token, run ID) for reporting results back to the commit status API.
|
||||
The CI workflow runs each test pair sequentially via a shell loop that sets the appropriate `MOLECULE_PLATFORM_*` environment variables and invokes `molecule test` directly.
|
||||
|
||||
### Path-based CI filtering
|
||||
|
||||
|
||||
+2
-2
@@ -36,7 +36,7 @@ ci = [
|
||||
"build==1.5.1",
|
||||
"twine==6.2.0",
|
||||
# Reusable CI/CD and dev tools (auto-merge, pr-review, pre-push checks, etc.)
|
||||
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.47.8",
|
||||
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.51.0",
|
||||
]
|
||||
# Lint and type-checking tools (validate job)
|
||||
lint = [
|
||||
@@ -56,7 +56,7 @@ molecule = [
|
||||
dev = [
|
||||
"grm[ci,lint,molecule]",
|
||||
# Reusable CI/CD and dev tools (pre-push hooks, create-task, create-pr)
|
||||
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.47.8",
|
||||
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@v0.51.0",
|
||||
# Non-Python dev dependency: checkmake (Makefile linter)
|
||||
# Install via: go install github.com/checkmake/checkmake/cmd/checkmake@latest
|
||||
]
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Clean up stale runner registrations from Gitea.
|
||||
|
||||
A runner is considered stale if it hasn't been online for more than a
|
||||
configurable threshold (default: 1 hour). Stale runners accumulate when:
|
||||
- A runner host is rebuilt or re-provisioned (old registration remains)
|
||||
- A runner is re-registered (old entry remains alongside the new one)
|
||||
- A runner process dies and the healthcheck can't auto-recover
|
||||
|
||||
This script queries the Gitea API for all runners, identifies stale ones,
|
||||
and deletes them via ``DELETE /api/v1/admin/actions/runners/{id}``.
|
||||
|
||||
Usage::
|
||||
|
||||
python3 scripts/cleanup_stale_runners.py --gitea-url https://git.example.com --token <admin-token>
|
||||
python3 scripts/cleanup_stale_runners.py --gitea-url https://git.example.com --token <admin-token> --dry-run
|
||||
python3 scripts/cleanup_stale_runners.py --gitea-url https://git.example.com --token <admin-token> \\
|
||||
--stale-threshold 3600
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request # noqa: PTH123 # nosec B404
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _api_request(base_url: str, token: str, method: str, path: str) -> Any:
|
||||
url = f"{base_url.rstrip('/')}/api/v1{path}"
|
||||
req = urllib.request.Request(url, method=method) # nosec B310
|
||||
req.add_header("Authorization", f"token {token}")
|
||||
req.add_header("Accept", "application/json")
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp: # noqa: PTH123 # nosec B310
|
||||
if resp.status == 204:
|
||||
return None
|
||||
raw = resp.read()
|
||||
return json.loads(raw) if raw else None
|
||||
except urllib.error.HTTPError as e:
|
||||
detail = e.read().decode("utf-8", errors="replace")
|
||||
raise RuntimeError(f"Gitea API error {e.code}: {detail}") from e
|
||||
|
||||
|
||||
def list_runners(base_url: str, token: str) -> list[dict[str, Any]]:
|
||||
data = _api_request(base_url, token, "GET", "/admin/actions/runners")
|
||||
if data is None:
|
||||
return []
|
||||
if isinstance(data, list):
|
||||
return data
|
||||
if isinstance(data, dict):
|
||||
return data.get("runners", [])
|
||||
return []
|
||||
|
||||
|
||||
def delete_runner(base_url: str, token: str, runner_id: int) -> bool:
|
||||
try:
|
||||
_api_request(base_url, token, "DELETE", f"/admin/actions/runners/{runner_id}")
|
||||
return True
|
||||
except RuntimeError as e:
|
||||
print(f" ERROR deleting runner {runner_id}: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Clean up stale Gitea runner registrations")
|
||||
parser.add_argument("--gitea-url", required=True, help="Gitea base URL")
|
||||
parser.add_argument("--token", required=True, help="Gitea admin API token")
|
||||
parser.add_argument(
|
||||
"--stale-threshold",
|
||||
type=int,
|
||||
default=3600,
|
||||
help="Seconds since last_online before a runner is considered stale (default: 3600 = 1h)",
|
||||
)
|
||||
parser.add_argument("--dry-run", action="store_true", help="List stale runners without deleting")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
runners = list_runners(args.gitea_url, args.token)
|
||||
if not runners:
|
||||
print("No runners found.")
|
||||
return 0
|
||||
|
||||
now = int(time.time())
|
||||
stale: list[dict[str, Any]] = []
|
||||
online: list[dict[str, Any]] = []
|
||||
|
||||
for runner in runners:
|
||||
last_online = runner.get("last_online", 0) or 0
|
||||
seconds_since = now - last_online
|
||||
runner["seconds_since_online"] = seconds_since
|
||||
if seconds_since > args.stale_threshold:
|
||||
stale.append(runner)
|
||||
else:
|
||||
online.append(runner)
|
||||
|
||||
print(f"Total runners: {len(runners)}")
|
||||
print(f"Online (within {args.stale_threshold}s): {len(online)}")
|
||||
print(f"Stale (>{args.stale_threshold}s): {len(stale)}")
|
||||
print()
|
||||
|
||||
if not stale:
|
||||
print("No stale runners to clean up.")
|
||||
return 0
|
||||
|
||||
print("Stale runners:")
|
||||
for r in stale:
|
||||
rid = r.get("id", "?")
|
||||
name = r.get("name", "?")
|
||||
uuid = r.get("uuid", "?")[:8]
|
||||
secs = r.get("seconds_since_online", 0)
|
||||
hours = secs / 3600
|
||||
print(f" id={rid} name={name} uuid={uuid}... offline={hours:.1f}h ago")
|
||||
|
||||
if args.dry_run:
|
||||
print("\n--dry-run: not deleting. Remove --dry-run to clean up.")
|
||||
return 0
|
||||
|
||||
print(f"\nDeleting {len(stale)} stale runners...")
|
||||
deleted = 0
|
||||
for r in stale:
|
||||
rid = r.get("id")
|
||||
if rid is None:
|
||||
continue
|
||||
if delete_runner(args.gitea_url, args.token, rid):
|
||||
deleted += 1
|
||||
print(f" Deleted runner id={rid} ({r.get('name', '?')})")
|
||||
|
||||
print(f"\nDone: {deleted}/{len(stale)} stale runners deleted.")
|
||||
return 0 if deleted == len(stale) else 1
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,217 @@
|
||||
"""Tests for cleanup_stale_runners.py."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from scripts.cleanup_stale_runners import (
|
||||
_api_request,
|
||||
delete_runner,
|
||||
list_runners,
|
||||
main,
|
||||
)
|
||||
|
||||
|
||||
class TestListRunners:
|
||||
"""Tests for list_runners()."""
|
||||
|
||||
@patch("scripts.cleanup_stale_runners._api_request")
|
||||
def test_returns_list_of_runners(self, mock_req: MagicMock) -> None:
|
||||
mock_req.return_value = [{"id": 1, "name": "runner-1"}, {"id": 2, "name": "runner-2"}]
|
||||
result = list_runners("https://git.example.com", "token")
|
||||
assert len(result) == 2
|
||||
assert result[0]["id"] == 1
|
||||
|
||||
@patch("scripts.cleanup_stale_runners._api_request")
|
||||
def test_returns_empty_on_none(self, mock_req: MagicMock) -> None:
|
||||
mock_req.return_value = None
|
||||
result = list_runners("https://git.example.com", "token")
|
||||
assert result == []
|
||||
|
||||
@patch("scripts.cleanup_stale_runners._api_request")
|
||||
def test_extracts_runners_from_dict(self, mock_req: MagicMock) -> None:
|
||||
mock_req.return_value = {"runners": [{"id": 1}]}
|
||||
result = list_runners("https://git.example.com", "token")
|
||||
assert len(result) == 1
|
||||
assert result[0]["id"] == 1
|
||||
|
||||
@patch("scripts.cleanup_stale_runners._api_request")
|
||||
def test_returns_empty_on_non_list_non_dict(self, mock_req: MagicMock) -> None:
|
||||
mock_req.return_value = "not a list"
|
||||
result = list_runners("https://git.example.com", "token")
|
||||
assert result == []
|
||||
|
||||
|
||||
class TestDeleteRunner:
|
||||
"""Tests for delete_runner()."""
|
||||
|
||||
@patch("scripts.cleanup_stale_runners._api_request")
|
||||
def test_returns_true_on_success(self, mock_req: MagicMock) -> None:
|
||||
mock_req.return_value = None
|
||||
assert delete_runner("https://git.example.com", "token", 42) is True
|
||||
|
||||
@patch("scripts.cleanup_stale_runners._api_request")
|
||||
def test_returns_false_on_error(self, mock_req: MagicMock) -> None:
|
||||
mock_req.side_effect = RuntimeError("API error 404: not found")
|
||||
assert delete_runner("https://git.example.com", "token", 42) is False
|
||||
|
||||
|
||||
class TestApiRequest:
|
||||
"""Tests for _api_request()."""
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.urllib.request.urlopen")
|
||||
def test_returns_json_on_success(self, mock_urlopen: MagicMock) -> None:
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.status = 200
|
||||
mock_resp.read.return_value = b'{"key": "value"}'
|
||||
mock_urlopen.return_value.__enter__.return_value = mock_resp
|
||||
result = _api_request("https://git.example.com", "token", "GET", "/test")
|
||||
assert result == {"key": "value"}
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.urllib.request.urlopen")
|
||||
def test_returns_none_on_204(self, mock_urlopen: MagicMock) -> None:
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.status = 204
|
||||
mock_urlopen.return_value.__enter__.return_value = mock_resp
|
||||
result = _api_request("https://git.example.com", "token", "DELETE", "/test/1")
|
||||
assert result is None
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.urllib.request.urlopen")
|
||||
def test_returns_none_on_empty_body(self, mock_urlopen: MagicMock) -> None:
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.status = 200
|
||||
mock_resp.read.return_value = b""
|
||||
mock_urlopen.return_value.__enter__.return_value = mock_resp
|
||||
result = _api_request("https://git.example.com", "token", "GET", "/test")
|
||||
assert result is None
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.urllib.request.urlopen")
|
||||
def test_raises_on_http_error(self, mock_urlopen: MagicMock) -> None:
|
||||
import urllib.error
|
||||
|
||||
mock_error = urllib.error.HTTPError(
|
||||
"url",
|
||||
404,
|
||||
"Not Found",
|
||||
{},
|
||||
None,
|
||||
)
|
||||
mock_error.read = MagicMock(return_value=b'{"message": "not found"}')
|
||||
mock_urlopen.side_effect = mock_error
|
||||
import pytest
|
||||
|
||||
with pytest.raises(RuntimeError, match="404"):
|
||||
_api_request("https://git.example.com", "token", "GET", "/test")
|
||||
|
||||
|
||||
class TestMain:
|
||||
"""Tests for main()."""
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.list_runners")
|
||||
def test_no_runners(self, mock_list: MagicMock) -> None:
|
||||
mock_list.return_value = []
|
||||
rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
|
||||
assert rc == 0
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.list_runners")
|
||||
def test_no_stale_runners(self, mock_list: MagicMock) -> None:
|
||||
now = int(time.time())
|
||||
mock_list.return_value = [
|
||||
{"id": 1, "name": "runner-1", "last_online": now - 60},
|
||||
]
|
||||
rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
|
||||
assert rc == 0
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.list_runners")
|
||||
def test_dry_run_does_not_delete(self, mock_list: MagicMock) -> None:
|
||||
now = int(time.time())
|
||||
mock_list.return_value = [
|
||||
{"id": 1, "name": "runner-1", "last_online": now - 7200},
|
||||
]
|
||||
with patch("scripts.cleanup_stale_runners.delete_runner") as mock_del:
|
||||
rc = main(
|
||||
[
|
||||
"--gitea-url",
|
||||
"https://git.example.com",
|
||||
"--token",
|
||||
"t",
|
||||
"--dry-run",
|
||||
]
|
||||
)
|
||||
assert rc == 0
|
||||
mock_del.assert_not_called()
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.list_runners")
|
||||
@patch("scripts.cleanup_stale_runners.delete_runner")
|
||||
def test_deletes_stale_runners(self, mock_del: MagicMock, mock_list: MagicMock) -> None:
|
||||
now = int(time.time())
|
||||
mock_list.return_value = [
|
||||
{"id": 1, "name": "runner-1", "last_online": now - 60},
|
||||
{"id": 2, "name": "runner-2", "last_online": now - 7200},
|
||||
{"id": 3, "name": "runner-3", "last_online": now - 9999},
|
||||
]
|
||||
mock_del.return_value = True
|
||||
rc = main(
|
||||
[
|
||||
"--gitea-url",
|
||||
"https://git.example.com",
|
||||
"--token",
|
||||
"t",
|
||||
"--stale-threshold",
|
||||
"3600",
|
||||
]
|
||||
)
|
||||
assert rc == 0
|
||||
assert mock_del.call_count == 2
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.list_runners")
|
||||
@patch("scripts.cleanup_stale_runners.delete_runner")
|
||||
def test_returns_1_on_partial_failure(self, mock_del: MagicMock, mock_list: MagicMock) -> None:
|
||||
now = int(time.time())
|
||||
mock_list.return_value = [
|
||||
{"id": 1, "name": "runner-1", "last_online": now - 7200},
|
||||
{"id": 2, "name": "runner-2", "last_online": now - 7200},
|
||||
]
|
||||
mock_del.side_effect = [True, False]
|
||||
rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
|
||||
assert rc == 1
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.list_runners")
|
||||
def test_runner_with_zero_last_online(self, mock_list: MagicMock) -> None:
|
||||
"""Runners with last_online=0 should be considered stale."""
|
||||
mock_list.return_value = [
|
||||
{"id": 1, "name": "runner-1", "last_online": 0},
|
||||
]
|
||||
with patch("scripts.cleanup_stale_runners.delete_runner") as mock_del:
|
||||
mock_del.return_value = True
|
||||
rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
|
||||
assert rc == 0
|
||||
mock_del.assert_called_once()
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.list_runners")
|
||||
def test_runner_with_missing_last_online(self, mock_list: MagicMock) -> None:
|
||||
"""Runners with missing last_online should be considered stale."""
|
||||
mock_list.return_value = [
|
||||
{"id": 1, "name": "runner-1"},
|
||||
]
|
||||
with patch("scripts.cleanup_stale_runners.delete_runner") as mock_del:
|
||||
mock_del.return_value = True
|
||||
rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
|
||||
assert rc == 0
|
||||
mock_del.assert_called_once()
|
||||
|
||||
@patch("scripts.cleanup_stale_runners.list_runners")
|
||||
@patch("scripts.cleanup_stale_runners.delete_runner")
|
||||
def test_skips_runner_with_none_id(self, mock_del: MagicMock, mock_list: MagicMock) -> None:
|
||||
"""Runners with id=None should be skipped during deletion."""
|
||||
now = int(time.time())
|
||||
mock_list.return_value = [
|
||||
{"id": None, "name": "bad-runner", "last_online": now - 7200},
|
||||
{"id": 2, "name": "runner-2", "last_online": now - 7200},
|
||||
]
|
||||
mock_del.return_value = True
|
||||
rc = main(["--gitea-url", "https://git.example.com", "--token", "t"])
|
||||
# 1/2 deleted (None id skipped), so rc=1 (partial)
|
||||
assert rc == 1
|
||||
mock_del.assert_called_once_with("https://git.example.com", "t", 2)
|
||||
+1
-1
@@ -1,3 +1,3 @@
|
||||
"""Gitea Runner Manager — lean CLI for managing Gitea Actions runners."""
|
||||
|
||||
__version__ = "0.18.8"
|
||||
__version__ = "0.22.0"
|
||||
|
||||
+21
-1
@@ -145,11 +145,25 @@ def cli(ctx: click.Context, become_password_file: str | None, verbose: bool) ->
|
||||
"Example: docker:docker://alpine:latest"
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--force-reregister/--no-force-reregister",
|
||||
default=False,
|
||||
help=_("Force re-registration even if .runner file exists (env: GITEA_FORCE_REREGISTER)"),
|
||||
)
|
||||
@click.option(
|
||||
"--ask-become-pass/--no-ask-become-pass",
|
||||
default=True,
|
||||
help=_("Prompt for sudo password (default)"),
|
||||
)
|
||||
@click.option(
|
||||
"--auto-recover-token",
|
||||
default=None,
|
||||
help=_(
|
||||
"Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). "
|
||||
"When set, the healthcheck can automatically re-register the runner "
|
||||
"if it becomes unregistered. Requires admin or org-level access."
|
||||
),
|
||||
)
|
||||
@_handle_errors("Installation failed: {error}")
|
||||
def install(
|
||||
host: str,
|
||||
@@ -161,10 +175,14 @@ def install(
|
||||
admin_token: str | None,
|
||||
integration_retries: int,
|
||||
labels: str | None,
|
||||
force_reregister: bool,
|
||||
ask_become_pass: bool,
|
||||
auto_recover_token: str | None,
|
||||
) -> None:
|
||||
if labels is None:
|
||||
labels = os.getenv("GITEA_RUNNER_LABELS")
|
||||
if auto_recover_token is None:
|
||||
auto_recover_token = os.getenv("GITEA_AUTO_RECOVER_TOKEN")
|
||||
manager = RunnerManager()
|
||||
manager.install(
|
||||
host=host,
|
||||
@@ -176,9 +194,11 @@ def install(
|
||||
admin_token=admin_token,
|
||||
integration_retries=integration_retries,
|
||||
labels=labels,
|
||||
force_reregister=force_reregister,
|
||||
ask_become_pass=ask_become_pass,
|
||||
become_password_file=_get_become_password_file(),
|
||||
verbose=_get_verbose(),
|
||||
auto_recover_token=auto_recover_token,
|
||||
)
|
||||
|
||||
|
||||
@@ -506,7 +526,7 @@ def list_runners(ask_become_pass: bool, no_status: bool) -> None:
|
||||
click.echo(f"{_('NAME'):<18} {_('HOST'):<16} {_('USER'):<10} {_('LABELS'):<30} {_('STATUS')}")
|
||||
click.echo("-" * 90)
|
||||
for r in runners:
|
||||
click.echo(f"{r['name']:<18} {r['host']:<16} {r['user']:<10} {r['labels']:<30} {r['status']}")
|
||||
click.echo(f"{r['name']:<18} {r['host']:<16} {r['user']:<10} {(r['labels'] or ''):<30} {r['status']}")
|
||||
|
||||
|
||||
@cli.command(name="trigger-workflow", help=_("Trigger a Gitea Actions workflow via the API."))
|
||||
|
||||
@@ -87,8 +87,10 @@ class RunnerManager:
|
||||
integration_retries: int = 3,
|
||||
ask_become_pass: bool = False,
|
||||
labels: str | None = None,
|
||||
force_reregister: bool = False,
|
||||
become_password_file: str | None = None,
|
||||
verbose: bool = False,
|
||||
auto_recover_token: str | None = None,
|
||||
) -> None:
|
||||
"""Install a runner on a remote host using Ansible."""
|
||||
if not name:
|
||||
@@ -98,16 +100,19 @@ class RunnerManager:
|
||||
if not token:
|
||||
raise AnsibleError(_("GITEA_REGISTRATION_TOKEN must be set (or pass --token)"))
|
||||
|
||||
extra_vars: dict[str, str | int] = {
|
||||
extra_vars: dict[str, str | int | bool] = {
|
||||
"registration_token": token,
|
||||
"gitea_runner_name": name,
|
||||
"gitea_url": gitea_url,
|
||||
"gitea_runner_integration_retries": integration_retries,
|
||||
"gitea_runner_force_reregister": force_reregister,
|
||||
}
|
||||
if admin_token:
|
||||
extra_vars["gitea_admin_token"] = admin_token
|
||||
if labels is not None:
|
||||
extra_vars["gitea_runner_labels"] = labels
|
||||
if auto_recover_token:
|
||||
extra_vars["gitea_runner_auto_recover_api_token"] = auto_recover_token
|
||||
|
||||
with track_steps() as tracker:
|
||||
tracker.begin(_("Installing Gitea Runner on {host}", host=host))
|
||||
|
||||
@@ -367,6 +367,14 @@
|
||||
"ru": "Токен регистрации (env: GITEA_REGISTRATION_TOKEN)",
|
||||
"zh": "注册令牌(环境变量: GITEA_REGISTRATION_TOKEN)"
|
||||
},
|
||||
"Force re-registration even if .runner file exists (env: GITEA_FORCE_REREGISTER)": {
|
||||
"bg": "Принудителна повторна регистрация, дори ако .runner файлът съществува (env: GITEA_FORCE_REREGISTER)",
|
||||
"de": "Erneute Registrierung erzwingen, auch wenn .runner-Datei existiert (env: GITEA_FORCE_REREGISTER)",
|
||||
"en": "Force re-registration even if .runner file exists (env: GITEA_FORCE_REREGISTER)",
|
||||
"pl": "Wymuś ponowną rejestrację, nawet jeśli plik .runner istnieje (env: GITEA_FORCE_REREGISTER)",
|
||||
"ru": "Принудительно повторно зарегистрировать, даже если файл .runner существует (env: GITEA_FORCE_REREGISTER)",
|
||||
"zh": "即使存在 .runner 文件也强制重新注册(环境变量: GITEA_FORCE_REREGISTER)"
|
||||
},
|
||||
"Remove a registered Gitea Runner completely.": {
|
||||
"bg": "Пълно премахване на регистриран Gitea Runner.",
|
||||
"de": "Einen registrierten Gitea Runner vollständig entfernen.",
|
||||
@@ -774,5 +782,13 @@
|
||||
"pl": "Workflow uruchomiony pomyślnie. ID uruchomienia: {run_id}",
|
||||
"ru": "Workflow успешно запущен. ID запуска: {run_id}",
|
||||
"zh": "工作流触发成功。运行 ID:{run_id}"
|
||||
},
|
||||
"Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.": {
|
||||
"bg": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
|
||||
"de": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
|
||||
"en": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
|
||||
"pl": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
|
||||
"ru": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access.",
|
||||
"zh": "Gitea API token for healthcheck auto-recovery (env: GITEA_AUTO_RECOVER_TOKEN). When set, the healthcheck can automatically re-register the runner if it becomes unregistered. Requires admin or org-level access."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -50,9 +50,11 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -73,9 +75,11 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=False,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -114,9 +118,11 @@ class TestCLI:
|
||||
admin_token=None,
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -165,9 +171,11 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -188,9 +196,11 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -226,9 +236,92 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels="docker:docker://alpine:latest",
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
def test_install_force_reregister(self, mock_manager_class: MagicMock) -> None:
|
||||
mock_manager = MagicMock()
|
||||
mock_manager_class.return_value = mock_manager
|
||||
|
||||
runner = CliRunner(env=_TEST_ENV)
|
||||
result = runner.invoke(cli, ["install", "host1", "--user", "ubuntu", "--token", "tok", "--force-reregister"])
|
||||
assert result.exit_code == 0
|
||||
mock_manager.install.assert_called_once_with(
|
||||
host="host1",
|
||||
user="ubuntu",
|
||||
key=None,
|
||||
name=None,
|
||||
token="tok",
|
||||
gitea_url="https://git.example.com",
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=True,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
def test_install_with_auto_recover_token(self, mock_manager_class: MagicMock) -> None:
|
||||
"""--auto-recover-token passes the token to the manager for healthcheck auto-recovery."""
|
||||
mock_manager = MagicMock()
|
||||
mock_manager_class.return_value = mock_manager
|
||||
|
||||
runner = CliRunner(env=_TEST_ENV)
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["install", "host1", "--user", "ubuntu", "--token", "tok", "--auto-recover-token", "api-tok"],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
mock_manager.install.assert_called_once_with(
|
||||
host="host1",
|
||||
user="ubuntu",
|
||||
key=None,
|
||||
name=None,
|
||||
token="tok",
|
||||
gitea_url="https://git.example.com",
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token="api-tok",
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
def test_install_auto_recover_token_from_env(self, mock_manager_class: MagicMock) -> None:
|
||||
"""GITEA_AUTO_RECOVER_TOKEN env var is used when --auto-recover-token is not passed."""
|
||||
mock_manager = MagicMock()
|
||||
mock_manager_class.return_value = mock_manager
|
||||
|
||||
env = {**_TEST_ENV, "GITEA_AUTO_RECOVER_TOKEN": "env-tok"}
|
||||
runner = CliRunner(env=env)
|
||||
result = runner.invoke(cli, ["install", "host1", "--user", "ubuntu", "--token", "tok"])
|
||||
assert result.exit_code == 0
|
||||
mock_manager.install.assert_called_once_with(
|
||||
host="host1",
|
||||
user="ubuntu",
|
||||
key=None,
|
||||
name=None,
|
||||
token="tok",
|
||||
gitea_url="https://git.example.com",
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token="env-tok",
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -250,9 +343,11 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels="",
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -274,9 +369,11 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels="docker:docker://alpine:latest",
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -307,9 +404,11 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=pw_file,
|
||||
verbose=False,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
finally:
|
||||
import os
|
||||
@@ -334,9 +433,11 @@ class TestCLI:
|
||||
admin_token="",
|
||||
integration_retries=3,
|
||||
labels=None,
|
||||
force_reregister=False,
|
||||
ask_become_pass=True,
|
||||
become_password_file=None,
|
||||
verbose=True,
|
||||
auto_recover_token=None,
|
||||
)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
@@ -741,6 +842,27 @@ class TestCLI:
|
||||
assert "active" in result.output
|
||||
mock_manager.list_runners.assert_called_once_with(become_pass=None, no_status=False)
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
def test_list_with_none_labels(self, mock_manager_class: MagicMock) -> None:
|
||||
"""Runners with labels=None should not crash the list command."""
|
||||
mock_manager = MagicMock()
|
||||
mock_manager.list_runners.return_value = [
|
||||
{
|
||||
"name": "r1",
|
||||
"host": "10.0.0.1",
|
||||
"user": "ubuntu",
|
||||
"labels": None,
|
||||
"status": "active",
|
||||
},
|
||||
]
|
||||
mock_manager_class.return_value = mock_manager
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["list"])
|
||||
assert result.exit_code == 0
|
||||
assert "r1" in result.output
|
||||
assert "active" in result.output
|
||||
|
||||
@patch("grm.cli.RunnerManager")
|
||||
def test_list_no_status(self, mock_manager_class: MagicMock) -> None:
|
||||
"""--no-status skips SSH checks and shows registry only."""
|
||||
|
||||
@@ -51,6 +51,7 @@ class TestRunnerManager:
|
||||
assert manager._captured_extra_vars["registration_token"] == "tok"
|
||||
assert manager._captured_extra_vars["gitea_runner_name"] == "192.168.1.10"
|
||||
assert manager._captured_extra_vars["gitea_url"] == "https://git.example.com"
|
||||
assert manager._captured_extra_vars["gitea_runner_force_reregister"] is False
|
||||
assert "Installing Gitea Runner on 192.168.1.10" in mock_executor.run.call_args.kwargs["description"]
|
||||
mock_registry.add.assert_called_once_with(
|
||||
name="192.168.1.10",
|
||||
@@ -76,6 +77,7 @@ class TestRunnerManager:
|
||||
assert "/key" in cmd_str
|
||||
assert manager._captured_extra_vars["registration_token"] == "preset"
|
||||
assert manager._captured_extra_vars["gitea_runner_name"] == "my-runner"
|
||||
assert manager._captured_extra_vars["gitea_runner_force_reregister"] is False
|
||||
assert "--ask-become-pass" not in cmd_str
|
||||
mock_registry.add.assert_called_once_with(
|
||||
name="my-runner",
|
||||
@@ -97,6 +99,52 @@ class TestRunnerManager:
|
||||
cmd_str = " ".join(cmd)
|
||||
assert "--ask-become-pass" in cmd_str
|
||||
|
||||
def test_install_force_reregister(self) -> None:
|
||||
mock_registry = MagicMock()
|
||||
manager = RunnerManager(registry=mock_registry)
|
||||
mock_executor = MagicMock()
|
||||
manager._executor = mock_executor
|
||||
|
||||
manager.install(
|
||||
"host1",
|
||||
"root",
|
||||
token="tok",
|
||||
gitea_url="https://git.example.com",
|
||||
force_reregister=True,
|
||||
)
|
||||
assert manager._captured_extra_vars["gitea_runner_force_reregister"] is True
|
||||
|
||||
def test_install_with_auto_recover_token(self) -> None:
|
||||
"""auto_recover_token is passed as extra_var to Ansible."""
|
||||
mock_registry = MagicMock()
|
||||
manager = RunnerManager(registry=mock_registry)
|
||||
mock_executor = MagicMock()
|
||||
manager._executor = mock_executor
|
||||
|
||||
manager.install(
|
||||
"host1",
|
||||
"root",
|
||||
token="tok",
|
||||
gitea_url="https://git.example.com",
|
||||
auto_recover_token="api-tok",
|
||||
)
|
||||
assert manager._captured_extra_vars["gitea_runner_auto_recover_api_token"] == "api-tok"
|
||||
|
||||
def test_install_without_auto_recover_token(self) -> None:
|
||||
"""When auto_recover_token is None, the extra_var is not set."""
|
||||
mock_registry = MagicMock()
|
||||
manager = RunnerManager(registry=mock_registry)
|
||||
mock_executor = MagicMock()
|
||||
manager._executor = mock_executor
|
||||
|
||||
manager.install(
|
||||
"host1",
|
||||
"root",
|
||||
token="tok",
|
||||
gitea_url="https://git.example.com",
|
||||
)
|
||||
assert "gitea_runner_auto_recover_api_token" not in manager._captured_extra_vars
|
||||
|
||||
def test_install_missing_gitea_url(self) -> None:
|
||||
manager = RunnerManager()
|
||||
with pytest.raises(AnsibleError, match="GITEA_URL must be set"):
|
||||
|
||||
Reference in New Issue
Block a user