Compare commits

...
Author SHA1 Message Date
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> b0f912dd70 ci: add notify-failure to auto-merge and cleanup jobs, clean up redundant env
- Add notify-failure composite action to ci/auto-merge and
  build-images/cleanup jobs (previously missing — failures in these
  jobs went unnoticed)
- Remove redundant step-level CI_GITEA_API_TOKEN/CI_GITEA_USERNAME
  env vars now that workflow-level env: block provides them
- Remove redundant step-level PYTHONPATH env (already in workflow env)
- Update ADR-0003 adoption scope to reflect notify-failure coverage
- Update ci-cd-workflow.md with explicit job list for notify-failure
- Update CHANGELOG with bug fix and cleanup entries

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-12 21:43:46 +02:00
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> 4fc4c5b0d8 docs: add ADR-0003 and update docs for composite actions
CI / validate (pull_request) Successful in 2m35s
CI / auto-merge (pull_request) Successful in 26s
New ADR-0003 documenting composite action design rationale and Gitea
1.27 constraints. Updated AGENTS.md, README.md, architecture.md, and
ci-cd-workflow.md with composite action documentation. CHANGELOG entry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-12 21:27:11 +02:00
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> 64ead7b010 ci: convert workflows to use composite actions
Replace inline setup-env, notify-failure, and quality-checks steps in
ci.yml, post-merge.yml, and build-images.yml with composite action
references. Also fix pre-existing make workflow-dryrun failure by
using env context instead of vars/secrets in container.credentials
(act_runner dry-run does not populate vars/secrets contexts).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-12 21:24:22 +02:00
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> 9b7c008fdc ci: add composite actions for setup-env, notify-failure, quality-checks
Three Gitea composite actions in .gitea/actions/ that eliminate repeated
multi-step sequences across workflows. Each consumer repo gets its own
copy (no cross-repo references). See ADR-0003 for design rationale.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-12 21:21:42 +02:00
gitea-actions-bot ad98d76c1f chore: update badge URLs to commit 43d0589a [skip ci] 2026-08-12 18:17:06 +00:00
devx-ci-bot 6aa1933dbd release: v0.50.4 [skip ci] 2026-08-12 18:16:14 +00:00
emil 08f38f3635 DEVX-156: fix: add tenacity retry to install_tools._download for transient network failures
Post-merge / detect-and-configure (push) Successful in 16s
Post-merge / release-and-maintain (push) Successful in 1m21s
2026-08-12 18:15:25 +00:00
gitea-actions-bot fc5e4634dd chore: update badge URLs to commit 1b620bc1 [skip ci] 2026-08-12 17:39:15 +00:00
devx-ci-bot 615d51335d release: v0.50.3 [skip ci] 2026-08-12 17:38:37 +00:00
emil 48596627d5 DEVX-155: refactor: extract wait_for_checks, consolidate ansible_checks, deprecate ci/discover_runners
Post-merge / detect-and-configure (push) Successful in 12s
Post-merge / release-and-maintain (push) Successful in 1m7s
2026-08-12 17:37:49 +00:00
gitea-actions-bot b6f93cc2a8 chore: update badge URLs to commit 86408e58 [skip ci] 2026-08-12 13:26:04 +00:00
devx-ci-bot 77dc4cc22a release: v0.50.2 [skip ci] 2026-08-12 13:25:20 +00:00
emil 7dda7e5a44 DEVX-153: fix: run molecule destroy on test failure to clean up containers
Post-merge / detect-and-configure (push) Successful in 18s
Post-merge / release-and-maintain (push) Successful in 1m15s
2026-08-12 13:24:25 +00:00
gitea-actions-bot 7daee73ceb chore: update badge URLs to commit 5aa3dd2f [skip ci] 2026-08-12 13:17:52 +00:00
devx-ci-bot 06b11617ce release: v0.50.1 [skip ci] 2026-08-12 13:16:50 +00:00
emil e0ccfd6a11 DEVX-154: fix: configure git remote with CI token for post-merge push
Post-merge / detect-and-configure (push) Successful in 57s
Post-merge / release-and-maintain (push) Successful in 1m40s
2026-08-12 13:15:10 +00:00
devx-ci-bot 18632bd543 release: v0.50.0 [skip ci] 2026-08-12 12:44:32 +00:00
emil 1fc7a03c1a DEVX-152: feat: sync missing features from v0.49.x line to master
Post-merge / detect-and-configure (push) Successful in 22s
Post-merge / release-and-maintain (push) Failing after 37s
2026-08-12 12:43:37 +00:00
gitea-actions-bot 808e7a2e42 chore: update badge URLs to commit e463f9a5 [skip ci] 2026-08-07 21:03:05 +00:00
devx-ci-bot dd8e6c69e9 release: v0.49.5 [skip ci] 2026-08-07 21:02:12 +00:00
emil ccb7023965 DEVX-151: perf: skip dep resolution in setup-image with --no-deps
Post-merge / detect-and-configure (push) Successful in 36s
Post-merge / release-and-maintain (push) Successful in 1m21s
2026-08-07 21:01:06 +00:00
gitea-actions-bot 7dd15f1461 chore: update badge URLs to commit 8c02351c [skip ci] 2026-08-07 20:44:08 +00:00
devx-ci-bot d4e4621fa1 release: v0.49.4 [skip ci] 2026-08-07 20:43:19 +00:00
emil a6f814c446 DEVX-150: fix: add container.credentials for private registry auth
Post-merge / detect-and-configure (push) Successful in 25s
Post-merge / release-and-maintain (push) Successful in 1m16s
2026-08-07 20:42:18 +00:00
gitea-actions-bot 04aa5acb1f chore: update badge URLs to commit 40269e2e [skip ci] 2026-08-07 20:34:03 +00:00
devx-ci-bot 6973f9d851 release: v0.49.3 [skip ci] 2026-08-07 20:33:16 +00:00
emil 2669a0ea73 DEVX-149: fix: retry ansible-galaxy collection install on transient timeouts
Post-merge / detect-and-configure (push) Successful in 5m50s
Post-merge / release-and-maintain (push) Successful in 1m14s
2026-08-07 20:26:52 +00:00
gitea-actions-bot 03f057b55a chore: update badge URLs to commit b534b155 [skip ci] 2026-08-07 15:37:23 +00:00
devx-ci-bot 706d6dafe0 release: v0.49.2 [skip ci] 2026-08-07 15:36:28 +00:00
gitea-adminandemil 03ddce427c DEVX-148: fix: add fallback URL for tea download
Post-merge / detect-and-configure (push) Successful in 2m1s
Post-merge / release-and-maintain (push) Successful in 1m24s
Co-authored-by: oblachno Admin <admin@oblachno.oblachno.fyi>
2026-08-07 15:33:54 +00:00
gitea-actions-bot 9642d6884c chore: update badge URLs to commit 3f33ebe6 [skip ci] 2026-08-07 14:32:31 +00:00
devx-ci-bot b2074d6635 release: v0.49.1 [skip ci] 2026-08-07 14:31:37 +00:00
gitea-adminandemil a6dddf25e7 DEVX-147: fix: add container images to build-images workflow
Post-merge / detect-and-configure (push) Successful in 3m17s
Post-merge / release-and-maintain (push) Successful in 1m27s
Co-authored-by: oblachno Admin <admin@oblachno.oblachno.fyi>
2026-08-07 14:27:45 +00:00
gitea-actions-bot 01130a7385 chore: update badge URLs to commit 58fdb2b6 [skip ci] 2026-08-07 14:23:22 +00:00
devx-ci-bot 07580c9280 release: v0.49.0 [skip ci] 2026-08-07 14:22:26 +00:00
gitea-adminandemil 6601d90bee DEVX-146: feat: add --include-roles and --exclude-roles to distribute_molecule
Post-merge / detect-and-configure (push) Successful in 55s
Post-merge / release-and-maintain (push) Successful in 1m28s
Co-authored-by: oblachno Admin <admin@oblachno.oblachno.fyi>
2026-08-07 14:20:54 +00:00
gitea-actions-bot 0df79fed53 chore: update badge URLs to commit b55c2d2f [skip ci] 2026-07-22 20:58:00 +00:00
devx-ci-bot cf8287e683 release: v0.48.0 [skip ci] 2026-07-22 20:57:10 +00:00
emil 9f1bdc4cf1 DEVX-145: feat: extract reusable components from infra and grm into devx
Post-merge / release-and-maintain (push) Failing after 373h11m10s
Post-merge / detect-and-configure (push) Failing after 373h11m25s
2026-07-22 20:56:27 +00:00
91 changed files with 12795 additions and 4029 deletions
+47
View File
@@ -0,0 +1,47 @@
name: 'Notify on failure'
description: 'Create a Gitea issue when a CI workflow fails (calls devx.ci.notify_failure)'
# Composite action for the common "Notify on failure" step pattern.
# Replaces the repeated inline:
# - name: Notify on failure
# if: failure()
# env:
# CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
# run: |
# . .venv/bin/activate 2>/dev/null || true
# export PATH="$HOME/.local/bin:$PATH"
# python3 -m devx.ci.notify_failure \
# --repo "${{ github.repository }}" \
# --run-id "${{ github.run_id }}" \
# --workflow "ci/validate" \
# --commit "${{ github.sha }}" \
# --auto-login
#
# Gitea 1.27 notes:
# - `if: failure()` is evaluated in the calling workflow's context and
# propagates correctly to composite action steps.
# - `secrets` are not accessible here; the calling workflow's top-level
# `env:` CI_GITEA_API_TOKEN is used via `${{ env.* }}`.
inputs:
workflow:
description: 'Workflow/job name used in the Gitea issue title (e.g., ci/validate)'
required: true
runs:
using: 'composite'
steps:
- name: Notify on failure
if: failure()
shell: bash
env:
CI_GITEA_API_TOKEN: ${{ env.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "${{ inputs.workflow }}" \
--commit "${{ github.sha }}" \
--auto-login
+89
View File
@@ -0,0 +1,89 @@
name: 'Quality checks'
description: 'Run lint, unit tests with coverage, test speed, docs, translations, and security scan'
# Composite action for the 6-step quality check sequence used by the
# devx validate job. Replaces the inline block:
# - Lint all
# - Unit tests with 100% coverage
# - Check unit test speed
# - Documentation gate (coverage + stale refs + lint + version refs + prose)
# - Translation completeness check
# - Dependency security scan
#
# Each step activates the venv defensively (`. .venv/bin/activate 2>/dev/null
# || true`) so the action works whether or not the setup step created a
# venv at the repo root (pre-built CI images symlink /opt/venv to .venv).
#
# Gitea 1.27 notes:
# - Every `run` step needs explicit `shell:`.
# - Inputs are string-typed; numeric thresholds are passed through as
# strings to `devx.tools.check_test_speed`.
inputs:
package:
description: 'Package name for doc version checks (e.g., devx, grm). Empty = no DEVX_DOC_VERSIONS_PKG override.'
required: false
default: ''
test-speed-max:
description: 'Max total test seconds (passed to check_test_speed --max-seconds)'
required: false
default: '15'
test-speed-max-single:
description: 'Max single test seconds (passed to check_test_speed --max-single-seconds)'
required: false
default: '0.5'
translations-file:
description: 'Path to translations.json (empty = default location src/devx/translations.json)'
required: false
default: ''
runs:
using: 'composite'
steps:
- name: Lint all
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
make lint-all
- name: Unit tests with 100% coverage
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
make pytest-cov
- name: Check unit test speed
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
python3 -m devx.tools.check_test_speed \
--max-seconds "${{ inputs.test-speed-max }}" \
--max-single-seconds "${{ inputs.test-speed-max-single }}"
- name: Documentation gate (coverage + stale refs + lint + version refs + prose)
shell: bash
env:
DEVX_DOC_COVERAGE_STRICT: "1"
DEVX_VALE_LEVEL: warning
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
if [ -n "${{ inputs.package }}" ]; then
export DEVX_DOC_VERSIONS_PKG="${{ inputs.package }}"
fi
make devx-docs-check
- name: Translation completeness check
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
if [ -n "${{ inputs.translations-file }}" ]; then
python3 -m devx.ci.check_translations --translations "${{ inputs.translations-file }}"
else
python3 -m devx.ci.check_translations
fi
- name: Dependency security scan
shell: bash
run: |
. .venv/bin/activate 2>/dev/null || true
# Install pip in venv if missing (needed by pip-audit)
.venv/bin/python -m ensurepip 2>/dev/null || true
PIPAPI_PYTHON_LOCATION=$PWD/.venv/bin/python \
pip-audit --desc --skip-editable 2>&1 || true
+37
View File
@@ -0,0 +1,37 @@
name: 'Set up environment'
description: 'Set up CI environment with venv and PATH (calls make setup-image)'
# Composite action for the common "Set up environment" step pattern.
# Replaces the repeated inline:
# - name: Set up environment
# env:
# CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
# run: make setup-image
#
# Gitea 1.27 notes:
# - Every `run` step needs explicit `shell:`.
# - Composite actions cannot access `secrets` directly; they read from
# the `env:` context which the calling workflow must populate.
# - The calling workflow's top-level `env:` block (CI_GITEA_API_TOKEN,
# CI_GITEA_USERNAME) is visible here via `${{ env.* }}`.
inputs:
extras:
description: 'Extra pip install groups passed to make setup-image (e.g., ci,lint,release)'
required: false
default: ''
runs:
using: 'composite'
steps:
- name: Set up environment
shell: bash
env:
CI_GITEA_API_TOKEN: ${{ env.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ env.CI_GITEA_USERNAME }}
run: |
if [ -n "${{ inputs.extras }}" ]; then
make setup-image EXTRAS="${{ inputs.extras }}"
else
make setup-image
fi
+22 -30
View File
@@ -29,9 +29,20 @@ concurrency:
group: build-images
cancel-in-progress: false
env:
PIP_BREAK_SYSTEM_PACKAGES: "1"
PYTHONPATH: src
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
jobs:
build-and-push:
runs-on: docker
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
credentials:
username: ${{ env.CI_GITEA_USERNAME }}
password: ${{ env.CI_GITEA_API_TOKEN }}
timeout-minutes: 30
outputs:
is-release: ${{ steps.check.outputs.is-release }}
@@ -40,13 +51,9 @@ jobs:
with:
fetch-depth: 1
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: make setup-release
- name: Check if this is a release commit
id: check
env:
PYTHONPATH: src
run: |
. .venv/bin/activate
python3 -m devx.ci.detect_release_commit
@@ -54,9 +61,6 @@ jobs:
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success' && steps.check.outputs.is-release == 'false')
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
run: |
. .venv/bin/activate
_TOKEN="$CI_GITEA_API_TOKEN"
@@ -68,10 +72,6 @@ jobs:
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success' && steps.check.outputs.is-release == 'false')
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
PYTHONPATH: src
run: |
. .venv/bin/activate
export PATH="$HOME/.local/bin:$PATH"
@@ -96,38 +96,27 @@ jobs:
--tag latest \
--registry git.oblachno.oblachno.fyi \
--push
- name: Notify on failure
if: failure()
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
PYTHONPATH: src
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "build-images/build-and-push" \
--commit "${{ github.sha }}" \
--auto-login
- uses: ./.gitea/actions/notify-failure
with:
workflow: "build-images/build-and-push"
cleanup:
needs: [build-and-push]
if: always() && needs.build-and-push.result == 'success'
runs-on: docker
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
credentials:
username: ${{ env.CI_GITEA_USERNAME }}
password: ${{ env.CI_GITEA_API_TOKEN }}
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: make setup-ci
- name: Clean up old image versions
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
PYTHONPATH: src
run: |
. .venv/bin/activate
python3 -m devx.tools.clean_images \
@@ -136,3 +125,6 @@ jobs:
--name oblachno-oss/runner-images/ci-quality \
--name oblachno-oss/runner-images/ci-full \
--keep 2
- uses: ./.gitea/actions/notify-failure
with:
workflow: "build-images/cleanup"
+23 -58
View File
@@ -18,7 +18,11 @@ jobs:
# Saves ~4x checkout+setup overhead vs 5 separate jobs.
validate:
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
credentials:
username: ${{ env.CI_GITEA_USERNAME }}
password: ${{ env.CI_GITEA_API_TOKEN }}
timeout-minutes: 15
defaults:
run:
@@ -29,43 +33,12 @@ jobs:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: make setup-image
# --- quality steps ---
- name: Lint all
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
make lint-all
- name: Unit tests with 100% coverage
run: |
. .venv/bin/activate 2>/dev/null || true
make pytest-cov
- name: Check unit test speed
run: |
. .venv/bin/activate 2>/dev/null || true
python3 -m devx.tools.check_test_speed --max-seconds 8 --max-single-seconds 0.5
- name: Documentation gate (coverage + stale refs + lint + version refs + prose)
env:
DEVX_DOC_COVERAGE_STRICT: "1"
DEVX_VALE_LEVEL: warning
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
make devx-docs-check
- name: Translation completeness check
run: |
. .venv/bin/activate 2>/dev/null || true
python3 -m devx.ci.check_translations
- name: Dependency security scan
run: |
. .venv/bin/activate 2>/dev/null || true
# Install pip in venv if missing (needed by pip-audit)
.venv/bin/python -m ensurepip 2>/dev/null || true
PIPAPI_PYTHON_LOCATION=$PWD/.venv/bin/python \
pip-audit --desc --skip-editable 2>&1 || true
- uses: ./.gitea/actions/setup-env
- uses: ./.gitea/actions/quality-checks
with:
package: devx
test-speed-max: "15"
test-speed-max-single: "0.5"
- name: Workflow dry-run validation
run: |
. .venv/bin/activate 2>/dev/null || true
@@ -117,19 +90,9 @@ jobs:
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.release --dry-run
- name: Notify on failure
if: failure()
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "ci/validate" \
--commit "${{ github.sha }}" \
--auto-login
- uses: ./.gitea/actions/notify-failure
with:
workflow: "ci/validate"
auto-merge:
# Auto-merge runs after validate passes. It reads the task ID
@@ -140,7 +103,11 @@ jobs:
github.event_name == 'pull_request' &&
needs.validate.result == 'success'
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
credentials:
username: ${{ env.CI_GITEA_USERNAME }}
password: ${{ env.CI_GITEA_API_TOKEN }}
timeout-minutes: 10
defaults:
run:
@@ -150,14 +117,10 @@ jobs:
with:
fetch-depth: 0
token: ${{ secrets.CI_GITEA_API_TOKEN }}
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: make setup-image
- uses: ./.gitea/actions/setup-env
- name: Post approval review
env:
REVIEWER_GITEA_API_TOKEN: ${{ secrets.REVIEWER_GITEA_API_TOKEN }}
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
PR_NUMBER: ${{ github.event.number }}
REPOSITORY: ${{ github.repository }}
run: |
@@ -171,7 +134,6 @@ jobs:
--body "Auto-approved: all CI checks passed (validate job)."
- name: Squash merge with task ID
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
VIKUNJA_TOKEN: ${{ secrets.VIKUNJA_TOKEN }}
DEVX_VIKUNJA_PROJECT_ID: "8"
HEAD_REF: ${{ github.head_ref }}
@@ -185,3 +147,6 @@ jobs:
"$PR_TITLE" \
"$REPOSITORY" \
"$PR_NUMBER"
- uses: ./.gitea/actions/notify-failure
with:
workflow: "ci/auto-merge"
+21 -43
View File
@@ -35,7 +35,11 @@ env:
jobs:
detect-and-configure:
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
credentials:
username: ${{ env.CI_GITEA_USERNAME }}
password: ${{ env.CI_GITEA_API_TOKEN }}
timeout-minutes: 10
defaults:
run:
@@ -48,10 +52,7 @@ jobs:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: make setup-image
- uses: ./.gitea/actions/setup-env
- name: Ensure branch protection and labels
env:
DEVX_REPO_NAME: devx
@@ -81,25 +82,19 @@ jobs:
--base "HEAD~1" \
--head "HEAD" \
--github-output
- name: Notify on failure
if: failure()
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "post-merge/detect-and-configure" \
--commit "${{ github.sha }}" \
--auto-login
- uses: ./.gitea/actions/notify-failure
with:
workflow: "post-merge/detect-and-configure"
release-and-maintain:
needs: [detect-and-configure]
if: always() && needs.detect-and-configure.result == 'success'
runs-on: docker
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
container:
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
credentials:
username: ${{ env.CI_GITEA_USERNAME }}
password: ${{ env.CI_GITEA_API_TOKEN }}
timeout-minutes: 15
outputs:
tag: ${{ steps.release-tag.outputs.tag }}
@@ -112,28 +107,24 @@ jobs:
fetch-depth: 0
ref: master
token: ${{ secrets.CI_GITEA_API_TOKEN }}
- name: Set up environment
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: make setup-image EXTRAS=release
- uses: ./.gitea/actions/setup-env
with:
extras: "release"
- name: Configure git
run: |
git config user.name "devx-ci-bot"
git config user.email "devx-ci-bot@oblachno.fyi"
git remote set-url origin "https://devx-ci-bot:${CI_GITEA_API_TOKEN}@git.oblachno.oblachno.fyi/oblachno-oss/devx.git"
# --- release + publish (only if user-facing changes, not a release commit) ---
- name: Run release
id: release-tag
if: needs.detect-and-configure.outputs.is-release == 'false' && needs.detect-and-configure.outputs.user-facing-changed == 'true'
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.release
- name: Build and publish release
if: steps.release-tag.outputs.tag != ''
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
@@ -143,8 +134,6 @@ jobs:
# --- sync-wiki + vikunja (skip on automated/release commits) ---
- name: Sync documentation to wiki
if: needs.detect-and-configure.outputs.is-automated == 'false'
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
python3 -m devx.ci.sync_wiki --repo "${{ github.repository }}" --verify
@@ -159,7 +148,6 @@ jobs:
# --- badges (always run — even on release commits) ---
- name: Generate and push badges
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
PRE_COMMIT_ALLOW_NO_CONFIG: "1"
run: |
. .venv/bin/activate 2>/dev/null || true
@@ -168,16 +156,6 @@ jobs:
git fetch origin master
git reset --hard origin/master
python3 -m devx.ci.push_badges
- name: Notify on failure
if: failure()
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
run: |
. .venv/bin/activate 2>/dev/null || true
export PATH="$HOME/.local/bin:$PATH"
python3 -m devx.ci.notify_failure \
--repo "${{ github.repository }}" \
--run-id "${{ github.run_id }}" \
--workflow "post-merge/release-and-maintain" \
--commit "${{ github.sha }}" \
--auto-login
- uses: ./.gitea/actions/notify-failure
with:
workflow: "post-merge/release-and-maintain"
+1 -1
View File
@@ -59,7 +59,7 @@ repos:
- id: check-test-speed
name: unit test speed check
entry: .venv/bin/python -m devx.tools.check_test_speed --max-seconds 6 --max-single-seconds 0.5
entry: .venv/bin/python -m devx.tools.check_test_speed --max-seconds 15 --max-single-seconds 0.5
language: system
types: [python]
pass_filenames: false
+12
View File
@@ -0,0 +1,12 @@
extends: existence
message: "Don't attribute human qualities to software or hardware ('%s')."
link: https://developers.google.com/style/anthropomorphism
level: suggestion
ignorecase: true
# Limited to the two verbs the guide itself names. Broader lists (wants, knows,
# thinks) can't tell a software subject from a human one: on a 950-file corpus
# they produced 8 false positives ('the customer wants', 'your audience knows')
# for every 2 real ones.
tokens:
- sees
- tells
+7 -2
View File
@@ -1,8 +1,13 @@
extends: existence
message: "'%s' should be in lowercase."
link: 'https://developers.google.com/style/colons'
nonword: true
level: warning
scope: sentence
# The match is the word itself, not ': X', and `nonword` is off. Both are
# required for a project Vocab to work: Vale compares accept.txt entries
# against the matched text, and `nonword: true` opts out of that entirely.
# So a proper noun after a colon can be exempted by adding it to accept.txt.
# The guide's other exemption, notice labels, is handled by the lookbehinds;
# headings are already excluded by `scope: sentence`. See issue #20.
tokens:
- '(?<!:[^ ]+?):\s[A-Z]'
- '(?<!Note: )(?<!Caution: )(?<!Warning: )(?<!Success: )(?<=:\s)[A-Z]\w+'
+1 -1
View File
@@ -6,4 +6,4 @@ level: error
nonword: true
tokens:
- '\d{1,2}(?:\.|/)\d{1,2}(?:\.|/)\d{4}'
- '\d{1,2} (?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)|May|Jun(?:e)|Jul(?:y)|Aug(?:ust)|Sep(?:tember)?|Oct(?:ober)|Nov(?:ember)?|Dec(?:ember)?) \d{4}'
- '\d{1,2} (?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?) \d{4}'
+14
View File
@@ -0,0 +1,14 @@
extends: existence
message: "Avoid the unverifiable claim '%s'."
link: https://developers.google.com/style/excessive-claims
level: suggestion
ignorecase: true
# The guide also names 'never', 'always', and 'ensure', but in technical writing
# those are usually legitimate instructions ('never commit secrets') rather than
# product claims: they accounted for 125 of 142 hits on a 950-file corpus.
# 'best practices' is a fixed term, not a superlative.
tokens:
- 'best(?! practices?)'
- simplest
- fastest
- guarantees?
+6 -4
View File
@@ -3,11 +3,13 @@ message: "Avoid first-person pronouns such as '%s'."
link: 'https://developers.google.com/style/pronouns#personal-pronouns'
ignorecase: true
level: warning
nonword: true
# The 'I' tokens use lookaround rather than consuming the surrounding
# whitespace. Matching ' I ' made the alert span cover both spaces, which shows
# up as a too-wide underline in editors, and read as "such as ' I '". Dropping
# `nonword` also lets a project Vocab apply, which it can't when set. See PR #50.
tokens:
- (?:^|\s)I\s
- (?:^|\s)I,\s
- \bI'm\b
- '(?<=^|\s)I(?=[\s,])'
- "\\bI'm\\b"
- \bme\b
- \bmy\b
- \bmine\b
+5 -2
View File
@@ -4,8 +4,11 @@ link: "https://developers.google.com/style/capitalization#capitalization-in-titl
level: warning
scope: heading
match: $sentence
indicators:
- ":"
# No `indicators: [":"]` here. That makes Vale require a capital after a colon,
# which is the Microsoft convention this rule was originally copied from. This
# guide says the opposite: "the first word after a colon is generally
# lowercase" (developers.google.com/style/colons), and Colons.yml enforces
# exactly that. See issue #58.
exceptions:
- Azure
- CLI
+13
View File
@@ -0,0 +1,13 @@
extends: existence
message: "Avoid the jargon '%s'."
link: https://developers.google.com/style/jargon
level: suggestion
ignorecase: true
# The guide also cites 'solution', 'support', and 'workload' as overloaded
# terms, but those have ordinary technical meanings and accounted for every hit
# on a 950-file corpus, so only the unambiguous figurative terms are listed.
tokens:
- break-glass
- camel ?case
- out-of-the-box
- swim ?lane
+6 -2
View File
@@ -6,6 +6,10 @@ level: error
nonword: true
action:
name: replace
# The delimiter is a lookahead so the replacement doesn't swallow the comma or
# space that follows (issue #18). `$` is included so the abbreviation is still
# caught at the end of a heading, table cell, or block, which accounted for 8
# of 10 occurrences on a 950-file corpus.
swap:
'\b(?:eg|e\.g\.)(?=[\s,;])': for example
'\b(?:ie|i\.e\.)(?=[\s,;])': that is
'\b(?:eg|e\.g\.)(?=[\s,;]|$)': for example
'\b(?:ie|i\.e\.)(?=[\s,;]|$)': that is
+22 -1
View File
@@ -3,5 +3,26 @@ message: "Use the Oxford comma in '%s'."
link: 'https://developers.google.com/style/commas'
scope: sentence
level: warning
nonword: true
# List items may be several words long, not just one. Four guards keep the
# false-positive rate down:
#
# 1. The comma can't be the one closing a fronted subordinate clause
# ('When your alarm rings, you turn it off and tumble out of bed.') --
# that comma separates clauses, not list items. Only the first comma of
# such a sentence is exempt, so 'When it rains, apples, pears or bananas
# get wet.' is still caught.
# 2. The item can't open with a clause-introducer (', which ...',
# ', specifically ...').
# 3. The item can't open with a subject pronoun followed by a verb, which
# marks a compound predicate rather than a list ('..., you walk to the
# fridge and get a snack.'). A pronoun directly followed by 'and'/'or'
# is a real list item, so ', you and me.' still matches.
# 4. Neither item may contain an auxiliary verb, which is another compound
# predicate signal (', it has some downsides and is officially
# discouraged.').
#
# The trailing anchor allows end-of-scope so list fragments ('Apples, pears
# or bananas') are still caught.
tokens:
- '(?:[^,]+,){1,}\s\w+\s(?:and|or)'
- '(?<!^(?i:when|whenever|while|if|unless|until|although|though|because|since|after|before|once|whereas|whether|as)\b[^,]{0,80}),\s(?!(?:which|who|whom|whose|that|where|when|while|because|since|although|though|if|unless|so|but|and|or|however|therefore|thus|specifically|especially|namely|then|take|see|note|consider|make|use|either|neither)\b)(?!(?i:i|you|we|they|he|she|it)\s+(?!(?:and|or)\b))(?:(?!\b(?:is|are|was|were|has|have|had|be|been|being|will|would|can|could|should|may|might|must|do|does|did)\b)\w+ ){0,4}\w+ (?:and|or) (?:(?!\b(?:is|are|was|were|has|have|had|be|been|being|will|would|can|could|should|may|might|must|do|does|did)\b)\w+ ){0,4}\w+(?:[.?!]|$)'
+9 -1
View File
@@ -3,5 +3,13 @@ message: "Use parentheses judiciously."
link: 'https://developers.google.com/style/parentheses'
nonword: true
level: suggestion
# `[^)]` rather than `.+`: a greedy match ran from the first '(' on a line to
# the last ')', so 'Text (one) and more (two).' produced a single alert
# covering everything between them. See issue #30.
# A bare 3-5 letter acronym is skipped: Acronyms.yml requires acronyms to be
# defined as 'Spelled Out Term (ACRONYM)', so flagging those parentheses would
# put the two rules in direct conflict. The acronym has to be the whole
# parenthetical — '(NASA rocket program)' is an ordinary aside and still
# flags. Length matches the {3,5} in Acronyms.yml. See PR #59.
tokens:
- '\(.+\)'
- '\((?![A-Z]{3,5}\))[^)]+\)'
+13
View File
@@ -0,0 +1,13 @@
extends: existence
message: "Avoid time-based words like '%s' in product documentation."
link: https://developers.google.com/style/timeless-documentation
level: suggestion
ignorecase: true
# The guide also names 'now' and 'new', but both have common senses that aren't
# time-anchored ('create a new project'): adding them took a 950-file corpus of
# technical documentation from 14 hits to 117. 'recently' is left out too — every
# hit in that corpus was the UI idiom 'recently used'.
tokens:
- currently
- latest
- soon
+4 -2
View File
@@ -4,5 +4,7 @@ link: "https://developers.google.com/style/units-of-measure"
nonword: true
level: error
tokens:
- \b\d+(?:B|kB|MB|GB|TB)
- \b\d+(?:ns|ms|s|min|h|d)
- '\b\d+(?:B|kB|MB|GB|TB)\b'
- '\b\d+(?:ns|ms|min|h|d)\b'
# Seconds are split out so a decade ('1990s') isn't read as a unit.
- '\b\d+s\b(?<!\b(?:19|20)\d\ds\b)'
+3 -54
View File
@@ -2,79 +2,28 @@ extends: substitution
message: "Use '%s' instead of '%s'."
link: "https://developers.google.com/style/word-list"
level: warning
# Case matters here: each key's own capitalization is what's being corrected,
# so ignorecase would make these match their own replacements. The rest of the
# word list lives in WordListCase.yml.
ignorecase: false
action:
name: replace
swap:
"(?:API Console|dev|developer) key": API key
"(?:cell ?phone|smart ?phone)": phone|mobile phone
"(?:dev|developer|APIs) console": API console
"(?:e-mail|Email|E-mail)": email
"(?:file ?path|path ?name)": path
"(?:kill|terminate|abort)": stop|exit|cancel|end
"(?:OAuth ?2|Oauth)": OAuth 2.0
"(?:ok|Okay)": OK|okay
"(?:WiFi|wifi)": Wi-Fi
'[\.]+apk': APK
'3\-D': 3D
'Google (?:I\-O|IO)': Google I/O
"tap (?:&|and) hold": touch & hold
"un(?:check|select)": clear
above: preceding
account name: username
action bar: app bar
admin: administrator
Ajax: AJAX
a\.k\.a|aka: or|also known as
Android device: Android-powered device
android: Android
API explorer: APIs Explorer
application: app
approx\.: approximately
authN: authentication
authZ: authorization
autoupdate: automatically update
cellular data: mobile data
cellular network: mobile network
chapter: documents|pages|sections
check box: checkbox
CLI: command-line tool
click on: click|click in
Cloud: Google Cloud Platform|GCP
Container Engine: Kubernetes Engine
content type: media type
curated roles: predefined roles
data are: data is
Developers Console: Google API Console|API Console
disabled?: turn off|off
ephemeral IP address: ephemeral external IP address
fewer data: less data
file name: filename
firewalls: firewall rules
functionality: capability|feature
Google account: Google Account
Google accounts: Google Accounts
Googling: search with Google
grayed-out: unavailable
HTTPs: HTTPS
in order to: to
ingest: import|load
k8s: Kubernetes
long press: touch & hold
network IP address: internal IP address
omnibox: address bar
open-source: open source
overview screen: recents screen
regex: regular expression
SHA1: SHA-1|HAS-SHA1
sign into: sign in to
sign-?on: single sign-on
static IP address: static external IP address
stylesheet: style sheet
synch: sync
tablename: table name
tablet: device
touch: tap
url: URL
vs\.: versus
World Wide Web: web
+68
View File
@@ -0,0 +1,68 @@
extends: substitution
message: "Use '%s' instead of '%s'."
link: "https://developers.google.com/style/word-list"
level: warning
# The case-insensitive half of the word list, so sentence-initial use is caught
# ('Touch the screen', not only 'touch the screen'). Entries that must stay
# case-sensitive are in WordList.yml.
ignorecase: true
action:
name: replace
swap:
"(?:API Console|dev|developer) key": API key
"(?:cell ?phone|smart ?phone)": phone|mobile phone
"(?:dev|developer|APIs) console": API console
"(?:e-mail|Email|E-mail)": email
"(?:file ?path|path ?name)": path
"(?:kill|terminate|abort)": stop|exit|cancel|end
# Longest form first: with the shortest alternative leading, 'OAuth 2' matched
# only 'OAuth', so applying the suggestion produced 'OAuth 2.0 2'. The rule is
# already case-insensitive, so the inline (?i) is redundant. See issue #41.
'\bOauth2\.0\b|\bOAuth ?2\b(?!\.0)|\bOauth\b(?! ?2)': OAuth 2.0
"(?:ok|Okay)": OK|okay
"(?:WiFi|wifi)": Wi-Fi
'[\.]+apk': APK
'3\-D': 3D
'Google (?:I\-O|IO)': Google I/O
"tap (?:&|and) hold": touch & hold
"un(?:check|select)": clear
above: preceding
account name: username
action bar: app bar
admin: administrator
a\.k\.a|aka: or|also known as
application: app
approx\.: approximately
autoupdate: automatically update
cellular data: mobile data
cellular network: mobile network
chapter: documents|pages|sections
check box: checkbox
click on: click|click in
content type: media type
curated roles: predefined roles
data are: data is
disabled?: turn off|off
ephemeral IP address: ephemeral external IP address
fewer data: less data
file name: filename
firewalls: firewall rules
functionality: capability|feature
grayed-out: unavailable
in order to: to
ingest: import|load
long press: touch & hold
network IP address: internal IP address
omnibox: address bar
open-source: open source
overview screen: recents screen
regex: regular expression
sign into: sign in to
'(?<!single )sign-?on': single sign-on
static IP address: static external IP address
stylesheet: style sheet
synch: sync
tablename: table name
tablet: device
'touch(?! ?(?:&|and) hold)': tap
vs\.: versus
+99 -4
View File
@@ -28,6 +28,11 @@ make workflow-check # workflow-lint + workflow-dryrun
make devx-check-doc-versions # Verify docs version refs match __version__
make devx-vale # Run Vale prose linter on docs and README
make clean # Remove caches, build artifacts, coverage data
make check-workflow-artifact-deps # Verify artifact download jobs depend on upload jobs
make check-workflow-tofu-init # Verify tofu-state jobs have a tofu-init step
make check-docker-init # Check Docker Compose services with healthchecks have init: true
make check-ansible-set-fact-to-json # Check set_fact tasks don't misuse to_json
make check-alert-rules # Validate Prometheus alert rules with promtool
```
`make setup` automatically installs all development tools:
@@ -53,6 +58,74 @@ The pre-commit hook runs actionlint automatically when workflow files change.
The CI `validate` job runs `make setup-image` then `make lint-all`.
CI also runs a best-effort `make workflow-dryrun` step (skipped if act_runner is not installed in the CI Docker image).
## Composite Actions (`.gitea/actions/`)
Reusable Gitea composite actions eliminate repeated multi-step sequences
across workflows. Each action lives in its own directory under
`.gitea/actions/<name>/action.yml` and is referenced via
`uses: ./.gitea/actions/<name>`.
### Available Composite Actions
| Action | Purpose | Inputs |
|--------|---------|--------|
| `setup-env` | Run `make setup-image` (with optional `EXTRAS=`) | `extras` (default: `""`) |
| `notify-failure` | Create a Gitea issue on job failure via `devx.ci.notify_failure` | `workflow` (required) |
| `quality-checks` | 6-step quality sequence: lint, tests, speed, docs, translations, security | `package`, `test-speed-max`, `test-speed-max-single`, `translations-file` |
### Gitea 1.27 Constraints
- Every `run` step in a composite action MUST have explicit `shell:`.
- Composite actions CANNOT access `secrets` directly. They read from
the calling workflow's `env:` context (for example, `${{ env.CI_GITEA_API_TOKEN }}`).
The calling workflow's top-level `env:` block must define the required
env vars.
- `if: failure()` in a composite action step is evaluated in the
calling workflow's job-status context.
### Usage Pattern
```yaml
jobs:
validate:
env:
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
steps:
- uses: actions/checkout@v4
- uses: ./.gitea/actions/setup-env
- uses: ./.gitea/actions/quality-checks
with:
package: devx
- uses: ./.gitea/actions/notify-failure
with:
workflow: "ci/validate"
```
### Same-Repo Copies (No Cross-Repo References)
Each repo (`devx`, `grm`, `infra`) gets its own copy of the composite
actions under its `.gitea/actions/` directory. There is no
`uses: oblachno-oss/devx/.gitea/actions/...@vX.Y.Z` reference. This
avoids a single-point-of-failure where a bad devx master commit would
break all repos' CI simultaneously. See ADR-0003 for the full
rationale.
### When NOT to Use Composite Actions
- **`make setup-release` / `make setup-ci`**: the `setup-env` action
only wraps `make setup-image`. Workflows that use other setup targets
(for example, `build-images.yml` uses `make setup-release`) keep the inline
setup step.
- **Deploy-specific setup**: infra deploy workflows have additional
steps (`install-collections`, `setup-vault`, `setup_ssh_key`) that
are NOT part of the common setup. The `setup-env` action only
replaces the `make setup-image` step; deploy-specific steps stay
inline.
- **Custom notification**: `security-scan.yml` uses a Mattermost
webhook, not `devx.ci.notify_failure`. The `notify-failure` action
does not apply.
## Architecture
devx is a reusable Python package providing development and CI/CD tools for oblachno-oss projects.
@@ -90,7 +163,12 @@ src/devx/
│ ├── doc_coverage.py # Documentation coverage check
│ ├── lint_docs.py # Documentation linter (structure, links, headings, code blocks, orphans)
│ ├── validate_deploy_ref.py # Validate git tag for deployments (--github-output)
── record_deployed_tag.py # Record deployed tag to Gitea repo variable
── record_deployed_tag.py # Record deployed tag to Gitea repo variable
│ ├── cancel_superseded_runs.py # Cancel in-flight CI runs for the same PR branch
│ ├── check_workflow_artifact_deps.py # Verify artifact download jobs depend on upload jobs
│ ├── check_workflow_tofu_init.py # Verify tofu-state jobs have a tofu-init step
│ ├── discover_runners.py # Deprecated wrapper → molecule/discover_runners
│ └── wait_for_checks.py # Poll Gitea Actions for job completion (replaces inline shell polling)
├── tools/ # Developer tooling modules (run locally or by CI)
│ ├── setup.py # Environment setup (venv, deps, hooks)
│ ├── install_tools.py # Install actionlint, git-cliff, act_runner, tea, hadolint, vale
@@ -113,10 +191,24 @@ src/devx/
│ ├── pr_logs.py # Fetch logs for failed CI jobs
│ ├── pr_label.py # Add labels to PRs (idempotent)
│ ├── pre_push_check.py # Validate Vikunja task existence before push
│ ├── check_docker_init.py # Check Docker Compose services with healthchecks have init: true
│ ├── check_ansible_set_fact_to_json.py # Thin wrapper → ansible_checks/set_fact_to_json
│ ├── check_alert_rules.py # Validate Prometheus alert rules with promtool
│ ├── check_ansible_no_log.py # Thin wrapper → ansible_checks/no_log
│ ├── check_ansible_patterns.py # Thin wrapper → ansible_checks/patterns
│ ├── check_jinja_expr.py # Thin wrapper → ansible_checks/jinja_expr
│ ├── check_ansible_no_state_absent_on_db.py # Thin wrapper → ansible_checks/no_state_absent_on_db
│ ├── ansible_checks/ # Composable Ansible check subpackage (canonical implementations)
│ │ ├── _shared.py # AnsibleFileFinder, AnsibleYAMLParser, ViolationReporter
│ │ ├── no_log.py # Check missing no_log on secret-handling tasks
│ │ ├── patterns.py # Detect dangerous failure-masking patterns
│ │ ├── set_fact_to_json.py # Check set_fact tasks don't misuse to_json
│ │ ├── no_state_absent_on_db.py # Prevent state:absent on DB paths
│ │ └── jinja_expr.py # Validate Jinja2 expressions in Ansible files
│ └── _shared.py # Shared tool utilities
├── opentofu.py # OpenTofu output helpers (get_tofu_output, get_tofu_vm_ip, get_tofu_vm_field)
├── utils/ # Shared utilities (reusable across projects)
│ ├── api.py # API response helpers (is_truthy, is_falsy)
│ ├── api.py # API response helpers (is_truthy, is_falsy) + APIClient base class
│ ├── ssh.py # SSH exec + wait_for_ssh (pure-Python socket check)
│ ├── crypto.py # Secret generation (shell-safe passwords)
│ ├── vault.py # Ansible vault encrypt/decrypt helpers
@@ -124,12 +216,15 @@ src/devx/
│ ├── confirm.py # Typed confirmation validation for destructive ops
│ ├── json_registry.py # File-locked JSON registry for local state
│ ├── step_tracker.py # Multi-step operation tracking with reports
── logging.py # XDG-compliant logging configuration
── logging.py # XDG-compliant logging configuration
│ ├── ui.py # say() — unified click.echo + logging output
│ └── jinja.py # Jinja2 environment helpers + Ansible-compatible filters
└── molecule/ # Optional molecule testing helpers (for Ansible projects)
├── discover_runners.py # Dynamic Gitea runner discovery
├── discover_runners.py # Dynamic Gitea runner discovery (canonical; ci/discover_runners is a deprecated wrapper)
├── distribute_molecule.py # Distribute molecule scenarios across runners (LPT scheduling, --roles-root for multi-role)
├── molecule_ci_guard.py # Run molecule with cross-runner fail-fast (--roles-root)
├── molecule_all.py # Run all molecule scenarios locally
├── molecule_changed.py # Detect which Ansible roles changed and output molecule scenarios
├── start_docker.py # Ensure Docker daemon is running for molecule tests
└── platforms.py # Supported molecule platforms
```
+146
View File
@@ -2,6 +2,152 @@
All notable changes to this project will be documented in this file.
## [Unreleased]
### Ci
- Add composite actions (setup-env, notify-failure, quality-checks) in `.gitea/actions/`
- Convert ci.yml, post-merge.yml, build-images.yml to use composite actions
- Add notify-failure to ci/auto-merge and build-images/cleanup jobs (previously missing)
- Fix `make workflow-dryrun` failure: use `env` context instead of
`vars`/`secrets` in `container.credentials` (act_runner dry-run
does not populate vars/secrets)
- Remove redundant step-level `CI_GITEA_API_TOKEN`/`CI_GITEA_USERNAME`
env vars (now in workflow-level `env:` block)
## [0.50.4] - 2026-08-12
### Bug Fixes
- Add tenacity retry to install_tools._download for transient network failures
## [0.50.3] - 2026-08-12
### Refactor
- Extract wait_for_checks, consolidate ansible_checks, deprecate ci/discover_runners
## [0.50.2] - 2026-08-12
### Bug Fixes
- Run molecule destroy on test failure to clean up containers
## [0.50.1] - 2026-08-12
### Bug Fixes
- Configure git remote with CI token for post-merge push
## [0.50.0] - 2026-08-12
### Features
- Sync missing features from v0.49.x line to master
## [0.49.5] - 2026-08-07
### Performance
- Skip dep resolution in setup-image with --no-deps
## [0.49.4] - 2026-08-07
### Bug Fixes
- Add container.credentials for private registry auth
## [0.49.3] - 2026-08-07
### Bug Fixes
- Retry ansible-galaxy collection install on transient timeouts
## [0.49.2] - 2026-08-07
### Bug Fixes
- Add fallback URL for tea download
## [0.49.1] - 2026-08-07
### Bug Fixes
- Add container images to build-images workflow
## [0.49.0] - 2026-08-07
### Features
- Add --include-roles and --exclude-roles to distribute_molecule
## [0.48.0] - 2026-07-22
### Features
- Extract reusable components from infra and grm into devx
## [0.49.5] - 2026-08-07
### Performance
- Skip dep resolution in setup-image with --no-deps
## [0.49.4] - 2026-08-07
### Bug Fixes
- Add container.credentials for private registry auth
## [0.49.3] - 2026-08-07
### Bug Fixes
- Retry ansible-galaxy collection install on transient timeouts
## [0.49.2] - 2026-08-07
### Bug Fixes
- Add fallback URL for tea download
## [0.49.1] - 2026-08-07
### Bug Fixes
- Add container images to build-images workflow
## [0.49.0] - 2026-08-07
### Features
- Add --include-roles and --exclude-roles to distribute_molecule
## [0.48.0] - 2026-07-22
### Features
- Extract reusable components from infra and grm into devx
## [0.48.0] - 2026-07-22
### Features
- Extract reusable components from infra and grm into devx
## [Unreleased]
### Features
- Extract reusable components from infra and grm into devx:
- `devx.utils.ui.say()` — unified click.echo + logging output
- `devx.utils.api.APIClient` — base HTTP API client class with retry logic
- `devx.utils.jinja` — Jinja2 environment helpers with Ansible-compatible filters
- `devx.i18n.configure_i18n()` — configurable `lang_env_var` and `translations_path_env_var`
- `devx.ci.cancel_superseded_runs` — cancel in-flight CI runs for the same PR branch
- `devx.ci.check_workflow_artifact_deps` — verify artifact download jobs depend on upload jobs
- `devx.ci.check_workflow_tofu_init` — verify tofu-state jobs have a tofu-init step
- `devx.tools.check_docker_init` — check Docker Compose services with healthchecks have init: true
- `devx.tools.check_ansible_set_fact_to_json` — check set_fact tasks don't misuse to_json
- `devx.tools.check_alert_rules` — validate Prometheus alert rules with promtool
- Add `jinja2` and `pyyaml` as core dependencies (previously in `deploy` extras only)
- Register new CLI commands: `devx ci cancel-superseded-runs`, `devx ci check-workflow-artifact-deps`,
`devx ci check-workflow-tofu-init`, `devx tools check-docker-init`,
`devx tools check-ansible-set-fact-to-json`, `devx tools check-alert-rules`
- Add Makefile targets for all new check tools
## [0.47.3] - 2026-07-17
### Bug Fixes
+27 -1
View File
@@ -1,4 +1,5 @@
.PHONY: all setup setup-ci setup-quality setup-release setup-image install update lint lint-all lint-dockerfiles test test-unit pytest-cov clean install-tools install-hooks activate-scripts checkmake check-mutable-globals check-dep-docs check-test-speed build-images push-images build-images-dry-run clean-images
.PHONY: check-workflow-artifact-deps check-workflow-tofu-init check-docker-init check-ansible-set-fact-to-json check-alert-rules
PYTHON := python3
VENV := .venv
@@ -65,7 +66,7 @@ setup-release: $(VENV)/bin/activate .env
# an older devx.mak that doesn't yet define devx-setup-image. Consumer repos
# (grm, infra) can safely alias to devx-setup-image since they install devx from PyPI.
setup-image:
@if [ -d /opt/venv ]; then ln -sf /opt/venv $(VENV); . $(VENV)/bin/activate && pip install --no-cache-dir -e . 2>/dev/null; \
@if [ -d /opt/venv ]; then ln -sf /opt/venv $(VENV); . $(VENV)/bin/activate && pip install --no-cache-dir --no-deps -e . 2>/dev/null; \
else echo "[setup-image] /opt/venv not found — falling back to setup-ci"; $(MAKE) setup-ci; fi
install-hooks:
@@ -113,6 +114,31 @@ pr-rebase: devx-pr-rebase
lint-all: lint workflow-lint lint-dockerfiles
@echo "[lint-all] All linting checks passed."
# ── Workflow / Ansible / Docker check tools ─────────────────────────────────
# Generic check tools ported from infra. These targets are no-ops in devx
# itself (no .gitea/workflows or ansible/ directory) but provide the
# canonical entry points for consumer repos that include devx.mak.
check-workflow-artifact-deps:
@$(BIN)/python -m devx.ci.check_workflow_artifact_deps || \
echo "[check-workflow-artifact-deps] No workflows directory found — skipping."
check-workflow-tofu-init:
@$(BIN)/python -m devx.ci.check_workflow_tofu_init || \
echo "[check-workflow-tofu-init] No workflows directory found — skipping."
check-docker-init:
@$(BIN)/python -m devx.tools.check_docker_init || \
echo "[check-docker-init] No ansible templates found — skipping."
check-ansible-set-fact-to-json:
@$(BIN)/python -m devx.tools.check_ansible_set_fact_to_json || \
echo "[check-ansible-set-fact-to-json] No ansible directory found — skipping."
check-alert-rules:
@$(BIN)/python -m devx.tools.check_alert_rules --template-path ansible/roles/observability/templates || \
echo "[check-alert-rules] No alert-rules template found — skipping."
# Note: Not aliased to devx-lint-dockerfiles for the same reason as setup-image —
# devx's own CI images may have an older devx.mak. Consumer repos can safely alias.
lint-dockerfiles:
+21 -15
View File
@@ -12,16 +12,16 @@ opinionated CI/CD pipeline: conventional commits, automated versioning via
git-cliff, squash-merge automation, Vikunja task tracking, wiki sync, and
quality badges.
> An open-source project from **Oblachno** (облачно means *cloudy* in Bulgarian).
> An open source project from **Oblachno** (облачно means *cloudy* in Bulgarian).
[![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/src/branch/master/LICENSE)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/python.svg)](https://www.python.org/downloads/)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/python.svg)](https://www.python.org/downloads/)
## Why devx?
@@ -87,7 +87,7 @@ extra index and list devx in your dependencies:
```toml
[project]
dependencies = [
"devx>=0.47.3",
"devx>=0.50.4",
]
[tool.pip]
@@ -101,8 +101,8 @@ pip install -e .
```
> **Note:** If your project requires a specific devx version, pin it in
> `dependencies` (for example, `"devx==0.47.3"`) or use a version constraint
> (for example, `"devx>=0.47.3,<0.48"`).
> `dependencies` (for example, `"devx==0.50.4"`) or use a version constraint
> (for example, `"devx>=0.50.4,<0.51"`).
### Optional extras
@@ -226,10 +226,6 @@ python -m devx.molecule.distribute_molecule --runner-index 1 --max-runners 3
python -m devx.molecule.distribute_molecule --list # list all scenarios
python -m devx.molecule.distribute_molecule --list-platforms # list platforms
# Run molecule tests with cross-runner fail-fast
python -m devx.molecule.molecule_ci_guard pair1 pair2
python -m devx.molecule.molecule_ci_guard --roles-root ansible/roles pair1 pair2
# Run all molecule scenarios locally (sequential)
python -m devx.molecule.molecule_all
python -m devx.molecule.molecule_all --bin .venv/bin
@@ -303,7 +299,6 @@ devx --version
| `devx molecule all` | Run all molecule scenarios on all supported platforms |
| `devx molecule discover-runners` | Discover available Gitea Actions runners |
| `devx molecule distribute` | Distribute molecule test pairs across parallel runners |
| `devx molecule guard` | Run molecule tests with CI failure polling |
See [CLI Commands](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki/CLI-Commands)
in the wiki for full command documentation with examples.
@@ -450,6 +445,17 @@ src/devx/
└── molecule/ # Optional molecule testing helpers (for Ansible projects)
```
### Composite Actions (`.gitea/actions/`)
Reusable Gitea composite actions for CI workflow steps:
- `setup-env` — runs `make setup-image` (with optional `EXTRAS=`)
- `notify-failure` — creates a Gitea issue on job failure
- `quality-checks` — 6-step quality gate (lint, tests, speed, docs, translations, security)
Each consumer repo gets its own copy (no cross-repo references). See
ADR-0003 for the design rationale.
### Design principles
- **Self-contained package** — `src/devx/` never imports from scripts outside the package
@@ -0,0 +1,108 @@
# ADR-0002: Ansible Check Tool Consolidation and wait_for_checks Extraction
Date: 2026-08-12
Status: Accepted
## Context
The devx package had two categories of code duplication and inline
workflow logic that were hard to test and maintain:
### 1. Ansible Check Tools — Duplicated Boilerplate
Five Ansible check tools (`check_ansible_no_log`,
`check_ansible_patterns`, `check_ansible_set_fact_to_json`,
`check_ansible_no_state_absent_on_db`, `check_jinja_expr`) each
implemented their own file discovery, YAML parsing, task iteration, and
violation reporting logic. While the check logic differed, the
supporting infrastructure was copy-pasted across all five modules:
- `find_task_files()` — glob YAML files, skip molecule
- YAML multi-document parsing with error handling
- Task iteration (bare lists, play dicts with `tasks`/`pre_tasks`/`post_tasks`/`handlers`, nested `block` tasks)
- Violation formatting (`path:line — message`)
This made it difficult to add new checks (each new tool repeated the
boilerplate) and risky to change shared behavior (fixes had to be
applied to all five modules independently).
### 2. Inline Job Polling in Workflow YAML
The `grm` repository's `ci.yml` workflow contained ~25 lines of inline
shell + Python polling logic to wait for the `molecule-tests` job to
complete before the auto-merge step. This logic:
- Was not testable (embedded in workflow YAML)
- Duplicated the Gitea API client pattern already used elsewhere
- Had no timeout handling, no error reporting, no retry logic
- Could not be reused by other repositories
### 3. Duplicate discover_runners Modules
`devx.ci.discover_runners` and `devx.molecule.discover_runners` were
near-identical modules. The `ci/` version had better error logging
(warnings on non-200 responses, 403 suppression for instance-level
queries), while the `molecule/` version silently swallowed errors.
Both were imported by different workflows, making it unclear which was
canonical.
## Decision
### 1. Composable `ansible_checks/` Subpackage
Consolidate the five Ansible check tools into a
`devx.tools.ansible_checks/` subpackage with shared utilities:
- `_shared.py``AnsibleFileFinder`, `AnsibleYAMLParser`,
`ViolationReporter` classes providing composable helpers
- `no_log.py`, `patterns.py`, `set_fact_to_json.py`,
`no_state_absent_on_db.py`, `jinja_expr.py` — canonical check
implementations using the shared utilities
The old modules (`check_ansible_*.py`, `check_jinja_expr.py`) remain as
**thin backward-compat wrappers** that re-export the canonical
implementation and preserve the CLI entry point. This avoids breaking
existing Makefile targets and workflow references.
**Composition over inheritance**: each check module picks the helpers it
needs. Tools that don't parse YAML (for example line-based scanners) can skip
`AnsibleYAMLParser` entirely.
### 2. Extracted `wait_for_checks` Module
Extract the inline polling logic into `devx.ci.wait_for_checks`:
- Polls the Gitea API for job completion status
- Configurable job name prefix, timeout, poll interval
- Exit codes: 0 (success), 1 (failure), 2 (timeout), 3 (API error)
- `--require-success/--no-require-success` flag for flexibility
- 100% test coverage with mocked API responses
This replaces the inline shell polling in `grm` `ci.yml` with a
reusable, testable Python module.
### 3. Deprecated `ci/discover_runners` Wrapper
Merge the `ci/discover_runners` implementation (with its better error
logging) into `molecule/discover_runners` as the canonical version.
Make `ci/discover_runners` a deprecated wrapper that:
- Re-exports all public symbols from `molecule.discover_runners`
- Emits a `DeprecationWarning` when run as `__main__`
- Preserves backward compatibility for existing workflow references
New code should import from `devx.molecule.discover_runners` directly.
## Consequences
- **New checks are easier to write**: import `_shared` helpers, implement
only the check-specific logic
- **Shared behavior can be fixed in one place**: file discovery, YAML
parsing, violation formatting
- **Workflow polling is testable**: `wait_for_checks` has 26 unit tests
covering success, failure, timeout, and API error scenarios
- **Backward compatibility preserved**: all existing Makefile targets,
workflow references, and test imports continue to work via wrappers
- **Migration path is gradual**: new code uses the subpackage; old code
can migrate at its own pace; wrappers can be removed in a future
release once all references are updated
+155
View File
@@ -0,0 +1,155 @@
# ADR-0003: Composite Actions for CI Workflow Reuse
Date: 2026-08-12
Status: Accepted
## Context
The devx repository's Gitea Actions workflows (`.gitea/workflows/ci.yml`,
`post-merge.yml`, `build-images.yml`) repeated multi-step
sequences across jobs:
1. **Set up environment**`make setup-image` (optionally with `EXTRAS=`).
Appeared verbatim in 4 jobs across `ci.yml` and `post-merge.yml`, each
with the same `CI_GITEA_API_TOKEN` env wiring.
2. **Notify on failure**`python3 -m devx.ci.notify_failure ... --auto-login`
with venv activation, PATH export, and 5 fixed CLI args. Appeared in
4 jobs (`ci/validate`, `post-merge/detect-and-configure`,
`post-merge/release-and-maintain`, `build-images/build-and-push`),
each differing only in the `--workflow` string.
3. **Quality checks** — a 6-step sequence (lint-all, pytest-cov,
check-test-speed, devx-docs-check, check-translations, pip-audit)
with venv activation boilerplate on every step. Appeared once in
`ci.yml` validate job, but the same sequence is needed by `grm` and
`infra` (Phase 2b/2c of the cross-repo refactoring plan).
This duplication had the following costs:
- **Drift risk**: a fix to notify-failure (for example, new flag,
different env var) had to be applied to 4 places; missing one caused
inconsistent failure notifications.
- **Workflow YAML noise**: the 6-step quality block obscured the
validate job's actual structure (detect-changes, pr-review,
release-dry-run).
- **Cross-repo reuse blocked**: `grm` and `infra` could not adopt the
same quality-checks sequence without copy-pasting the inline steps,
which would amplify the drift problem across 3 repos.
- **Gitea 1.27 constraints**: every `run` step needs explicit `shell:`;
composite actions cannot access `secrets` directly (only `env:`).
These constraints had to be re-discovered and re-applied per step.
## Decision
Introduce three Gitea composite actions in `.gitea/actions/`:
### 1. `setup-env/action.yml`
Wraps the `make setup-image` call. Single input `extras` (default empty)
forwarded to `make setup-image EXTRAS=`. Reads `CI_GITEA_API_TOKEN` and
`CI_GITEA_USERNAME` from the calling workflow's `env:` context.
### 2. `notify-failure/action.yml`
Wraps the `devx.ci.notify_failure` invocation. Single required input
`workflow` (the workflow/job name for the Gitea issue title). Step is
gated by `if: failure()` so it only runs on job failure. Reads
`CI_GITEA_API_TOKEN` from the calling workflow's `env:` context.
### 3. `quality-checks/action.yml`
Wraps the 6-step quality sequence. Inputs:
- `package` (default empty) — sets `DEVX_DOC_VERSIONS_PKG` for doc
version checks (for example, `devx`, `grm`).
- `test-speed-max` (default `15`) — total test seconds threshold.
- `test-speed-max-single` (default `0.5`) — per-test seconds threshold.
- `translations-file` (default empty) — path to `translations.json`
for repos whose translations live outside `src/devx/`.
Each step activates the venv defensively
(`. .venv/bin/activate 2>/dev/null || true`) so the action works with
both pre-built CI images (which symlink `/opt/venv` to `.venv`) and
fresh `make setup-image` runs.
### Adoption Scope
- **`ci.yml` validate job**: `setup-env` + `quality-checks` +
`notify-failure`.
- **`ci.yml` auto-merge job**: `setup-env` + `notify-failure` (no
quality checks — auto-merge only runs after validate passes).
- **`post-merge.yml` detect-and-configure**: `setup-env` +
`notify-failure`.
- **`post-merge.yml` release-and-maintain**: `setup-env` (with
`extras: "release"`) + `notify-failure`.
- **`build-images.yml` build-and-push**: `notify-failure` only. The
setup steps use `make setup-release` (not `make setup-image`), so
`setup-env` does not apply.
- **`build-images.yml` cleanup**: `notify-failure` only. The setup
step uses `make setup-ci` (not `make setup-image`).
### Same-Repo Copies (No Cross-Repo References)
Each consumer repo (`devx`, `grm`, `infra`) gets its own copy of the
composite actions under its `.gitea/actions/` directory. There is no
`uses: oblachno-oss/devx/.gitea/actions/...@vX.Y.Z` reference.
This avoids a single-point-of-failure where a bad `devx` master commit
would break all three repos' CI simultaneously. The cost is three
copies of ~30 lines of YAML each, updated manually when a composite
action changes. Given the stability of these patterns (the inline
versions were unchanged for months), this cost is acceptable.
## Consequences
### Positive
- **Workflow YAML is shorter and clearer**: the validate job's
quality block collapses from 32 lines to 5 lines. The intent
(`uses: ./.gitea/actions/quality-checks`) is more legible than
6 individually wrapped steps.
- **Drift eliminated**: a change to notify-failure (new flag, different
env var) is applied in one file. All 4 calling sites pick it up.
- **Cross-repo reuse enabled**: Phase 2b (`grm`) and Phase 2c (`infra`)
copy the same `action.yml` files and adopt the same `uses:` pattern.
The quality-checks sequence is now portable.
- **Gitea 1.27 constraints centralized**: the `shell: bash` and
`env:` (not `secrets`) patterns are encoded once per action, not
re-derived per step.
- **No release triggered**: changes to `.gitea/**` are classified as
workflow-only by `devx.ci.classify_changes`. Phase 2a does not
produce a new devx version. `grm`/`infra` bump to the Phase 1
release (v0.50.4), not a Phase 2a version.
### Negative
- **Three copies of each action**: when a composite action changes,
the change must be applied to `devx`, `grm`, and `infra`
independently. This is intentional (see Same-Repo Copies preceding)
but is a maintenance cost.
- **Composite action debugging is harder**: Gitea's log output for
composite action steps is nested under the action name. Finding the
failing step requires reading one more level of indentation.
- **`env:` propagation is implicit**: the calling workflow's top-level
`env:` block must define `CI_GITEA_API_TOKEN` for the composite
action to read it. A workflow that omits this will see an empty
token at runtime, not at lint time. actionlint does not catch this.
- **`quality-checks` is devx-shaped**: the `package` and
`translations-file` inputs exist because consumer repos (for example,
`grm`) have translations files outside the default
`src/devx/translations.json` location and need doc version checks
targeting their own package name.
A repo with a different translations path or package layout would need
a new input or a different action. This is acceptable for the current
3-repo scope.
### Neutral
- **`if: failure()` is preserved**: the `notify-failure` composite
action's step has `if: failure()`, which is evaluated in the
calling workflow's job-status context. This is the standard Gitea
Actions pattern for post-failure notification.
- **Venv activation is defensive**: `. .venv/bin/activate 2>/dev/null
|| true` does not fail if the venv is missing (pre-built image path)
or already active. This matches the inline pattern's behavior.
+12 -12
View File
@@ -8,16 +8,16 @@ parallel test distribution, and more into a single installable package.
It was extracted from the [GRM](https://git.oblachno.oblachno.fyi/oblachno-oss/grm)
project to be reusable across all oblachno-oss repositories.
> An open-source project from **Oblachno** (облачно means *cloudy* in Bulgarian).
> An open source project from **Oblachno** (облачно means *cloudy* in Bulgarian).
[![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/src/branch/master/LICENSE)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/82b4caf3bcf5fb7058654abab87cd2fea339d882/python.svg)](https://www.python.org/downloads/)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/43d0589aeee6d706356f79deef37662c9291b981/python.svg)](https://www.python.org/downloads/)
## Overview
@@ -74,14 +74,14 @@ Add devx to your `pyproject.toml` dependencies and configure the registry:
```toml
[project]
dependencies = [
"devx>=0.47.3",
"devx>=0.50.4",
]
[tool.pip]
extra-index-url = "https://git.oblachno.oblachno.fyi/api/packages/oblachno-oss/pypi/simple"
```
Pin a specific version if needed: `"devx==0.47.3"` or `"devx>=0.47.3,<0.48"`.
Pin a specific version if needed: `"devx==0.50.4"` or `"devx>=0.50.4,<0.51"`.
### Optional extras
@@ -104,8 +104,8 @@ devx is a self-contained Python package under `src/devx/`:
- **Dev tools** (`devx.tools`) — setup, install_tools, check_test_speed,
configure_repo, generate_badges, generate_cliff_config, install_checkmake
- **Molecule tools** (`devx.molecule`) — Optional, for projects with Ansible
roles: distribute_molecule, molecule_ci_guard, molecule_all, discover_runners,
start_docker, platforms
roles: distribute_molecule, molecule_all, discover_runners, start_docker,
platforms
See [Architecture](Architecture) for the full package structure, module
descriptions, design principles, and data flow diagrams.
@@ -132,7 +132,7 @@ devx provides a `devx` CLI with three command groups:
- `devx ci <command>` — CI/CD automation (17 commands)
- `devx tools <command>` — Developer tools (9 commands)
- `devx molecule <command>` — Molecule testing (4 commands, optional)
- `devx molecule <command>` — Molecule testing (3 commands, optional)
See [CLI Commands](CLI-Commands) for full command documentation with examples.
+3 -1
View File
@@ -3,5 +3,7 @@
"user/getting-started.md": "Getting-Started",
"user/cli-commands.md": "CLI-Commands",
"tech/architecture.md": "Architecture",
"tech/ci-cd-workflow.md": "CI-CD-Workflow"
"tech/ci-cd-workflow.md": "CI-CD-Workflow",
"decisions/0001-test-isolation-pytest-plugin-and-shift-left-quality-gates.md": "ADR-0001-Test-Isolation",
"decisions/0002-ansible-check-consolidation-and-wait-for-checks.md": "ADR-0002-Ansible-Check-Consolidation"
}
@@ -132,7 +132,7 @@ unblocked auto-merge across all three repos.
### 2. Double-Prefix Detection (MEDIUM impact)
`check_auto_merge_ready.py` now detects and rejects Vikunja task titles
that include the identifier prefix (for example, "DEVX-127: Fix...").
that include the identifier prefix (for example, "DEVX-127: Fix").
The validator adds the prefix automatically, so a double prefix would
fail validation.
+46 -11
View File
@@ -33,7 +33,8 @@ src/devx/
│ ├── notify_failure.py # Create Gitea issues on CI failures
│ ├── distribute_files.py # Distribute files across parallel runners
│ ├── integration_guard.py # Run pytest with cross-runner fail-fast
│ ├── discover_runners.py # Dynamic Gitea runner discovery
│ ├── discover_runners.py # Deprecated wrapper → molecule/discover_runners
│ ├── wait_for_checks.py # Poll Gitea Actions for job completion
│ ├── check_translations.py # Translation completeness check
│ └── doc_coverage.py # Documentation coverage check
├── tools/ # Developer tooling modules (run locally or by CI)
@@ -48,7 +49,7 @@ src/devx/
│ └── install_checkmake.py # Install checkmake (Makefile linter)
└── molecule/ # Optional molecule testing helpers (Ansible projects)
├── __init__.py
├── discover_runners.py # Dynamic Gitea runner discovery
├── discover_runners.py # Dynamic Gitea runner discovery (canonical)
├── distribute_molecule.py # Distribute scenarios across runners
├── molecule_ci_guard.py # Run molecule with cross-runner fail-fast
├── molecule_all.py # Run all molecule scenarios locally
@@ -87,11 +88,11 @@ overridden via environment variables with the `DEVX_` prefix. Provides:
- `GITEA_API_URL` / `VIKUNJA_API_URL` — API endpoints
- `REPO_OWNER` — repository owner (must be set per-project)
- `TASK_PREFIX` / `TASK_ID_RE` — task ID prefix and regex (for example, `DEVX-N`)
- `TASK_PREFIX` / `TASK_ID_RE` — task ID prefix and regular expression (for example, `DEVX-N`)
- `VIKUNJA_PROJECT_ID` — Vikunja project for task tracking
- `DEFAULT_TIMEOUT`, `DEFAULT_PER_PAGE` — HTTP client defaults
- `MAX_RETRIES`, `RETRY_BACKOFF_BASE`, `RETRY_STATUS_CODES` — retry config
- `CONVENTIONAL_RE` — conventional commit format regex
- `CONVENTIONAL_RE` — conventional commit format regular expression
### `exceptions.py`
@@ -109,7 +110,7 @@ wraps user-facing strings for translation.
Projects can extend translations by setting `DEVX_TRANSLATIONS_PATH` to a
custom JSON file. Keys from the project's file are merged on top of devx's
built-in translations, allowing projects to override or add keys without
built-in translations, allowing projects to override, or add keys without
modifying the package.
### `api_clients.py`
@@ -171,7 +172,7 @@ from `devx.api_clients`, `devx.config`, `devx.gitea_cli`, and `devx.i18n`.
Automated release using git-cliff. Calculates the next semver version from
conventional commits since the last tag, updates `__version__` in
`__init__.py` and `CHANGELOG.md`, runs lint and tests to verify the release
`__init__.py` and `CHANGELOG.md`, runs lint, and tests to verify the release
is healthy, commits with `release: vX.Y.Z [skip ci]`, creates an annotated
tag, and pushes both to master.
@@ -285,13 +286,24 @@ Click commands from `cli.py` and verifies each has documentation in
`architecture.md` and CI scripts in `ci-cd-workflow.md`. Supports
`--fail-on-missing` to enforce 100% coverage.
### `discover_runners.py`
### `discover_runners.py` (deprecated wrapper)
> **Deprecated:** Use `devx.molecule.discover_runners` instead. This
> module is a thin wrapper that re-exports the canonical implementation.
Discovers available Gitea Actions runners at three levels: repository,
organization, and instance (admin). Falls back to the `MOLECULE_RUNNERS` repo
organization, and instance (administrator). Falls back to the `MOLECULE_RUNNERS` repo
variable or `DEFAULT_MAX_RUNNERS` (3). Outputs runner count or a JSON index
array for use as a dynamic matrix in Gitea Actions.
### `wait_for_checks.py`
Polls the Gitea Actions API for job completion status. Used by auto-merge
jobs that need to wait for parallel jobs (for example molecule-tests) before
proceeding. Replaces inline shell polling in workflow YAML with a
reusable, testable Python module. Exit codes: 0 (success), 1 (job
failure), 2 (timeout), 3 (API error or no matching jobs).
### `distribute_files.py`
Distributes files matching a glob pattern across N parallel runners
@@ -330,7 +342,7 @@ Supports `--tool` to install specific tools and `--list` to show status.
Runs unit tests and enforces execution-time budgets. Two quality gates:
total suite time must not exceed `--max-seconds` (default: 10s), and no
individual test may exceed `--max-single-seconds` (default: 0.5s, 0 to
disable). Runs `make test-unit` with `PYTEST_ADDOPTS=--durations=0`.
off). Runs `make test-unit` with `PYTEST_ADDOPTS=--durations=0`.
### `check_test_isolation.py`
@@ -396,8 +408,13 @@ Intended for local development; CI uses the parallel matrix instead.
### `molecule/discover_runners.py`
Discovers available Gitea Actions runners for molecule tests. Same logic as
`devx.ci.discover_runners` but intended for molecule-specific workflows.
Discovers available Gitea Actions runners for molecule tests. This is the
canonical implementation; `devx.ci.discover_runners` is a deprecated wrapper
that re-exports from this module. Queries runners at repository,
organization, and instance (administrator) levels, with warnings logged
to stderr on non-200 responses (except 403 on instance-level, which is
expected without admin scope). Falls back to `MOLECULE_RUNNERS` env var
or `DEFAULT_MAX_RUNNERS` (3).
### `start_docker.py`
@@ -436,6 +453,24 @@ v2 failures. Supports loading custom platforms from a JSON file.
- **Secrets via environment** — secrets are passed via environment variables,
never on the command line.
## Composite Actions (`.gitea/actions/`)
Reusable Gitea composite actions eliminate repeated multi-step sequences
across workflows. Each action lives in `.gitea/actions/<name>/action.yml`
and is referenced via `uses: ./.gitea/actions/<name>`.
| Action | Purpose |
|--------|---------|
| `setup-env` | Run `make setup-image` (with optional `EXTRAS=`) |
| `notify-failure` | Create a Gitea issue on job failure via `devx.ci.notify_failure` |
| `quality-checks` | 6-step quality sequence: lint, tests, speed, docs, translations, security |
Each consumer repo (`devx`, `grm`, `infra`) gets its own copy — there are
no cross-repo composite action references. This avoids a single-point-of-failure
where a bad devx master commit would break all repos' CI simultaneously.
See ADR-0003 for the full design rationale and Gitea 1.27 constraints.
## Import rules
1. **`src/devx/` is self-contained** — the package never imports from outside `src/`
+36 -19
View File
@@ -38,22 +38,33 @@ The single validation job. Consolidates the former `quality`,
`detect-changes`, `release-dry-run`, `pr-review`, and `pre-merge-check`
jobs into one job to save checkout+setup overhead. Runs on every PR.
**Quality steps**
**Setup and quality steps** (composite actions)
The main quality gate:
The validate job uses three composite actions from `.gitea/actions/`:
1. **Lint all** — ruff check, ruff format check, pyright, bandit, actionlint
(via `make lint-all`)
2. **Unit tests with 100% coverage**`make pytest-cov`
3. **Check unit test speed** — `python -m devx.tools.check_test_speed
--max-seconds 4 --max-single-seconds 0.5`
4. **Documentation coverage check**`python -m devx.ci.doc_coverage
--fail-on-missing`
5. **Translation completeness check**`python -m devx.ci.check_translations`
6. **Dependency security scan**`pip-audit --desc --skip-editable`
(best-effort, non-blocking)
7. **Workflow dry-run validation**`make workflow-dryrun` via act_runner
(best-effort, skipped if act_runner is not installed)
1. **`setup-env`** — runs `make setup-image` to link the pre-built venv
and install the project (no-deps mode)
2. **`quality-checks`** — runs the 6-step quality gate:
- **Lint all** — ruff check, ruff format check, pyright, bandit,
actionlint (via `make lint-all`)
- **Unit tests with 100% coverage**`make pytest-cov`
- **Check unit test speed** — `python -m devx.tools.check_test_speed
--max-seconds 15 --max-single-seconds 0.5`
- **Documentation gate**`make devx-docs-check` (coverage + stale
refs + lint + version refs + prose)
- **Translation completeness check**`python -m devx.ci.check_translations`
- **Dependency security scan**`pip-audit --desc --skip-editable`
(best-effort, non-blocking)
3. **`notify-failure`** — creates a Gitea issue if any step fails
The quality-checks action accepts inputs (`package`, `test-speed-max`,
`test-speed-max-single`, `translations-file`) for cross-repo reuse.
See ADR-0003 for the composite action design rationale.
**Workflow dry-run validation** (inline step, not part of composite action)
`make workflow-dryrun` via act_runner (best-effort, skipped if
act_runner is not installed).
**`detect-changes` step**
@@ -574,8 +585,14 @@ picks up the new version number). This prevents infinite loops.
## Failure handling
Every job in the CI and post-merge workflows has a `notify_failure` step
that runs `if: failure()`. This creates a Gitea issue with the workflow name,
run ID, and commit SHA, ensuring failures that would otherwise go unnoticed
in the Actions tab are surfaced as issues. The issue is created via the tea
CLI with a `bug` label if available.
Every job in the CI, post-merge, and build-images workflows uses the
`notify-failure` composite action (`.gitea/actions/notify-failure`),
which runs `if: failure()`. This creates a Gitea issue with the workflow
name, run ID, and commit SHA, ensuring failures that would otherwise go
unnoticed in the Actions tab are surfaced as issues. The issue is created
via the tea CLI with a `bug` label if available.
Jobs with `notify-failure`:
- `ci/validate`, `ci/auto-merge`
- `post-merge/detect-and-configure`, `post-merge/release-and-maintain`
- `build-images/build-and-push`, `build-images/cleanup`
+135 -4
View File
@@ -83,9 +83,14 @@ devx ci detect-release-commit
### `devx ci discover-runners`
> **Deprecated:** Use `devx molecule discover-runners` instead. This
> command is a thin wrapper that re-exports the canonical implementation
> from `devx.molecule.discover_runners`. It will be removed in a future
> release.
Discover available Gitea Actions runners for dynamic job distribution.
Queries the Gitea API for registered runners at repository, organization, and
instance (admin) levels. Falls back to `MOLECULE_RUNNERS` repo variable or
instance (administrator) levels. Falls back to `MOLECULE_RUNNERS` repo variable or
`DEFAULT_MAX_RUNNERS` (3).
```bash
@@ -315,6 +320,82 @@ devx ci validate-commit-msg commit-msg.txt --branch master
Options:
- `--branch <branch>` — override branch detection (for CI use)
### `devx ci wait-for-checks`
Wait for Gitea Actions jobs to complete by polling the API. Used by
auto-merge jobs that need to wait for parallel jobs (for example molecule-tests)
before proceeding. Replaces inline shell polling in workflow YAML with
a reusable, testable Python module.
Exit codes:
- `0` — all matching jobs completed successfully
- `1` — one or more matching jobs failed (when `--require-success` is set)
- `2` — timeout reached before all jobs completed
- `3` — API error or no matching jobs found
```bash
devx ci wait-for-checks --job-name molecule-tests --repo oblachno-oss/grm
devx ci wait-for-checks --job-name molecule-tests --timeout 1200 --poll-interval 10
devx ci wait-for-checks --job-name molecule-tests --no-require-success
```
Options:
- `--job-name <prefix>` — job name prefix to match (required)
- `--repo <owner/name>` — repository (default: `$GITHUB_REPOSITORY`)
- `--timeout <seconds>` — max wait time (default: 1200 = 20 min)
- `--poll-interval <seconds>` — seconds between polls (default: 10)
- `--require-success / --no-require-success` — exit 1 if a job failed (default: yes)
### `devx ci cancel-superseded-runs`
Cancel in-flight CI runs for the same PR branch when a new push triggers
a new run. Uses the Gitea Actions API to list running pull_request runs
and cancel those with a lower run ID on the same branch.
```bash
devx ci cancel-superseded-runs \
--repo "$REPOSITORY" \
--current-run-id "$GITHUB_RUN_ID" \
--head-branch "$HEAD_REF"
```
Options:
- `--repo <owner/repo>` — repository (required)
- `--current-run-id <id>` — current run ID, not cancelled (required)
- `--head-branch <branch>` — PR head branch name (required)
- `--dry-run` — list superseded runs without cancelling
- `--base-url <url>` — Gitea base URL (default: `GITEA_API_URL` env var)
### `devx ci check-workflow-artifact-deps`
Verify that workflow jobs downloading artifacts depend on the uploading
job. Prevents the class of bug where a download job runs in parallel
with the upload job and fails because the artifact isn't available yet.
```bash
devx ci check-workflow-artifact-deps
devx ci check-workflow-artifact-deps --workflow .gitea/workflows/ci.yml
```
Options:
- `--workflow <path>` — check a specific workflow file
- `--workflows-dir <path>` — override workflows directory
### `devx ci check-workflow-tofu-init`
Verify that workflow jobs using tofu state (tofu output/plan/apply or
scripts that call them) have a tofu-init step in the same job.
```bash
devx ci check-workflow-tofu-init
devx ci check-workflow-tofu-init --workflow .gitea/workflows/deploy.yml
```
Options:
- `--workflow <path>` — check a specific workflow file
- `--workflows-dir <path>` — override workflows directory
- `--state-script <name>` — add a script that uses tofu state (repeatable)
## Tools Commands
### `devx tools check-test-speed`
@@ -323,7 +404,7 @@ Run unit tests and enforce execution-time budgets. Two quality gates:
- **Total suite time** must not exceed `--max-seconds` (default: 10s)
- **Per-test time** — no individual test may exceed `--max-single-seconds`
(default: 0.5s, 0 to disable)
(default: 0.5s, 0 to turn off)
Runs `make test-unit` with `PYTEST_ADDOPTS=--durations=0` so pytest emits
per-test timing lines.
@@ -369,7 +450,7 @@ devx tools check-test-isolation --src-dir src/
Pytest plugin options (automatic when devx is installed):
- `--no-test-isolation`disable static analysis and runtime subprocess audit
- `--no-test-isolation`turn off static analysis and runtime subprocess audit
- `--test-isolation-max-loop N` — max iterations per loop (default: 100)
### `devx tools configure-repo`
@@ -414,7 +495,7 @@ devx tools generate-cliff-config --prefix GRM --force # overwrite existing
Options:
- `--prefix <prefix>` — task ID prefix (default: `DEVX_TASK_PREFIX` env var
or `DEVX`)
- `--output <file>` — output file path (default: `cliff.toml`)
- `--output <file>` — output path (default: `cliff.toml`)
- `--force` — overwrite existing file
### `devx tools install-checkmake`
@@ -488,6 +569,56 @@ devx tools pr-rebase # auto-detect PR from current branch
Options (pass after `--`):
- `--pr <N>` — PR number (auto-detected from current branch if omitted)
### `devx tools check-docker-init`
Check that Docker Compose services with healthchecks have `init: true`.
Without `init: true`, CMD-SHELL healthchecks spawn child processes that
become zombies when PID 1 doesn't reap them.
```bash
devx tools check-docker-init
devx tools check-docker-init --path path/to/docker-compose.yml.j2
```
Options:
- `--path <path>` — check a specific file or directory
- `--templates-dir <path>` — override templates directory (default: `ansible/roles/`)
### `devx tools check-ansible-set-fact-to-json`
Check that Ansible `set_fact` tasks don't misuse `| to_json`. Using
`to_json` in `set_fact` converts native Python types to JSON strings,
causing iteration bugs (for example, iterating over characters instead
of list items).
```bash
devx tools check-ansible-set-fact-to-json
devx tools check-ansible-set-fact-to-json --path path/to/playbook.yml
```
Options:
- `--path <path>` — check a specific file or directory
- `--ansible-dir <path>` — override ansible directories (repeatable)
### `devx tools check-alert-rules`
Validate rendered Prometheus alert rules with `promtool check rules`.
Renders a Jinja2 template with test values and validates the output.
Skips (exits 0) if promtool is not on PATH.
```bash
devx tools check-alert-rules \
--template-path ansible/roles/observability/templates
devx tools check-alert-rules \
--template-path ansible/roles/observability/templates \
--var grafana_base_url=https://grafana.example.com
```
Options:
- `--template-path <path>` — path to templates directory (required)
- `--template-name <name>` — template filename (default: `alert-rules.yml.j2`)
- `--var key=value` — template variables (repeatable)
## Molecule Commands
Molecule commands require the `molecule` extra (`pip install devx[molecule]`).
+2 -2
View File
@@ -48,12 +48,12 @@ Add devx to your `pyproject.toml`:
```toml
[project]
dependencies = [
"devx>=0.47.3",
"devx>=0.50.4",
]
[project.optional-dependencies]
dev = [
"devx>=0.47.3",
"devx>=0.50.4",
]
```
+9 -4
View File
@@ -20,6 +20,8 @@ dependencies = [
"python-dotenv==1.2.2",
"click==8.4.2",
"tenacity==9.1.4", # retry logic for GiteaClient/VikunjaClient
"jinja2==3.1.6", # template rendering (devx.utils.jinja, check_alert_rules)
"pyyaml==6.0.3", # YAML parsing (workflow checks, ansible checks)
]
[project.scripts]
@@ -62,13 +64,16 @@ molecule = [
"ansible-core==2.21.1",
]
# Deploy tools (for infra staging/production deployments)
# Versions aligned with infra's pyproject.toml to avoid reinstalls on every CI job.
# bcrypt and PyJWT are infra deps not in devx core — included here so the CI
# image has them and setup-image can use --no-deps (skip dep resolution).
deploy = [
"ansible-core==2.21.1",
"boto3==1.43.37",
"boto3==1.43.44",
"docker==7.1.0",
"jinja2==3.1.6",
"pyyaml==6.0.3",
"cryptography==49.0.0",
"cryptography==50.0.0",
"bcrypt==5.0.0",
"PyJWT==2.13.0",
]
# Full dev environment (local development)
dev = [
+1 -1
View File
@@ -1,3 +1,3 @@
"""devx — reusable development and CI/CD tools for oblachno-oss projects."""
__version__ = "0.47.3"
__version__ = "0.50.4"
+185
View File
@@ -0,0 +1,185 @@
"""Cancel superseded CI runs for the same PR.
When a new push to a PR branch triggers a new CI run, any in-flight
runs for the same PR are wasting runner time. This script cancels
all but the latest running CI run for each PR branch.
Uses the Gitea Actions API:
GET /repos/{owner}/{repo}/actions/runs?status=in_progress&event=pull_request
POST /repos/{owner}/{repo}/actions/runs/{run_id}/cancel
Usage::
# CI (cancels superseded runs for the current PR):
python -m devx.ci.cancel_superseded_runs \\
--repo "$REPOSITORY" \\
--current-run-id "$GITHUB_RUN_ID" \\
--head-branch "$HEAD_REF"
# Dry-run (lists what would be cancelled without cancelling):
python -m devx.ci.cancel_superseded_runs \\
--repo "$REPOSITORY" \\
--current-run-id "$GITHUB_RUN_ID" \\
--head-branch "$HEAD_REF" \\
--dry-run
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.error
import urllib.request
_HTTP_NO_CONTENT = 204
_HTTP_NOT_FOUND = 404
_HTTP_BAD_REQUEST = 400
_PAGE_SIZE = 50
def _log(msg: str) -> None:
"""Log to stderr."""
print(f"[cancel-superseded] {msg}", file=sys.stderr, flush=True)
def _api_request(
method: str,
path: str,
token: str,
base_url: str,
body: dict | None = None,
) -> dict | list:
"""Make a Gitea API request."""
url = f"{base_url}/api/v1{path}"
headers = {
"Authorization": f"token {token}",
"Content-Type": "application/json",
"Accept": "application/json",
}
data = json.dumps(body).encode() if body else None
req = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, timeout=30) as resp: # nosec B310 — authenticated API request to known Gitea instance
if resp.status == _HTTP_NO_CONTENT:
return {}
return json.loads(resp.read().decode())
except urllib.error.HTTPError as e:
_log(f"API error {e.code} on {method} {path}: {e.read().decode()[:200]}")
raise
except urllib.error.URLError as e:
_log(f"URL error on {method} {path}: {e}")
raise
def list_running_runs(repo: str, token: str, base_url: str) -> list[dict]:
"""List all running CI runs for pull_request events."""
runs: list[dict] = []
page = 1
while True:
result = _api_request(
"GET",
f"/repos/{repo}/actions/runs?status=in_progress&event=pull_request&page={page}&limit=50",
token,
base_url,
)
# Gitea returns {"workflow_runs": [...], "total_count": N}
page_runs = result["workflow_runs"] if isinstance(result, dict) else result
if not page_runs:
break
runs.extend(page_runs)
if len(page_runs) < _PAGE_SIZE:
break
page += 1
return runs
def cancel_run(repo: str, run_id: int, token: str, base_url: str) -> bool:
"""Cancel a CI run. Returns True on success."""
try:
_api_request(
"POST",
f"/repos/{repo}/actions/runs/{run_id}/cancel",
token,
base_url,
)
except (urllib.error.HTTPError, urllib.error.URLError):
return False
return True
def main() -> int:
parser = argparse.ArgumentParser(description="Cancel superseded CI runs for the same PR.")
parser.add_argument("--repo", required=True, help="owner/repo")
parser.add_argument("--current-run-id", required=True, help="Current run ID (not cancelled)")
parser.add_argument("--head-branch", required=True, help="PR head branch name")
parser.add_argument("--dry-run", action="store_true", help="List without cancelling")
parser.add_argument(
"--base-url",
default=os.environ.get("GITEA_API_URL", "https://git.oblachno.oblachno.fyi"),
help="Gitea base URL",
)
args = parser.parse_args()
token = os.environ.get("CI_GITEA_API_TOKEN") or os.environ.get("CI_GITEA_TOKEN")
if not token:
_log("No CI_GITEA_API_TOKEN or CI_GITEA_TOKEN set — skipping")
return 0
current_run_id = int(args.current_run_id)
_log(f"Listing running PR runs for {args.repo}...")
try:
runs = list_running_runs(args.repo, token, args.base_url)
except urllib.error.HTTPError as e:
if e.code in (_HTTP_NOT_FOUND, _HTTP_BAD_REQUEST):
_log(
f"Actions runs API not usable (HTTP {e.code}) — "
f"Gitea {args.base_url} may not support this endpoint or status filter. "
f"Skipping cancel-superseded (non-fatal)."
)
return 0
raise
_log(f"Found {len(runs)} running PR runs")
# Group by head_branch — only cancel runs for the SAME branch
# that are older than the current run
same_branch_runs = [
r
for r in runs
if r.get("head_branch") == args.head_branch
and int(r.get("id", 0)) != current_run_id
and int(r.get("id", 0)) < current_run_id
]
if not same_branch_runs:
_log(f"No superseded runs for branch {args.head_branch}")
return 0
_log(f"Found {len(same_branch_runs)} superseded run(s) for branch {args.head_branch}:")
for r in same_branch_runs:
run_id = r.get("id")
created = r.get("created_at", "?")
_log(f" Run #{run_id} (created: {created})")
if args.dry_run:
_log("[dry-run] Would cancel the above runs")
return 0
cancelled = 0
for r in same_branch_runs:
run_id = int(r["id"])
_log(f"Cancelling run #{run_id}...")
if cancel_run(args.repo, run_id, token, args.base_url):
cancelled += 1
_log(f" Cancelled run #{run_id}")
else:
_log(f" Failed to cancel run #{run_id}")
_log(f"Cancelled {cancelled}/{len(same_branch_runs)} superseded runs")
return 0
if __name__ == "__main__": # pragma: no cover
raise SystemExit(main())
+163
View File
@@ -0,0 +1,163 @@
"""Check that workflow jobs downloading artifacts depend on the uploading job.
This prevents the class of bug where a job downloads an artifact produced by
another job but does not declare that job in its ``needs`` list. When both
jobs run in parallel, the download fails because the artifact hasn't been
uploaded yet.
The check scans all workflow YAML files for:
- ``gitea-upload-artifact`` / ``actions/upload-artifact`` steps
- ``gitea-download-artifact`` / ``actions/download-artifact`` steps
For each download, it finds the job(s) that upload an artifact with a
matching name and verifies that at least one uploading job is in the
downloading job's ``needs`` list.
Artifact names with ``${{ ... }}`` expressions are matched literally
(both sides use the same expression, so they resolve to the same value
at runtime).
Usage::
python -m devx.ci.check_workflow_artifact_deps
python -m devx.ci.check_workflow_artifact_deps --workflow .gitea/workflows/ci.yml
Exit code 0 if all artifact dependencies are satisfied, 1 otherwise.
"""
from __future__ import annotations
import sys
from pathlib import Path
import click
import yaml
REPO_ROOT = Path.cwd()
WORKFLOWS_DIR = REPO_ROOT / ".gitea" / "workflows"
UPLOAD_ACTIONS = ("upload-artifact",)
DOWNLOAD_ACTIONS = ("download-artifact",)
def _is_artifact_action(uses: str, action_types: tuple[str, ...]) -> bool:
"""Check if a step's ``uses`` field references an artifact action."""
if not uses:
return False
uses_lower = uses.lower()
return any(action in uses_lower for action in action_types)
def _extract_artifact_info(workflow: dict) -> tuple[dict[str, list[str]], list[tuple[str, str, str]]]:
"""Extract artifact upload and download info from a workflow.
Returns:
uploads: Mapping of artifact_name list of job names that upload it.
downloads: List of (job_name, artifact_name, step_name) tuples.
"""
uploads: dict[str, list[str]] = {}
downloads: list[tuple[str, str, str]] = []
jobs = workflow.get("jobs", {})
for job_name, job_def in jobs.items():
for step in job_def.get("steps", []):
uses = step.get("uses", "")
with_data = step.get("with", {})
artifact_name = with_data.get("name", "")
step_name = step.get("name", "")
if _is_artifact_action(uses, UPLOAD_ACTIONS):
if artifact_name:
uploads.setdefault(artifact_name, []).append(job_name)
elif _is_artifact_action(uses, DOWNLOAD_ACTIONS) and artifact_name:
downloads.append((job_name, artifact_name, step_name))
return uploads, downloads
def _check_workflow(filepath: Path) -> list[str]:
"""Check a single workflow file for missing artifact dependencies.
Returns a list of error messages (empty if all OK).
"""
errors: list[str] = []
content = filepath.read_text(encoding="utf-8")
try:
workflow = yaml.safe_load(content)
except yaml.YAMLError as exc:
return [f"{filepath}: cannot parse YAML: {exc}"]
if not isinstance(workflow, dict):
return [f"{filepath}: not a valid workflow (expected dict)"]
uploads, downloads = _extract_artifact_info(workflow)
jobs = workflow.get("jobs", {})
for dl_job, artifact_name, step_name in downloads:
uploading_jobs = uploads.get(artifact_name, [])
if not uploading_jobs:
# Artifact not uploaded in this workflow — may come from an
# external source (e.g., S3). Skip.
continue
dl_job_def = jobs.get(dl_job, {})
needs_raw = dl_job_def.get("needs", [])
needs = {needs_raw} if isinstance(needs_raw, str) else set(needs_raw or [])
# Check if any uploading job is in the download job's needs
if not any(uploader in needs for uploader in uploading_jobs):
# Check if the download step has continue-on-error: true
# (valid guard when the uploading job may be skipped due to
# Gitea Actions' needs skip behavior — the download will
# fail gracefully if the artifact doesn't exist).
dl_steps = dl_job_def.get("steps", [])
step_def = next((s for s in dl_steps if s.get("name", "") == step_name), {})
if step_def.get("continue-on-error") is True:
continue
uploaders_str = ", ".join(sorted(uploading_jobs))
errors.append(
f"{filepath.name}::{dl_job}: step '{step_name}' downloads "
f"artifact '{artifact_name}' produced by job(s) "
f"[{uploaders_str}] but none are in its 'needs' list "
f"(current needs: {sorted(needs) or 'none'}). "
f"Add the uploading job to 'needs' or guard the download "
f"with an if: condition checking the upload job's result."
)
return errors
@click.command()
@click.option(
"--workflow",
type=click.Path(exists=True, path_type=Path),
help="Check a specific workflow file (default: all in .gitea/workflows/).",
)
@click.option(
"--workflows-dir",
type=click.Path(exists=True, path_type=Path),
default=None,
help="Override the workflows directory (default: .gitea/workflows/).",
)
def main(workflow: Path | None, workflows_dir: Path | None) -> None:
"""Check that artifact download jobs depend on upload jobs."""
wdir = workflows_dir or WORKFLOWS_DIR
files = [workflow] if workflow else sorted(wdir.glob("*.yml"))
all_errors: list[str] = []
for f in files:
errors = _check_workflow(f)
all_errors.extend(errors)
if all_errors:
click.echo("[check-workflow-artifact-deps] FAIL: missing artifact dependencies found:")
for err in all_errors:
click.echo(f" - {err}")
sys.exit(1)
else:
click.echo("[check-workflow-artifact-deps] OK: all artifact downloads have upload jobs in needs.")
if __name__ == "__main__": # pragma: no cover
main()
+145
View File
@@ -0,0 +1,145 @@
"""Check that workflow jobs using tofu state have a tofu-init step.
This prevents the class of bug where a job runs ``tofu output`` or calls
a script that uses tofu state without first running ``tofu init``,
causing "Required plugins are not installed" errors.
The check scans all workflow YAML files for jobs that:
- Call scripts that use ``tofu output`` (configurable via --state-scripts)
- Call ``tofu output`` directly
- Call ``tofu plan`` or ``tofu apply`` directly
For each such job, it verifies the same job has a ``tofu-init`` step,
either:
- Directly via ``tofu init`` in a step's run command
- Via ``create_staging_deployment.py --phase tofu-init``
- Via ``create_production_deployment.py --phase tofu-init``
Usage::
python -m devx.ci.check_workflow_tofu_init
python -m devx.ci.check_workflow_tofu_init --workflow .gitea/workflows/deploy.yml
Exit code 0 if all jobs have tofu-init, 1 otherwise.
"""
from __future__ import annotations
import sys
from pathlib import Path
import click
import yaml
REPO_ROOT = Path.cwd()
WORKFLOWS_DIR = REPO_ROOT / ".gitea" / "workflows"
# Scripts that call `tofu output`, `tofu plan`, or `tofu apply` internally.
# If a job calls any of these, it must have a tofu-init step.
# NOTE: destroy_orphans.py reads terraform.tfstate directly from disk
# (does not invoke `tofu output`), so it does NOT need tofu-init.
DEFAULT_TOFU_STATE_SCRIPTS: set[str] = {
"preflight_deploy.py",
}
# Commands that directly use tofu state (must be preceded by tofu init).
TOFU_STATE_COMMANDS = ("tofu output", "tofu plan", "tofu apply", "tofu show")
# Commands that initialize tofu (counted as tofu-init steps).
TOFU_INIT_COMMANDS = (
"tofu init",
"--phase tofu-init",
"tofu-init",
)
def _check_workflow(filepath: Path, state_scripts: set[str]) -> list[str]:
"""Check a single workflow file for missing tofu-init steps.
Returns a list of error messages (empty if all OK).
"""
errors: list[str] = []
content = filepath.read_text(encoding="utf-8")
try:
workflow = yaml.safe_load(content)
except yaml.YAMLError as exc:
return [f"{filepath}: cannot parse YAML: {exc}"]
jobs = workflow.get("jobs", {})
for job_name, job_def in jobs.items():
steps = job_def.get("steps", [])
if not steps:
continue
uses_tofu_state = False
has_tofu_init = False
for step in steps:
run_cmd = step.get("run", "")
if not run_cmd:
continue
# Check if this step uses tofu state
for script in state_scripts:
if script in run_cmd:
uses_tofu_state = True
for cmd in TOFU_STATE_COMMANDS:
if cmd in run_cmd:
uses_tofu_state = True
# Check if this step initializes tofu
for cmd in TOFU_INIT_COMMANDS:
if cmd in run_cmd:
has_tofu_init = True
if uses_tofu_state and not has_tofu_init:
errors.append(
f"{filepath.name}::{job_name}: uses tofu state "
f"(tofu output/plan/apply or {state_scripts}) "
f"but has no tofu-init step. Add a step running "
f"'create_*_deployment.py --phase tofu-init' before "
f"the first tofu state access."
)
return errors
@click.command()
@click.option(
"--workflow",
type=click.Path(exists=True, path_type=Path),
help="Check a specific workflow file (default: all in .gitea/workflows/).",
)
@click.option(
"--workflows-dir",
type=click.Path(exists=True, path_type=Path),
default=None,
help="Override the workflows directory (default: .gitea/workflows/).",
)
@click.option(
"--state-script",
"state_scripts",
multiple=True,
default=None,
help="Add a script name that uses tofu state (can be repeated). Overrides the default list if any are specified.",
)
def main(workflow: Path | None, workflows_dir: Path | None, state_scripts: tuple[str, ...]) -> None:
"""Check that workflow jobs using tofu state have a tofu-init step."""
scripts = set(state_scripts) if state_scripts else DEFAULT_TOFU_STATE_SCRIPTS
wdir = workflows_dir or WORKFLOWS_DIR
files = [workflow] if workflow else sorted(wdir.glob("*.yml"))
all_errors: list[str] = []
for f in files:
errors = _check_workflow(f, scripts)
all_errors.extend(errors)
if all_errors:
click.echo("[check-workflow-tofu-init] FAIL: missing tofu-init steps found:")
for err in all_errors:
click.echo(f" - {err}")
sys.exit(1)
else:
click.echo("[check-workflow-tofu-init] OK: all tofu-state jobs have tofu-init.")
if __name__ == "__main__": # pragma: no cover
main()
+28 -173
View File
@@ -1,19 +1,18 @@
#!/usr/bin/env python3
"""Discover available Gitea Actions runners for dynamic job distribution.
Queries the Gitea API for registered runners at three levels:
1. Repository level: GET /repos/{owner}/{repo}/actions/runners
2. Organization level: GET /orgs/{org}/actions/runners
3. Instance (admin) level: GET /admin/actions/runners
.. deprecated:: Phase 1c
Use :mod:`devx.molecule.discover_runners` instead. This module is a
thin wrapper that re-exports the canonical implementation from
:mod:`devx.molecule.discover_runners` for backward compatibility
with existing workflow references and Makefile targets.
Falls back to the ``MOLECULE_RUNNERS`` repo variable or environment
variable, then to ``DEFAULT_MAX_RUNNERS`` (3).
Outputs:
- ``--count``: prints the number of available runners
- ``--indices``: prints a JSON array [0, 1, ..., N-1] for use as a
dynamic matrix in Gitea Actions
- (default): prints both as ``count=N`` and ``indices=[0,1,...]``
The canonical implementation lives in
:mod:`devx.molecule.discover_runners` because runner discovery is
primarily used by the molecule test distribution pipeline. CI
workflows that still reference ``python -m devx.ci.discover_runners``
will continue to work via this wrapper, but new code should import
from :mod:`devx.molecule.discover_runners` directly.
Usage:
python3 -m devx.ci.discover_runners --owner oblachno-oss --repo devx
@@ -23,172 +22,28 @@ Usage:
from __future__ import annotations
import json
import os
import sys
import warnings
import click
import requests
from devx.config import GITEA_API_URL, REPO_NAME, REPO_OWNER
from devx.i18n import _
from devx.tokens import get_ci_token
DEFAULT_MAX_RUNNERS = 3
def query_runners(api_url: str, token: str, owner: str, repo: str) -> int:
"""Query the Gitea API for registered runners at all levels.
Returns the total count of active runners. If the API call fails
(e.g., no admin access for instance-level runners), falls back to
what we can see. Fallbacks are logged to stderr for debugging.
"""
headers = {"Authorization": f"token {token}"}
total = 0
# 1. Repository-level runners
try:
r = requests.get(
f"{api_url}/repos/{owner}/{repo}/actions/runners",
headers=headers,
timeout=10,
)
if r.status_code == 200:
data = r.json()
total += data.get("total_count", 0)
else:
click.echo(_("Warning: repo-level runners query returned HTTP {status}", status=r.status_code), err=True)
except (requests.RequestException, ValueError) as e:
click.echo(_("Warning: repo-level runners query failed: {error}", error=e), err=True)
# 2. Organization-level runners
try:
r = requests.get(
f"{api_url}/orgs/{owner}/actions/runners",
headers=headers,
timeout=10,
)
if r.status_code == 200:
data = r.json()
total += data.get("total_count", 0)
else:
click.echo(_("Warning: org-level runners query returned HTTP {status}", status=r.status_code), err=True)
except (requests.RequestException, ValueError) as e:
click.echo(_("Warning: org-level runners query failed: {error}", error=e), err=True)
# 3. Instance-level runners (requires admin scope)
try:
r = requests.get(
f"{api_url}/admin/actions/runners",
headers=headers,
timeout=10,
)
if r.status_code == 200:
data = r.json()
total += data.get("total_count", 0)
elif r.status_code != 403: # 403 is expected without admin scope
click.echo(
_("Warning: instance-level runners query returned HTTP {status}", status=r.status_code),
err=True,
)
except (requests.RequestException, ValueError) as e:
click.echo(_("Warning: instance-level runners query failed: {error}", error=e), err=True)
return total
def get_runner_count(api_url: str, token: str | None, owner: str, repo: str) -> int:
"""Determine the number of available runners.
Tries the Gitea API first, then falls back to env vars, then default.
"""
# Try API query if we have a token
if token:
api_count = query_runners(api_url, token, owner, repo)
if api_count > 0:
return api_count
# Fall back to MOLECULE_RUNNERS env var (set by CI from repo variable)
env_count = os.environ.get("MOLECULE_RUNNERS")
if env_count:
try:
count = int(env_count)
if count > 0:
return count
except ValueError:
pass
# Fall back to default
return DEFAULT_MAX_RUNNERS
def generate_indices(count: int) -> list[str]:
"""Generate a list of runner indices ["1", "2", ..., "N"].
Uses 1-based string indices because Gitea Actions renders
integer 0 and string "0" as empty in ${{ matrix.runner-index }}
expressions, causing --runner-index to be passed without a value.
The distribute_molecule.py script converts these back to 0-based
internally.
"""
return [str(i + 1) for i in range(count)]
@click.command()
@click.option("--owner", default=None, help="Repository owner (for API query).")
@click.option("--repo", default=None, help="Repository name (for API query).")
@click.option("--count", "output_count", is_flag=True, help="Output only the count.")
@click.option("--indices", "output_indices", is_flag=True, help="Output only the JSON indices array.")
@click.option(
"--github-output",
"github_output",
is_flag=True,
default=False,
help="Write results to $GITHUB_OUTPUT file (for CI workflow steps).",
from devx.molecule.discover_runners import ( # noqa: F401 — re-exported for backward compat
DEFAULT_MAX_RUNNERS,
generate_indices,
get_runner_count,
main,
query_runners,
)
def main(
owner: str | None,
repo: str | None,
output_count: bool,
output_indices: bool,
github_output: bool,
) -> None:
try:
token = get_ci_token()
except click.ClickException:
token = None
if owner is None:
owner = os.environ.get("DEVX_REPO_OWNER", "") or REPO_OWNER
if repo is None:
repo = os.environ.get("DEVX_REPO_NAME", "") or REPO_NAME
_DEPRECATION_MSG = (
"devx.ci.discover_runners is deprecated; use devx.molecule.discover_runners instead. "
"This wrapper will be removed in a future release."
)
count = get_runner_count(GITEA_API_URL, token, owner, repo)
indices = generate_indices(count)
if github_output:
gh_output = os.environ.get("GITHUB_OUTPUT")
if not gh_output:
raise click.ClickException("GITHUB_OUTPUT environment variable is not set")
with open(gh_output, "a", encoding="utf-8") as f: # noqa: PTH123
f.write(f"runner-count={count}\n")
f.write(f"runner-indices={json.dumps(indices)}\n")
click.echo(_("Runner count: {count}", count=count))
click.echo(_("Runner indices: {indices}", indices=indices))
return
if output_count:
click.echo(str(count))
return
if output_indices:
click.echo(json.dumps(indices))
return
# Default: output both as key=value pairs for CI consumption
click.echo(_("count={count}", count=count))
click.echo(_("indices={indices}", indices=json.dumps(indices)))
def _emit_deprecation_warning() -> None:
"""Emit a DeprecationWarning when this module is imported for CLI use."""
warnings.warn(_DEPRECATION_MSG, DeprecationWarning, stacklevel=2)
if __name__ == "__main__": # pragma: no cover
main()
_emit_deprecation_warning()
sys.exit(main())
+209
View File
@@ -0,0 +1,209 @@
#!/usr/bin/env python3
"""Wait for Gitea Actions jobs to complete.
Polls the Gitea API for job completion status. Used by auto-merge
jobs that need to wait for molecule-tests or other parallel jobs
before proceeding.
Exits:
0 all matching jobs completed successfully
1 one or more matching jobs failed
2 timeout reached before all jobs completed
3 API error or job not found
Usage:
python3 -m devx.ci.wait_for_checks \\
--job-name molecule-tests \\
--repo oblachno-oss/grm \\
--timeout 1200 \\
--poll-interval 10
"""
from __future__ import annotations
import os
import sys
import time
import click
import requests
from devx.config import GITEA_API_URL
from devx.i18n import _
from devx.tokens import get_ci_token
def query_job_status(api_url: str, token: str, repo: str, job_name_prefix: str) -> list[dict]:
"""Query the Gitea API for the status of jobs matching *job_name_prefix*.
Fetches the most recent pull_request runs (up to 3) and inspects
their jobs. Returns a list of ``{"name": str, "status": str,
"conclusion": str | None}`` dicts for jobs whose name starts with
*job_name_prefix*. On API errors, logs a warning to stderr and
returns an empty list callers treat this as "no information yet"
and retry on the next poll.
"""
headers = {"Authorization": f"token {token}"}
matches: list[dict] = []
try:
r = requests.get(
f"{api_url}/repos/{repo}/actions/runs",
headers=headers,
params={"limit": 5, "event": "pull_request"},
timeout=10,
)
if r.status_code != 200:
click.echo(
_("Warning: actions runs query returned HTTP {status}", status=r.status_code),
err=True,
)
return []
runs = r.json()
if isinstance(runs, dict):
runs = runs.get("runs", [])
except (requests.RequestException, ValueError) as e:
click.echo(_("Warning: actions runs query failed: {error}", error=e), err=True)
return []
for run in runs[:3]:
run_id = run.get("id")
if run_id is None:
continue
try:
jr = requests.get(
f"{api_url}/repos/{repo}/actions/runs/{run_id}/jobs",
headers=headers,
timeout=10,
)
if jr.status_code != 200:
click.echo(
_(
"Warning: jobs query for run {run_id} returned HTTP {status}",
run_id=run_id,
status=jr.status_code,
),
err=True,
)
continue
jobs = jr.json()
if isinstance(jobs, dict):
jobs = jobs.get("jobs", [])
except (requests.RequestException, ValueError) as e:
click.echo(
_("Warning: jobs query for run {run_id} failed: {error}", run_id=run_id, error=e),
err=True,
)
continue
for job in jobs:
name = job.get("name", "")
if name.startswith(job_name_prefix):
matches.append(
{
"name": name,
"status": job.get("status", "unknown"),
"conclusion": job.get("conclusion"),
}
)
return matches
def poll_until_complete(
api_url: str,
token: str,
repo: str,
job_name: str,
timeout: int,
interval: int,
require_success: bool = True,
) -> int:
"""Poll *query_job_status* until all matching jobs complete or *timeout*.
Returns:
0 all matching jobs completed successfully (or any completed, when
*require_success* is False)
1 at least one matching job completed with a non-success conclusion
(only when *require_success* is True)
2 *timeout* reached before all matching jobs completed
3 no matching jobs found at all within *timeout*
"""
deadline = time.monotonic() + timeout
found_any = False
while time.monotonic() < deadline:
jobs = query_job_status(api_url, token, repo, job_name)
if jobs:
found_any = True
all_completed = all(j["status"] == "completed" for j in jobs)
if all_completed:
if require_success and any(j["conclusion"] != "success" for j in jobs):
click.echo(
_("Job(s) completed with non-success conclusion: {jobs}", jobs=jobs),
err=True,
)
return 1
click.echo(_("All matching jobs completed successfully: {jobs}", jobs=jobs))
return 0
# Not all completed (or no jobs yet) — sleep and retry.
time.sleep(min(interval, max(0, deadline - time.monotonic())))
if not found_any:
click.echo(_("No matching jobs found for prefix '{prefix}' within timeout.", prefix=job_name), err=True)
return 3
click.echo(_("Timeout reached waiting for jobs matching '{prefix}'.", prefix=job_name), err=True)
return 2
@click.command()
@click.option("--job-name", required=True, help="Job name prefix to match (e.g. 'molecule-tests').")
@click.option(
"--repo",
default=None,
help="Repository as owner/name (default: $GITHUB_REPOSITORY env var).",
)
@click.option("--timeout", type=int, default=1200, help="Max seconds to wait (default: 1200 = 20 min).")
@click.option("--poll-interval", "interval", type=int, default=10, help="Seconds between polls (default: 10).")
@click.option(
"--require-success/--no-require-success",
default=True,
help="Exit 1 if a matched job failed (default: yes).",
)
def main(job_name: str, repo: str | None, timeout: int, interval: int, require_success: bool) -> None:
"""Wait for Gitea Actions jobs matching --job-name to complete."""
if repo is None:
repo = os.environ.get("GITHUB_REPOSITORY", "")
if not repo or "/" not in repo:
raise click.ClickException(_("--repo is required (or set GITHUB_REPOSITORY=owner/name)"))
if timeout <= 0:
raise click.ClickException(_("--timeout must be positive"))
if interval <= 0:
raise click.ClickException(_("--poll-interval must be positive"))
try:
token = get_ci_token()
except click.ClickException as e:
click.echo(str(e), err=True)
sys.exit(3)
click.echo(
_(
"Waiting for jobs matching '{prefix}' in {repo} (timeout={timeout}s, interval={interval}s)",
prefix=job_name,
repo=repo,
timeout=timeout,
interval=interval,
)
)
code = poll_until_complete(
GITEA_API_URL,
token,
repo,
job_name,
timeout,
interval,
require_success=require_success,
)
sys.exit(code)
if __name__ == "__main__": # pragma: no cover
main()
+49
View File
@@ -158,6 +158,13 @@ def ci_validate_commit_msg(args: tuple[str, ...]) -> None:
_run_module("devx.ci.validate_commit_msg", list(args))
@ci.command("wait-for-checks")
@click.argument("args", nargs=-1)
def ci_wait_for_checks(args: tuple[str, ...]) -> None:
"""Wait for Gitea Actions jobs to complete (polls API)."""
_run_module("devx.ci.wait_for_checks", list(args))
@ci.command("distribute-files")
@click.argument("args", nargs=-1)
def ci_distribute_files(args: tuple[str, ...]) -> None:
@@ -172,6 +179,27 @@ def ci_integration_guard(args: tuple[str, ...]) -> None:
_run_module("devx.ci.integration_guard", list(args))
@ci.command("cancel-superseded-runs")
@click.argument("args", nargs=-1)
def ci_cancel_superseded_runs(args: tuple[str, ...]) -> None:
"""Cancel superseded CI runs for the same PR branch."""
_run_module("devx.ci.cancel_superseded_runs", list(args))
@ci.command("check-workflow-artifact-deps")
@click.argument("args", nargs=-1)
def ci_check_workflow_artifact_deps(args: tuple[str, ...]) -> None:
"""Check that artifact download jobs depend on upload jobs."""
_run_module("devx.ci.check_workflow_artifact_deps", list(args))
@ci.command("check-workflow-tofu-init")
@click.argument("args", nargs=-1)
def ci_check_workflow_tofu_init(args: tuple[str, ...]) -> None:
"""Check that workflow jobs using tofu state have a tofu-init step."""
_run_module("devx.ci.check_workflow_tofu_init", list(args))
@cli.group()
def tools() -> None:
"""Development tool commands."""
@@ -240,6 +268,27 @@ def tools_pr_rebase(args: tuple[str, ...]) -> None:
_run_module("devx.tools.pr_rebase", list(args))
@tools.command("check-docker-init")
@click.argument("args", nargs=-1)
def tools_check_docker_init(args: tuple[str, ...]) -> None:
"""Check that Docker Compose services with healthchecks have init: true."""
_run_module("devx.tools.check_docker_init", list(args))
@tools.command("check-ansible-set-fact-to-json")
@click.argument("args", nargs=-1)
def tools_check_ansible_set_fact_to_json(args: tuple[str, ...]) -> None:
"""Check that Ansible set_fact tasks don't misuse to_json."""
_run_module("devx.tools.check_ansible_set_fact_to_json", list(args))
@tools.command("check-alert-rules")
@click.argument("args", nargs=-1)
def tools_check_alert_rules(args: tuple[str, ...]) -> None:
"""Validate rendered Prometheus alert rules with promtool."""
_run_module("devx.tools.check_alert_rules", list(args))
@cli.group()
def molecule() -> None:
"""Molecule testing commands (requires devx[molecule])."""
+34 -5
View File
@@ -6,6 +6,10 @@ Supported: en, bg, de, ru, zh, pl.
Projects can extend translations by setting DEVX_TRANSLATIONS_PATH to a
JSON file with additional keys. Keys from the project's file are merged
on top of devx's built-in translations.
Projects that use different env var names (e.g. GRM_LANG instead of
DEVX_LANG) can call :func:`configure_i18n` at import time to override
the defaults.
"""
from __future__ import annotations
@@ -14,15 +18,39 @@ import json
import os
from pathlib import Path
# Configurable env var names — projects can override via configure_i18n()
_lang_env_var = "DEVX_LANG"
_translations_path_env_var = "DEVX_TRANSLATIONS_PATH"
# Load built-in translations
_BUILTIN_TRANSLATIONS: dict[str, dict[str, str]] = json.loads(
(Path(__file__).parent / "translations.json").read_text(encoding="utf-8")
)
def configure_i18n(
*,
lang_env_var: str = "DEVX_LANG",
translations_path_env_var: str = "DEVX_TRANSLATIONS_PATH",
) -> None:
"""Override the env var names used for language and translations path.
This allows downstream projects (e.g. grm) to use their own env var
names (e.g. ``GRM_LANG``) while still using devx's i18n system.
Args:
lang_env_var: Environment variable name for language selection.
translations_path_env_var: Environment variable name for the
path to a JSON file with project-specific translations.
"""
global _lang_env_var, _translations_path_env_var
_lang_env_var = lang_env_var
_translations_path_env_var = translations_path_env_var
def _load_project_translations() -> dict[str, dict[str, str]]:
"""Load project-specific translations from DEVX_TRANSLATIONS_PATH if set."""
path = os.getenv("DEVX_TRANSLATIONS_PATH")
"""Load project-specific translations from the configured env var if set."""
path = os.getenv(_translations_path_env_var)
if not path:
return {}
p = Path(path)
@@ -41,10 +69,11 @@ TRANSLATIONS: dict[str, dict[str, str]] = {**_BUILTIN_TRANSLATIONS, **_load_proj
def _(key: str, **kwargs: object) -> str:
"""Return a translated string for the given key.
Translation is opt-in via the ``DEVX_LANG`` environment variable.
If unset, English is always returned regardless of system locale.
Translation is opt-in via the configured language environment variable
(default ``DEVX_LANG``). If unset, English is always returned regardless
of system locale.
"""
lang = os.getenv("DEVX_LANG", "en")
lang = os.getenv(_lang_env_var, "en")
if lang not in ("en", "bg", "de", "ru", "zh", "pl"):
lang = "en"
template = TRANSLATIONS.get(key, {}).get(lang, key)
+21 -11
View File
@@ -30,6 +30,7 @@ import click
import requests
from devx.config import GITEA_API_URL, REPO_NAME, REPO_OWNER
from devx.i18n import _
from devx.tokens import get_ci_token
DEFAULT_MAX_RUNNERS = 3
@@ -40,7 +41,7 @@ def query_runners(api_url: str, token: str, owner: str, repo: str) -> int:
Returns the total count of active runners. If the API call fails
(e.g., no admin access for instance-level runners), falls back to
what we can see.
what we can see. Fallbacks are logged to stderr for debugging.
"""
headers = {"Authorization": f"token {token}"}
total = 0
@@ -55,8 +56,10 @@ def query_runners(api_url: str, token: str, owner: str, repo: str) -> int:
if r.status_code == 200:
data = r.json()
total += data.get("total_count", 0)
except (requests.RequestException, ValueError):
pass
else:
click.echo(_("Warning: repo-level runners query returned HTTP {status}", status=r.status_code), err=True)
except (requests.RequestException, ValueError) as e:
click.echo(_("Warning: repo-level runners query failed: {error}", error=e), err=True)
# 2. Organization-level runners
try:
@@ -68,8 +71,10 @@ def query_runners(api_url: str, token: str, owner: str, repo: str) -> int:
if r.status_code == 200:
data = r.json()
total += data.get("total_count", 0)
except (requests.RequestException, ValueError):
pass
else:
click.echo(_("Warning: org-level runners query returned HTTP {status}", status=r.status_code), err=True)
except (requests.RequestException, ValueError) as e:
click.echo(_("Warning: org-level runners query failed: {error}", error=e), err=True)
# 3. Instance-level runners (requires admin scope)
try:
@@ -81,8 +86,13 @@ def query_runners(api_url: str, token: str, owner: str, repo: str) -> int:
if r.status_code == 200:
data = r.json()
total += data.get("total_count", 0)
except (requests.RequestException, ValueError):
pass
elif r.status_code != 403: # 403 is expected without admin scope
click.echo(
_("Warning: instance-level runners query returned HTTP {status}", status=r.status_code),
err=True,
)
except (requests.RequestException, ValueError) as e:
click.echo(_("Warning: instance-level runners query failed: {error}", error=e), err=True)
return total
@@ -163,8 +173,8 @@ def main(
with open(gh_output, "a", encoding="utf-8") as f: # noqa: PTH123
f.write(f"runner-count={count}\n")
f.write(f"runner-indices={json.dumps(indices)}\n")
click.echo(f"Runner count: {count}")
click.echo(f"Runner indices: {indices}")
click.echo(_("Runner count: {count}", count=count))
click.echo(_("Runner indices: {indices}", indices=indices))
return
if output_count:
@@ -176,8 +186,8 @@ def main(
return
# Default: output both as key=value pairs for CI consumption
click.echo(f"count={count}")
click.echo(f"indices={json.dumps(indices)}")
click.echo(_("count={count}", count=count))
click.echo(_("indices={indices}", indices=json.dumps(indices)))
if __name__ == "__main__": # pragma: no cover
+42 -2
View File
@@ -104,21 +104,36 @@ def discover_scenarios(root: Path | None = None) -> list[str]:
return sorted(scenarios)
def discover_multi_role_scenarios(roles_root: Path | None = None) -> list[tuple[str, str]]:
def discover_multi_role_scenarios(
roles_root: Path | None = None,
include_roles: list[str] | None = None,
exclude_roles: list[str] | None = None,
) -> list[tuple[str, str]]:
"""Discover (role, scenario) pairs across all roles under *roles_root*.
Scans ``roles_root/*/molecule/*/`` for scenario directories, skipping
``common`` and directories starting with ``_``. Returns a sorted list of
``(role_name, scenario_name)`` tuples.
If *include_roles* is given, only roles whose name is in the list are
returned. If *exclude_roles* is given, roles whose name is in the list
are skipped. Both filters are case-insensitive.
"""
if roles_root is None:
roles_root = DEFAULT_ROLES_ROOT
if not roles_root.is_dir():
raise click.ClickException(_("Roles directory not found: {path}", path=str(roles_root)))
include_set = {r.lower() for r in include_roles} if include_roles else None
exclude_set = {r.lower() for r in exclude_roles} if exclude_roles else None
pairs: list[tuple[str, str]] = []
for role_dir in sorted(roles_root.iterdir()):
if not role_dir.is_dir():
continue
role_name = role_dir.name
if include_set is not None and role_name.lower() not in include_set:
continue
if exclude_set is not None and role_name.lower() in exclude_set:
continue
mol_dir = role_dir / "molecule"
if not mol_dir.is_dir():
continue
@@ -348,6 +363,24 @@ def _write_github_env(key: str, value: str) -> None:
help="JSON file with custom platform list (each entry: name, image, command). "
"Overrides the default platform matrix. Useful for projects with custom test images.",
)
@click.option(
"--include-roles",
"include_roles",
type=str,
default=None,
help="Comma-separated list of role names to include (multi-role mode only). "
"Only scenarios from these roles are distributed. Case-insensitive. "
"Example: --include-roles docker_base,crowdsec,disk_cleanup,app_hardening",
)
@click.option(
"--exclude-roles",
"exclude_roles",
type=str,
default=None,
help="Comma-separated list of role names to exclude (multi-role mode only). "
"Scenarios from these roles are skipped. Case-insensitive. "
"Example: --exclude-roles docker_base,crowdsec,disk_cleanup,app_hardening",
)
def cli(
runner_index: int | None,
max_runners: int,
@@ -358,11 +391,18 @@ def cli(
molecule_root: Path | None,
roles_root: Path | None,
platforms_file: Path | None,
include_roles: str | None,
exclude_roles: str | None,
) -> None:
platforms = load_platforms(platforms_file)
# Parse role filters
include_list = [r.strip() for r in include_roles.split(",")] if include_roles else None
exclude_list = [r.strip() for r in exclude_roles.split(",")] if exclude_roles else None
# Multi-role mode: discover (role, scenario) pairs across all roles
if roles_root is not None:
role_scenarios = discover_multi_role_scenarios(roles_root)
role_scenarios = discover_multi_role_scenarios(
roles_root, include_roles=include_list, exclude_roles=exclude_list
)
if list_all:
for role, scenario in role_scenarios:
click.echo(f"{role}|{scenario}")
+158
View File
@@ -0,0 +1,158 @@
"""Detect which Ansible roles changed and output their molecule scenarios.
Usage::
python -m devx.molecule.molecule_changed --print-targets
python -m devx.molecule.molecule_changed --base origin/master --print-roles
Outputs the list of make targets (e.g. molecule-docker-base) for roles
that have changed files vs the base ref. Used by ``make molecule-changed``
to run only the molecule scenarios affected by the current diff.
Role-to-target mapping is derived from the directory structure:
ansible/roles/<role>/ molecule-<role>
For roles with multiple scenarios (e.g. app_container has customer-apps,
nextcloud, postgres-upgrade, simple-app), the base target runs all
scenarios for that role.
Playbooks that change also trigger molecule for the roles they include.
Shared infrastructure changes (ansible.cfg, requirements.yml, molecule/)
trigger all scenarios.
"""
from __future__ import annotations
import subprocess # nosec B404 — used to run git, a trusted binary
from pathlib import Path
import click
REPO_ROOT = Path.cwd()
# Map role names to make targets.
ROLE_TARGET_MAP: dict[str, str] = {
"app_container": "molecule-app-container",
"app_hardening": "molecule-app-hardening",
"crowdsec": "molecule-crowdsec",
"disk_cleanup": "molecule-disk-cleanup",
"docker_base": "molecule-docker-base",
"observability": "molecule-observability",
"restore": "molecule-restore",
"sso_config": "molecule-sso-config",
"storage": "molecule-storage",
"zitadel": "molecule-zitadel",
}
# Playbooks that map to molecule scenarios (via roles they include).
PLAYBOOK_ROLE_MAP: dict[str, list[str]] = {
"ansible/playbooks/deploy-observability.yml": ["observability", "docker_base", "zitadel", "crowdsec"],
"ansible/playbooks/deploy-customer.yml": ["app_container", "docker_base", "app_hardening", "sso_config"],
"ansible/playbooks/configure-oidc.yml": ["sso_config", "app_container"],
"ansible/playbooks/prepare-vms.yml": ["docker_base", "app_hardening", "storage", "disk_cleanup", "crowdsec"],
}
# Shared infrastructure that affects all molecule tests.
SHARED_PATHS = (
"ansible/ansible.cfg",
"ansible/requirements.yml",
"ansible/molecule/",
)
# Minimum path parts for a role file: ansible/roles/<role> (3 parts).
# Files inside the role have more parts, but we only need the role name.
_MIN_ROLE_PATH_PARTS = 3
def _run_git(args: list[str]) -> str: # pragma: no cover
"""Run a git command and return stdout."""
result = subprocess.run( # nosec
["git", *args],
cwd=REPO_ROOT,
capture_output=True,
text=True,
check=False,
)
return result.stdout
def get_changed_files(base: str) -> list[str]:
"""Get list of changed files vs base ref."""
for ref in [base, "master"]:
output = _run_git(["diff", "--name-only", f"{ref}...HEAD"])
if output.strip():
return sorted(output.strip().splitlines())
return []
def detect_changed_roles(changed_files: list[str]) -> set[str]:
"""Detect which roles have changed files."""
roles: set[str] = set()
for filepath in changed_files:
# Check if file is in a role directory
if filepath.startswith("ansible/roles/"):
parts = filepath.split("/")
if len(parts) >= _MIN_ROLE_PATH_PARTS:
roles.add(parts[2])
# Check if file is a playbook that maps to roles
if filepath in PLAYBOOK_ROLE_MAP:
roles.update(PLAYBOOK_ROLE_MAP[filepath])
# Check shared infrastructure — triggers all roles
for shared in SHARED_PATHS:
if filepath.startswith(shared):
return set(ROLE_TARGET_MAP.keys())
return roles
def roles_to_targets(roles: set[str]) -> list[str]:
"""Convert role names to make targets."""
targets = []
for role in sorted(roles):
target = ROLE_TARGET_MAP.get(role)
if target:
targets.append(target)
return targets
@click.command()
@click.option(
"--base",
default="origin/master",
help="Base ref to compare against (default: origin/master).",
)
@click.option(
"--print-targets",
is_flag=True,
help="Print make targets (e.g. molecule-docker-base).",
)
@click.option(
"--print-roles",
is_flag=True,
help="Print role names (default if no --print-targets).",
)
def main(base: str, print_targets: bool, print_roles: bool) -> None:
"""Detect which Ansible roles changed and output molecule scenarios."""
changed_files = get_changed_files(base)
if not changed_files:
click.echo("No changed files detected.", err=True)
return
roles = detect_changed_roles(changed_files)
if not roles:
click.echo("No molecule scenarios affected by changes.", err=True)
return
if print_targets:
for target in roles_to_targets(roles):
click.echo(target)
else:
for role in sorted(roles):
click.echo(role)
if __name__ == "__main__": # pragma: no cover
main()
+44
View File
@@ -246,18 +246,62 @@ def cli(pairs: tuple[str, ...], roles_root: Path | None) -> None:
with contextlib.suppress(ProcessLookupError):
os.killpg(os.getpgid(process.pid), signal.SIGKILL)
process.wait()
# Clean up containers left behind by the killed test.
click.echo(_("Cleaning up: running molecule destroy for {scenario}", scenario=scenario))
destroy_cmd = ["molecule", "destroy"]
if scenario != "default":
destroy_cmd.extend(["-s", scenario])
with contextlib.suppress(subprocess.SubprocessError, OSError):
subprocess.run( # nosec B603, B607
destroy_cmd,
cwd=str(cwd),
env=env,
check=False,
capture_output=True,
timeout=120,
)
sys.exit(1)
time.sleep(1)
except KeyboardInterrupt:
with contextlib.suppress(ProcessLookupError):
os.killpg(os.getpgid(process.pid), signal.SIGTERM)
process.wait()
# Clean up containers left behind by the interrupted test.
click.echo(_("Cleaning up: running molecule destroy for {scenario}", scenario=scenario))
destroy_cmd = ["molecule", "destroy"]
if scenario != "default":
destroy_cmd.extend(["-s", scenario])
with contextlib.suppress(subprocess.SubprocessError, OSError):
subprocess.run( # nosec B603, B607
destroy_cmd,
cwd=str(cwd),
env=env,
check=False,
capture_output=True,
timeout=120,
)
sys.exit(1)
rc = process.returncode
if rc != 0:
click.echo(_("FAILED: {pair} exited with code {code}", pair=pair, code=rc))
# Run molecule destroy to clean up containers left behind by the
# failed test. Without this, containers stay running and accumulate
# on the runner, consuming disk/memory and degrading CI performance.
click.echo(_("Cleaning up: running molecule destroy for {scenario}", scenario=scenario))
destroy_cmd = ["molecule", "destroy"]
if scenario != "default":
destroy_cmd.extend(["-s", scenario])
with contextlib.suppress(subprocess.SubprocessError, OSError):
subprocess.run( # nosec B603, B607
destroy_cmd,
cwd=str(cwd),
env=env,
check=False,
capture_output=True,
timeout=120,
)
sys.exit(rc)
click.echo(_("PASSED: {pair}", pair=pair))
+34
View File
@@ -0,0 +1,34 @@
"""Ansible check tools — composable validators for Ansible playbooks and roles.
Each check module exports a ``check_*`` function that returns a list of
violation strings. The shared utilities in :mod:`devx.tools.ansible_checks._shared`
handle file discovery, YAML parsing, and violation reporting.
The old entry points (``devx.tools.check_ansible_*``, ``devx.tools.check_jinja_expr``)
remain as thin wrappers for backward compatibility with existing Makefile
targets and workflow references.
"""
from devx.tools.ansible_checks._shared import (
DEFAULT_ANSIBLE_DIRS,
AnsibleFileFinder,
AnsibleYAMLParser,
ViolationReporter,
)
from devx.tools.ansible_checks.jinja_expr import check_jinja_expr
from devx.tools.ansible_checks.no_log import check_no_log
from devx.tools.ansible_checks.no_state_absent_on_db import check_no_state_absent_on_db
from devx.tools.ansible_checks.patterns import check_patterns
from devx.tools.ansible_checks.set_fact_to_json import check_set_fact_to_json
__all__ = [
"DEFAULT_ANSIBLE_DIRS",
"AnsibleFileFinder",
"AnsibleYAMLParser",
"ViolationReporter",
"check_jinja_expr",
"check_no_log",
"check_no_state_absent_on_db",
"check_patterns",
"check_set_fact_to_json",
]
+178
View File
@@ -0,0 +1,178 @@
"""Shared utilities for Ansible check tools.
Provides composable helpers for file discovery, YAML parsing, and
violation reporting used by the modules in :mod:`devx.tools.ansible_checks`.
Composition over inheritance: each check module picks the helpers it
needs. Tools that don't parse YAML (e.g. line-based scanners) can skip
:class:`AnsibleYAMLParser` entirely.
"""
from __future__ import annotations
import sys
from collections.abc import Iterator
from pathlib import Path
from typing import Final
import click
import yaml
from devx.i18n import _
#: Default Ansible directories scanned by checks that accept ``--ansible-dir``.
#: Immutable tuple (not a list) to avoid module-level mutable globals.
DEFAULT_ANSIBLE_DIRS: Final[tuple[str, ...]] = ("ansible/roles", "ansible/playbooks")
class AnsibleFileFinder:
"""File discovery helpers for Ansible YAML files."""
@staticmethod
def find_task_files(base: Path, skip_molecule: bool = True) -> list[Path]:
"""Find all YAML files under *base*, recursively.
If *base* is a single YAML file, returns ``[base]``. If *base* is
not a file or directory, returns ``[]``. When *skip_molecule* is
True, files with ``molecule`` in their path parts are excluded.
"""
if base.is_file() and base.suffix in (".yml", ".yaml"):
return [base]
if not base.is_dir():
return []
files: list[Path] = []
for f in sorted(base.rglob("*.yml")) + sorted(base.rglob("*.yaml")):
if skip_molecule and "molecule" in f.parts:
continue
files.append(f)
return files
@staticmethod
def find_yaml_files(base: Path, skip_molecule: bool = True) -> list[Path]:
"""Find YAML files under *base* using ``glob`` (non-recursive rglob).
Unlike :meth:`find_task_files`, this uses ``base.glob("**/*.yml")``
and does not check the suffix when *base* is a single file (any
file is accepted). Used by the Jinja expression checker which
scans all YAML files including defaults/handlers.
"""
if base.is_file():
return [base]
files: list[Path] = []
for pattern in ("**/*.yml", "**/*.yaml"):
files.extend(base.glob(pattern))
if skip_molecule:
return [f for f in files if "molecule" not in f.parts]
return files
@staticmethod
def find_task_and_playbook_files(base: Path, skip_molecule: bool = True) -> list[Path]:
"""Find task files (``tasks/*.yml``) and playbook files (``playbooks/*.yml``).
Used by the no_log checker which scans role task files and
top-level playbook files. When *skip_molecule* is True, molecule
scenario files are excluded.
"""
task_files = list(base.rglob("tasks/*.yml")) + list(base.rglob("tasks/*.yaml"))
task_files += list(base.glob("playbooks/*.yml")) + list(base.glob("playbooks/*.yaml"))
if skip_molecule:
task_files = [f for f in task_files if "molecule" not in f.parts]
return sorted(task_files)
class AnsibleYAMLParser:
"""YAML parsing helpers for Ansible files."""
@staticmethod
def parse_file(content: str) -> list[dict]:
"""Parse multi-document YAML from *content*.
Returns a list of non-None documents. On ``YAMLError`` or
``OSError``, returns an empty list (the caller skips the file).
"""
try:
docs = list(yaml.safe_load_all(content))
except (yaml.YAMLError, OSError):
return []
return [d for d in docs if d]
@staticmethod
def iter_tasks(doc: dict | list) -> Iterator[tuple[dict, int]]:
"""Yield ``(task_dict, line_number)`` tuples from a YAML document.
Handles:
- Bare task lists (role tasks files): ``[task1, task2, ...]``
- Play dicts with ``hosts`` key: iterates ``tasks``,
``pre_tasks``, ``post_tasks``, ``handlers`` sections
- Nested ``block`` tasks
The line number is the 1-based index within the task section
(not the file line number callers use it for display only).
"""
if isinstance(doc, list):
for i, item in enumerate(doc):
if isinstance(item, dict):
if any(k in item for k in ("tasks", "pre_tasks", "post_tasks", "handlers")):
yield from AnsibleYAMLParser._iter_play_sections(item)
else:
yield item, i + 1
block = item.get("block")
if isinstance(block, list):
for j, bt in enumerate(block):
if isinstance(bt, dict):
yield bt, i + j + 1
elif isinstance(doc, dict):
yield from AnsibleYAMLParser._iter_play_sections(doc)
@staticmethod
def _iter_play_sections(doc: dict) -> Iterator[tuple[dict, int]]:
"""Yield tasks from play sections (tasks, pre_tasks, post_tasks, handlers)."""
for section_key in ("tasks", "pre_tasks", "post_tasks", "handlers"):
section = doc.get(section_key)
if isinstance(section, list):
for i, task in enumerate(section):
if isinstance(task, dict):
yield task, i + 1
block = task.get("block")
if isinstance(block, list):
for j, bt in enumerate(block):
if isinstance(bt, dict):
yield bt, i + j + 1
class ViolationReporter:
"""Standardized violation formatting and reporting."""
@staticmethod
def format_violation(
filepath: Path,
repo_root: Path,
line_num: int | None,
message: str,
) -> str:
"""Format a violation as ``"{relative_path}:{line_num} — message"``.
Falls back to the full path if *filepath* is not relative to
*repo_root*. When *line_num* is None, omits the line number.
"""
try:
display_path = filepath.relative_to(repo_root)
except ValueError:
display_path = filepath
if line_num is not None:
return f"{display_path}:{line_num}{message}"
return f"{display_path}{message}"
@staticmethod
def report(violations: list[str], tool_name: str) -> None:
"""Print violations and exit with the appropriate code.
Prints ``[{tool_name}] FAIL`` or ``[{tool_name}] OK`` and exits
1 if violations are non-empty, 0 otherwise.
"""
if violations:
click.echo(_("[{tool}] FAIL: {count} violation(s) found.", tool=tool_name, count=len(violations)))
for v in violations:
click.echo(f" - {v}")
sys.exit(1)
click.echo(_("[{tool}] OK: no violations found.", tool=tool_name))
+226
View File
@@ -0,0 +1,226 @@
"""Validate Jinja2 expressions in Ansible files by rendering them.
Extracted from :mod:`devx.tools.check_jinja_expr` as part of the
Ansible check tool consolidation. The old module remains as a thin
wrapper for backward compatibility.
"""
from __future__ import annotations
import re
from pathlib import Path
from jinja2 import Environment
from jinja2.exceptions import TemplateSyntaxError, UndefinedError
from devx.tools.ansible_checks._shared import AnsibleFileFinder, ViolationReporter
REPO_ROOT = Path.cwd()
MOCK_CONTEXT: dict[str, object] = {
"now": lambda fmt=None: (
"2026-01-01T00:00:00+00:00"
if fmt
else type(
"Now",
(),
{
"timestamp": lambda self: 1735689600.0,
"strftime": lambda self, fmt: "2026-01-01T00:00:00+00:00",
},
)()
),
"ansible_date_time": {
"iso8601": "2026-01-01T00:00:00+00:00",
"epoch": "1735689600",
},
"ansible_facts": {
"service_mgr": "systemd",
"architecture": "x86_64",
"distribution_release": "noble",
"virtualization_type": "none",
"interfaces": ["eth0", "lo"],
"hostname": "test-host",
},
"ansible_host": "10.0.0.1",
"env": "staging",
"environment": "staging",
"customer_id": "test",
"zitadel_domain": "zitadel.test",
"_env_name": "staging",
"_observability_data_root": "/opt",
"skip_zitadel_stack": False,
"skip_htpasswd": False,
"skip_observability_stack": False,
"backup_enabled": True,
"app_filter": "",
"app_domain": "test.example.com",
"oidc_client_id": "test-client-id",
"oidc_client_secret": "test-secret", # nosec B105 — mock value for Jinja rendering, not a real secret
"s3_backup_bucket": "test-bucket",
"s3_endpoint": "https://s3.test",
"s3_access_key": "test-key",
"s3_secret_key": "test-secret", # nosec B105 — mock value for Jinja rendering, not a real secret
}
EXPR_PATTERN = re.compile(r"\{\{(.*?)\}\}", re.DOTALL)
def _default_ansible_dirs() -> list[Path]:
"""Return the default directories to scan for Ansible files."""
return [
REPO_ROOT / "ansible" / "playbooks",
REPO_ROOT / "ansible" / "roles",
]
def _find_yaml_files(path: Path) -> list[Path]:
"""Find Ansible YAML files (tasks, playbooks, handlers) in a path."""
return AnsibleFileFinder.find_yaml_files(path, skip_molecule=True)
def _extract_expressions(content: str) -> list[str]:
"""Extract Jinja expressions from file content."""
expressions = []
for match in EXPR_PATTERN.finditer(content):
raw = match.group(1)
if "\n" in raw:
continue
expr = raw.strip()
if not expr or expr.startswith("%") or len(expr) <= 1:
continue
if expr.startswith(".") or "println" in expr:
continue
if ".State." in expr or ".NetworkSettings." in expr:
continue
if expr.count("(") != expr.count(")"):
continue
if expr.count("{") != expr.count("}"):
continue
if expr.count("[") != expr.count("]"):
continue
expressions.append(expr)
return expressions
def _render_expression(expr: str) -> tuple[bool, str]:
"""Try to render a Jinja expression. Returns (success, error_msg)."""
try:
env = Environment(autoescape=False, keep_trailing_newline=True) # nosec B701 — Ansible Jinja, not web-facing # noqa: S701
def _strftime(string_format: str, second: float | None = None, utc: bool = False) -> str:
if isinstance(string_format, (int, float)) and isinstance(second, str) and "%" in second:
raise ValueError( # noqa: TRY301
"Invalid value for epoch value — strftime filter arguments "
"are reversed. The format string must be the piped value: "
"'%format%' | strftime(epoch), not epoch | strftime('%format%')"
)
return str(string_format)
env.filters["strftime"] = _strftime
env.filters["b64decode"] = lambda x: x
env.filters["b64encode"] = lambda x: x
env.filters["regex_replace"] = lambda x, pattern, replacement="": x
env.filters["int"] = lambda x, default=0: (
int(x) if isinstance(x, (int, float, str)) and str(x).lstrip("-").isdigit() else default
)
env.filters["bool"] = bool
env.filters["basename"] = lambda x: str(x).rsplit("/", 1)[-1]
env.filters["dirname"] = lambda x: str(x).rsplit("/", 1)[0] if "/" in str(x) else "."
env.filters["combine"] = lambda *args, **kwargs: args[0]
env.filters["from_json"] = lambda x: x
env.filters["to_json"] = lambda x: x
env.filters["ternary"] = lambda x, true_val, false_val=None: true_val if x else false_val
env.filters["dict2items"] = lambda x: [
{"key": k, "value": v} for k, v in (x.items() if isinstance(x, dict) else [])
]
env.filters["map"] = lambda x, attribute=None: x
env.filters["default"] = lambda x, default_value="", boolean=False: x if x else default_value
env.filters["from_yaml"] = lambda x: x
env.filters["difference"] = lambda x, y: x
env.filters["join"] = lambda x, sep="": sep.join(str(i) for i in (x if isinstance(x, list) else [x]))
env.filters["list"] = lambda x: list(x) if isinstance(x, (list, tuple)) else [x]
env.filters["length"] = lambda x: len(x) if hasattr(x, "__len__") else 0
env.filters["items"] = lambda x: list(x.items()) if isinstance(x, dict) else []
env.filters["first"] = lambda x: x[0] if isinstance(x, (list, str)) and x else x
env.filters["last"] = lambda x: x[-1] if isinstance(x, (list, str)) and x else x
env.filters["upper"] = lambda x: str(x).upper()
env.filters["lower"] = lambda x: str(x).lower()
env.filters["replace"] = lambda x, old, new: str(x).replace(old, new)
env.filters["split"] = lambda x, sep=None: str(x).split(sep) if sep else str(x).split()
env.filters["trim"] = lambda x: str(x).strip()
env.filters["sort"] = lambda x: sorted(x) if isinstance(x, list) else x
env.filters["unique"] = lambda x: list(set(x)) if isinstance(x, list) else x
env.filters["count"] = lambda x: len(x) if hasattr(x, "__len__") else 0
env.filters["float"] = lambda x, default=0.0: (
float(x) if isinstance(x, (int, float, str)) and str(x).replace(".", "").lstrip("-").isdigit() else default
)
env.filters["string"] = str
env.filters["indent"] = lambda x, width=4: str(x)
env.filters["to_nice_json"] = str
env.filters["to_nice_yaml"] = str
env.filters["from_yaml_all"] = lambda x: x
env.filters["groupby"] = lambda x: x
env.filters["dictsort"] = lambda x: list(x.items()) if isinstance(x, dict) else []
env.filters["max"] = lambda x: max(x) if isinstance(x, list) and x else x
env.filters["min"] = lambda x: min(x) if isinstance(x, list) and x else x
env.filters["reverse"] = lambda x: list(reversed(x)) if isinstance(x, list) else x
env.filters["flatten"] = lambda x: x
env.filters["product"] = lambda x: x
env.filters["zip"] = lambda x: x
env.filters["subelements"] = lambda x: x
env.filters["json_query"] = lambda x: x
env.filters["type_debug"] = lambda x: type(x).__name__
env.globals["lookup"] = lambda *args, **kwargs: ""
env.globals["query"] = lambda *args, **kwargs: []
template = env.from_string("{{ " + expr + " }}")
result = template.render(**MOCK_CONTEXT)
except TemplateSyntaxError as e:
return False, f"Syntax error: {e.message}"
except UndefinedError as e:
return True, f"Skipped (undefined: {e})"
except Exception as e:
error_msg = str(e)
if "Invalid value for epoch" in error_msg:
return False, f"strftime filter argument error: {error_msg}"
return True, f"Skipped ({type(e).__name__}: {error_msg})"
else:
return True, result
def _check_file(filepath: Path, repo_root: Path) -> list[str]:
"""Check all Jinja expressions in a file. Returns list of violations."""
violations = []
content = filepath.read_text()
expressions = _extract_expressions(content)
for expr in expressions:
success, msg = _render_expression(expr)
if not success:
display_path = ViolationReporter.format_violation(filepath, repo_root, None, "")
display_path = display_path.removesuffix("")
violations.append(f"{display_path}: `{{{{ {expr} }}}}` — {msg}")
return violations
def check_jinja_expr(path: Path | None, ansible_dirs: list[Path] | None = None) -> list[str]:
"""Validate Jinja2 expressions in Ansible files.
Args:
path: Specific file or directory to check. If None, *ansible_dirs*
is used.
ansible_dirs: Directories to scan when *path* is None.
Returns:
List of violation messages (empty if all renderable expressions pass).
"""
if path:
files = _find_yaml_files(path)
else:
files: list[Path] = []
for d in ansible_dirs or _default_ansible_dirs():
files.extend(_find_yaml_files(d))
all_violations: list[str] = []
for f in files:
all_violations.extend(_check_file(f, REPO_ROOT))
return all_violations
+128
View File
@@ -0,0 +1,128 @@
"""Check Ansible tasks for missing no_log on secret-handling tasks.
Extracted from :mod:`devx.tools.check_ansible_no_log` as part of the
Ansible check tool consolidation. The old module remains as a thin
wrapper for backward compatibility.
"""
from __future__ import annotations
import re
from pathlib import Path
from devx.tools.ansible_checks._shared import AnsibleFileFinder, AnsibleYAMLParser
# Patterns that indicate a task is handling secrets.
SECRET_PATTERNS = [
re.compile(r"\{\{[^}]*_secrets\.", re.IGNORECASE),
re.compile(r"\{\{[^}]*password", re.IGNORECASE),
re.compile(r"\{\{[^}]*_secret\b", re.IGNORECASE),
re.compile(r"\{\{[^}]*api_key", re.IGNORECASE),
re.compile(r"\{\{[^}]*(?:vault_token|auth_token|access_token|bot_token)", re.IGNORECASE),
]
TASK_VALUE_KEYS = {
"shell",
"command",
"ansible.builtin.shell",
"ansible.builtin.command",
"ansible.builtin.template",
"ansible.builtin.copy",
"ansible.builtin.debug",
"template",
"copy",
"debug",
"cmd",
"msg",
"content",
}
NON_VALUE_KEYS = {
"name",
"when",
"loop",
"loop_control",
"changed_when",
"failed_when",
"no_log",
"register",
"tags",
"vars",
"become",
"become_user",
"delegate_to",
"run_once",
"environment",
"with_items",
"with_dict",
"with_list",
}
def _contains_secret(value: object) -> bool:
"""Recursively check if a value contains secret-like variable references."""
if isinstance(value, str):
return any(p.search(value) for p in SECRET_PATTERNS)
if isinstance(value, dict):
return any(_contains_secret(v) for v in value.values())
if isinstance(value, list):
return any(_contains_secret(item) for item in value)
return False
def _has_no_log(task: dict) -> bool:
"""Check if a task has no_log set to a non-False value."""
no_log = task.get("no_log", False)
return no_log is not False and no_log is not None
def _check_task(task: dict, file_path: Path, task_num: int) -> list[str]:
"""Check a single task for missing no_log on secret values."""
violations: list[str] = []
if _has_no_log(task):
return violations
has_secrets = False
for key, value in task.items():
if key in NON_VALUE_KEYS:
continue
if _contains_secret(value):
has_secrets = True
break
if has_secrets:
task_name = task.get("name", "<unnamed>")
violations.append(
f"{file_path}:{task_num}: Task '{task_name}' references secrets "
f"but has no no_log. Add `no_log: true` or "
f'`no_log: "{{{{ not (debug_mode | default(false) | bool) }}}}"` '
f"to prevent credential leakage in Ansible output."
)
return violations
def check_no_log(path: Path, ansible_dirs: list[Path] | None = None) -> list[str]:
"""Check all Ansible task files for missing no_log on secret-handling tasks.
Args:
path: The base directory to scan (or a specific file).
ansible_dirs: Unused kept for API symmetry with other checks.
The no_log checker scans *path* directly.
Returns:
List of violation messages (empty if all OK).
"""
all_violations: list[str] = []
task_files = AnsibleFileFinder.find_task_and_playbook_files(path)
for task_file in task_files:
try:
content = task_file.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
docs = AnsibleYAMLParser.parse_file(content)
for doc in docs:
for task, task_num in AnsibleYAMLParser.iter_tasks(doc):
all_violations.extend(_check_task(task, task_file, task_num))
return all_violations
# Backward-compat alias for the old public function name.
check_directory = check_no_log
@@ -0,0 +1,100 @@
"""Check Ansible tasks for ``state: absent`` on database data directories.
Extracted from :mod:`devx.tools.check_ansible_no_state_absent_on_db` as
part of the Ansible check tool consolidation. The old module remains as
a thin wrapper for backward compatibility.
"""
from __future__ import annotations
import re
from pathlib import Path
from devx.tools.ansible_checks._shared import AnsibleFileFinder, ViolationReporter
REPO_ROOT = Path.cwd()
DB_PATH_PATTERNS = (
re.compile(r"postgres/zitadel-db", re.IGNORECASE),
re.compile(r"postgres/\w+-db", re.IGNORECASE),
re.compile(r"/var/lib/postgresql/data", re.IGNORECASE),
re.compile(r"/var/lib/postgresql/data/\w+-db", re.IGNORECASE),
)
DESTRUCTIVE_PATTERNS = (
re.compile(r"state:\s*absent", re.IGNORECASE),
re.compile(r"rm\s+-rf.*\bdb\b", re.IGNORECASE),
)
ALLOWED_CONTEXT_KEYWORDS = (
"upgrade-postgres",
"PG_VERSION",
"pg_version",
)
ALLOW_MARKER = "lint:allow-state-absent"
def _find_task_files(base: Path) -> list[Path]:
"""Find all YAML task files under a base directory, skipping molecule."""
return AnsibleFileFinder.find_task_files(base, skip_molecule=True)
def _check_file(filepath: Path, repo_root: Path) -> list[str]:
"""Check a YAML file for state: absent on DB data directory paths."""
try:
content = filepath.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
return []
if not any(p.search(content) for p in DB_PATH_PATTERNS):
return []
display_path = ViolationReporter.format_violation(filepath, repo_root, None, "")
display_path = display_path.removesuffix("")
violations: list[str] = []
lines = content.splitlines()
for i, line in enumerate(lines):
for db_pattern in DB_PATH_PATTERNS:
if not db_pattern.search(line):
continue
context_start = max(0, i - 5)
context_end = min(len(lines), i + 6)
context = "\n".join(lines[context_start:context_end])
if any(kw in context for kw in ALLOWED_CONTEXT_KEYWORDS):
continue
if ALLOW_MARKER in context:
continue
for dp in DESTRUCTIVE_PATTERNS:
if dp.search(context):
violations.append(
f"{display_path}:{i + 1} — destructive operation "
f"({dp.pattern!r}) near DB data directory path "
f"({db_pattern.pattern!r}). "
f"Database directories must never be wiped automatically (ADR-0028). "
f"If this is legitimate (e.g. PG upgrade), add "
f"#{ALLOW_MARKER} to the task."
)
break
return violations
def check_no_state_absent_on_db(path: Path | None, ansible_dirs: list[Path] | None = None) -> list[str]:
"""Check that no Ansible task uses state: absent on a DB data directory.
Args:
path: Specific file or directory to check. If None, *ansible_dirs*
is used.
ansible_dirs: Directories to scan when *path* is None.
Returns:
List of violation messages (empty if clean).
"""
if path:
files = _find_task_files(path)
else:
files: list[Path] = []
for d in ansible_dirs or []:
files.extend(_find_task_files(d))
all_violations: list[str] = []
for f in files:
all_violations.extend(_check_file(f, REPO_ROOT))
return all_violations
+226
View File
@@ -0,0 +1,226 @@
"""Check Ansible tasks for dangerous patterns that mask failures.
Extracted from :mod:`devx.tools.check_ansible_patterns` as part of the
Ansible check tool consolidation. The old module remains as a thin
wrapper for backward compatibility.
"""
from __future__ import annotations
import re
from pathlib import Path
from devx.tools.ansible_checks._shared import AnsibleFileFinder, AnsibleYAMLParser
REPO_ROOT = Path.cwd()
# Comment marker to explicitly allow a pattern on a specific task
ALLOW_MARKER = "lint:allow-failure-masking"
# Patterns that mask failures when used in shell/command tasks
OR_TRUE_PATTERN = re.compile(r"\|\|\s*true\b", re.IGNORECASE)
REDIRECT_DEVNULL_PATTERN = re.compile(r"2>/dev/null")
# Module keys that accept shell/command strings
SHELL_MODULE_KEYS = frozenset(
{
"shell",
"command",
"ansible.builtin.shell",
"ansible.builtin.command",
"cmd",
"ansible.builtin.raw",
"raw",
}
)
# Task keys whose values might contain shell commands
COMMAND_VALUE_KEYS = frozenset(
{
"shell",
"command",
"ansible.builtin.shell",
"ansible.builtin.command",
"cmd",
"raw",
"ansible.builtin.raw",
}
)
LEGITIMATE_COMMAND_PREFIXES = (
"docker rm",
"docker stop",
"docker rmi",
"docker network rm",
"docker volume rm",
"pkill",
"kill",
"journalctl --vacuum",
"apt-get clean",
"apt-get autoremove",
"docker image prune",
"docker container prune",
"docker volume prune",
"docker builder prune",
"find / -name",
"chmod",
"rm -f",
"docker network connect",
"curl.*api/v2/admin/tsdb/snapshot",
)
LEGITIMATE_TASK_NAME_KEYWORDS = (
"remove",
"cleanup",
"clean up",
"prune",
"purge",
"disconnect",
"stop",
"kill",
"strip suid",
"suid",
"vacuum",
"ensure.*absent",
"may not exist",
"if exists",
"optional",
"best effort",
"no-op",
"noop",
"idempotent",
"sync",
)
CRITICAL_TASK_KEYWORDS = (
"password",
"secret",
"provision",
"oidc",
)
LEGITIMATE_FAILED_WHEN_KEYWORDS = (
"stop",
"start",
"check",
"wait",
"migrate",
"restart",
"rebuild",
"restore",
"remove",
"cleanup",
"sync",
"download",
"extract",
"verify",
)
def _is_legitimate_or_true(command_str: str, task_name: str) -> bool:
"""Check if a || true in a command is in a legitimate context."""
name_lower = task_name.lower()
if any(re.search(kw, name_lower) for kw in LEGITIMATE_TASK_NAME_KEYWORDS):
return True
cmd_lower = command_str.lower()
return any(re.search(prefix, cmd_lower) for prefix in LEGITIMATE_COMMAND_PREFIXES)
def _is_legitimate_devnull(command_str: str, task_name: str) -> bool:
"""Check if a 2>/dev/null in a command is in a legitimate context."""
return _is_legitimate_or_true(command_str, task_name)
def _check_task(task: dict, filepath: Path, task_num: int, repo_root: Path) -> list[str]:
"""Check a single task for dangerous failure-masking patterns."""
violations: list[str] = []
try:
display_path = filepath.relative_to(repo_root)
except ValueError:
display_path = filepath
task_name = task.get("name", "<unnamed>")
if ALLOW_MARKER in task_name:
return violations
for key in COMMAND_VALUE_KEYS:
value = task.get(key)
if value is None:
continue
value_str = str(value)
if OR_TRUE_PATTERN.search(value_str) and not _is_legitimate_or_true(value_str, task_name):
violations.append(
f"{display_path}:{task_num} — task '{task_name}' uses "
f"'|| true' in {key} which may mask real failures. "
f"If this is a cleanup/idempotency operation, rename the "
f"task to include 'remove'/'cleanup'/'prune' or add "
f"#{ALLOW_MARKER} to the task."
)
failed_when = task.get("failed_when")
if failed_when is False:
name_lower = task_name.lower()
is_legitimate = any(kw in name_lower for kw in LEGITIMATE_FAILED_WHEN_KEYWORDS)
if not is_legitimate:
for kw in CRITICAL_TASK_KEYWORDS:
if kw in name_lower:
violations.append(
f"{display_path}:{task_num} — critical task '{task_name}' "
f"has failed_when: false, which masks failures on "
f"a {kw}-related operation. Remove failed_when: false "
f"or add #{ALLOW_MARKER} if masking is intentional."
)
break
return violations
def _check_file(filepath: Path, repo_root: Path) -> list[str]:
"""Check a YAML file for dangerous failure-masking patterns."""
try:
content = filepath.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
return []
if not (
OR_TRUE_PATTERN.search(content) or "failed_when: false" in content or REDIRECT_DEVNULL_PATTERN.search(content)
):
return []
has_allow_marker = ALLOW_MARKER in content
docs = AnsibleYAMLParser.parse_file(content)
violations: list[str] = []
for doc in docs:
for task, task_num in AnsibleYAMLParser.iter_tasks(doc):
violations.extend(_check_task(task, filepath, task_num, repo_root))
if has_allow_marker:
violations = []
return violations
def _check_tasks(doc: dict, filepath: Path, errors: list[str], repo_root: Path) -> None:
"""Check top-level tasks and nested task sections in a playbook doc."""
for task, task_num in AnsibleYAMLParser._iter_play_sections(doc):
errors.extend(_check_task(task, filepath, task_num, repo_root))
def _find_task_files(base: Path) -> list[Path]:
"""Find all YAML task files under a base directory, skipping molecule."""
return AnsibleFileFinder.find_task_files(base, skip_molecule=True)
def check_patterns(path: Path | None, ansible_dirs: list[Path] | None = None) -> list[str]:
"""Check Ansible tasks for dangerous failure-masking patterns.
Args:
path: Specific file or directory to check. If None, *ansible_dirs*
is used.
ansible_dirs: Directories to scan when *path* is None.
Returns:
List of violation messages (empty if clean).
"""
if path:
files = _find_task_files(path)
else:
files: list[Path] = []
for d in ansible_dirs or []:
files.extend(_find_task_files(d))
all_violations: list[str] = []
for f in files:
all_violations.extend(_check_file(f, REPO_ROOT))
return all_violations
@@ -0,0 +1,134 @@
"""Check that Ansible ``set_fact`` tasks don't misuse ``| to_json``.
Extracted from :mod:`devx.tools.check_ansible_set_fact_to_json` as part
of the Ansible check tool consolidation. The old module remains as a
thin wrapper for backward compatibility.
"""
from __future__ import annotations
from pathlib import Path
import yaml
from devx.tools.ansible_checks._shared import AnsibleFileFinder, ViolationReporter
REPO_ROOT = Path.cwd()
TO_JSON_FILTERS = ("| to_json", "| to_nice_json", "|to_json", "|to_nice_json")
def _find_task_files(base: Path) -> list[Path]:
"""Find all YAML task files under a base directory."""
return AnsibleFileFinder.find_task_files(base, skip_molecule=False)
def _check_file(filepath: Path, repo_root: Path) -> list[str]:
"""Check a single YAML file for set_fact + to_json misuse."""
errors: list[str] = []
try:
content = filepath.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
return errors
try:
docs = list(yaml.safe_load_all(content))
except yaml.YAMLError as exc:
return [f"{filepath}: cannot parse YAML: {exc}"]
for doc in docs:
if isinstance(doc, list):
for item in doc:
if isinstance(item, dict):
if any(k in item for k in ("tasks", "pre_tasks", "post_tasks", "handlers", "roles")):
_check_tasks(item, filepath, errors, repo_root)
else:
_check_task(item, filepath, errors, repo_root)
block = item.get("block")
if isinstance(block, list):
_check_task_list(block, filepath, errors, repo_root)
elif isinstance(doc, dict):
_check_tasks(doc, filepath, errors, repo_root)
return errors
def _check_tasks(doc: dict, filepath: Path, errors: list[str], repo_root: Path) -> None:
"""Check top-level tasks and nested task sections in a playbook doc."""
tasks = doc.get("tasks")
if isinstance(tasks, list):
_check_task_list(tasks, filepath, errors, repo_root)
for role_key in ("pre_tasks", "post_tasks", "handlers"):
section = doc.get(role_key)
if isinstance(section, list):
_check_task_list(section, filepath, errors, repo_root)
roles = doc.get("roles")
if isinstance(roles, list):
for role_entry in roles:
if isinstance(role_entry, dict):
role_tasks = role_entry.get("tasks")
if isinstance(role_tasks, list):
_check_task_list(role_tasks, filepath, errors, repo_root)
def _check_task_list(tasks: list, filepath: Path, errors: list[str], repo_root: Path) -> None:
"""Check a list of task definitions for set_fact + to_json."""
for task in tasks:
if not isinstance(task, dict):
continue
_check_task(task, filepath, errors, repo_root)
block = task.get("block")
if isinstance(block, list):
_check_task_list(block, filepath, errors, repo_root)
def _check_task(task: dict, filepath: Path, errors: list[str], repo_root: Path) -> None:
"""Check a single task for set_fact + to_json misuse."""
has_set_fact = False
for key in task:
if key in {"set_fact", "ansible.builtin.set_fact"}:
has_set_fact = True
break
if not has_set_fact:
return
set_fact_body = task.get("set_fact") or task.get("ansible.builtin.set_fact")
if not isinstance(set_fact_body, dict):
return
task_name = task.get("name", "(unnamed)")
for fact_name, fact_value in set_fact_body.items():
if fact_name in ("cacheable",):
continue
value_str = str(fact_value)
for filter_pattern in TO_JSON_FILTERS:
if filter_pattern in value_str:
display_path = ViolationReporter.format_violation(filepath, repo_root, None, "")
display_path = display_path.removesuffix("")
errors.append(
f"{display_path}: task '{task_name}' "
f"sets fact '{fact_name}' with '{filter_pattern.strip()}' "
f"— this converts native Python types to JSON strings. "
f"Remove the filter to preserve the native type, or use "
f"'| from_json' in the consuming task if the string "
f"representation is intentional."
)
break
def check_set_fact_to_json(path: Path | None, ansible_dirs: list[Path] | None = None) -> list[str]:
"""Check that set_fact tasks don't misuse to_json.
Args:
path: Specific file or directory to check. If None, *ansible_dirs*
is used.
ansible_dirs: Directories to scan when *path* is None.
Returns:
List of error messages (empty if all OK).
"""
if path:
files = _find_task_files(path)
else:
files: list[Path] = []
for d in ansible_dirs or []:
files.extend(_find_task_files(d))
all_errors: list[str] = []
for f in files:
all_errors.extend(_check_file(f, REPO_ROOT))
return all_errors
+86
View File
@@ -0,0 +1,86 @@
"""Validate Prometheus alert rules with promtool check rules.
Renders an alert-rules Jinja2 template with test values and validates
the output with ``promtool check rules``. Exits 0 if valid, non-zero
otherwise. Skips (exits 0) if promtool is not on PATH.
Usage::
python -m devx.tools.check_alert_rules \\
--template-path ansible/roles/observability/templates \\
--template-name alert-rules.yml.j2
# With extra template variables:
python -m devx.tools.check_alert_rules \\
--template-path ansible/roles/observability/templates \\
--template-name alert-rules.yml.j2 \\
--var grafana_base_url=https://grafana.test.example.com
"""
from __future__ import annotations
import shutil
import subprocess # nosec B404 — used to run promtool, a trusted binary
import sys
import tempfile
from pathlib import Path
import click
from devx.utils.jinja import make_env, render_template
@click.command()
@click.option(
"--template-path",
type=click.Path(exists=True, path_type=Path),
required=True,
help="Path to the directory containing the Jinja2 template.",
)
@click.option(
"--template-name",
default="alert-rules.yml.j2",
help="Name of the Jinja2 template file to render.",
)
@click.option(
"--var",
"template_vars",
multiple=True,
help="Template variables in key=value format (can be repeated). "
"Example: --var grafana_base_url=https://grafana.example.com",
)
def main(template_path: Path, template_name: str, template_vars: tuple[str, ...]) -> None:
"""Validate rendered alert rules with promtool."""
if not shutil.which("promtool"):
click.echo("promtool not found in PATH — skipping alert rules validation")
return
# Parse template variables
kwargs: dict[str, str] = {}
for v in template_vars:
if "=" in v:
key, value = v.split("=", 1)
kwargs[key] = value
env = make_env(str(template_path))
output = render_template(env, template_name, **kwargs)
with tempfile.NamedTemporaryFile(mode="w", suffix=".yml", delete=False) as f:
f.write(output)
tmp_path = f.name
click.echo("[check-alert-rules] Validating rendered rules with promtool...")
result = subprocess.run( # nosec
["promtool", "check", "rules", tmp_path],
capture_output=True,
text=True,
check=False,
)
click.echo(result.stdout, nl=False)
if result.returncode != 0:
click.echo(result.stderr, nl=False, err=True)
sys.exit(result.returncode)
if __name__ == "__main__": # pragma: no cover
main()
+71
View File
@@ -0,0 +1,71 @@
"""Check Ansible tasks for missing no_log on secret-handling tasks.
Thin wrapper around :mod:`devx.tools.ansible_checks.no_log` for
backward compatibility. The check logic lives in the subpackage; this
module preserves the CLI entry point and re-exports the internal
helpers so existing tests and imports continue to work.
Usage::
python -m devx.tools.check_ansible_no_log
python -m devx.tools.check_ansible_no_log --path ansible/roles/my_role
python -m devx.tools.check_ansible_no_log --ansible-dir ansible/roles
Exit code 0 if all secret-handling tasks have no_log, 1 otherwise.
"""
from __future__ import annotations
import sys
from pathlib import Path
import click
from devx.tools.ansible_checks.no_log import (
NON_VALUE_KEYS, # noqa: F401
SECRET_PATTERNS, # noqa: F401 — re-exported for backward compat
TASK_VALUE_KEYS, # noqa: F401
_check_task, # noqa: F401
_contains_secret, # noqa: F401
_has_no_log, # noqa: F401
check_directory, # noqa: F401
check_no_log, # noqa: F401
)
REPO_ROOT = Path.cwd()
DEFAULT_ANSIBLE_DIR = REPO_ROOT / "ansible"
@click.command()
@click.option(
"--path",
type=click.Path(exists=True, path_type=Path),
help="Check a specific file or directory (default: ansible/).",
)
@click.option(
"--ansible-dir",
type=click.Path(exists=True, path_type=Path),
default=None,
help="Override the default ansible directory (default: ansible/).",
)
def main(path: Path | None, ansible_dir: Path | None) -> None:
"""Check that Ansible tasks handling secrets have no_log set."""
target = path or ansible_dir or DEFAULT_ANSIBLE_DIR
if not target.is_dir():
click.echo(f"Error: {target} is not a directory", err=True)
sys.exit(2)
violations = check_no_log(target)
if violations:
click.echo(f"Found {len(violations)} task(s) handling secrets without no_log:\n")
for v in violations:
click.echo(f" {v}")
click.echo(f"\nTotal: {len(violations)} violation(s).")
sys.exit(1)
click.echo(f"[check-ansible-no-log] All secret-handling tasks have no_log. ({target})")
if __name__ == "__main__": # pragma: no cover
main()
@@ -0,0 +1,72 @@
"""Check Ansible tasks for ``state: absent`` on database data directories.
Thin wrapper around
:mod:`devx.tools.ansible_checks.no_state_absent_on_db` for backward
compatibility. The check logic lives in the subpackage; this module
preserves the CLI entry point and re-exports the internal helpers so
existing tests and imports continue to work.
Usage::
python -m devx.tools.check_ansible_no_state_absent_on_db
python -m devx.tools.check_ansible_no_state_absent_on_db --path ansible/roles/zitadel/tasks/main.yml
Exit code 0 if no violations found, 1 otherwise.
"""
from __future__ import annotations
import sys
from pathlib import Path
import click
from devx.tools.ansible_checks.no_state_absent_on_db import (
ALLOW_MARKER, # noqa: F401 — re-exported for backward compat
ALLOWED_CONTEXT_KEYWORDS, # noqa: F401
DB_PATH_PATTERNS, # noqa: F401
DESTRUCTIVE_PATTERNS, # noqa: F401
REPO_ROOT,
_check_file, # noqa: F401
_find_task_files, # noqa: F401
check_no_state_absent_on_db, # noqa: F401
)
DEFAULT_ANSIBLE_DIRS: list[Path] = [
REPO_ROOT / "ansible" / "playbooks",
REPO_ROOT / "ansible" / "roles",
]
@click.command()
@click.option(
"--path",
type=click.Path(exists=True, path_type=Path),
help="Check a specific file or directory (default: ansible/playbooks + ansible/roles).",
)
@click.option(
"--ansible-dir",
"ansible_dirs",
type=click.Path(exists=True, path_type=Path),
multiple=True,
default=None,
help="Override the default ansible directories (can be repeated). Defaults to ansible/playbooks and ansible/roles.",
)
def main(path: Path | None, ansible_dirs: tuple[Path, ...]) -> None:
"""Check that no Ansible task uses state: absent on a DB data directory."""
dirs = list(ansible_dirs) if ansible_dirs else DEFAULT_ANSIBLE_DIRS
all_violations = check_no_state_absent_on_db(path) if path else check_no_state_absent_on_db(None, dirs)
if all_violations:
click.echo("[check-ansible-no-state-absent-on-db] FAIL: destructive operations on DB paths:")
for v in all_violations:
click.echo(f" - {v}")
click.echo(f"\nTotal: {len(all_violations)} violation(s).")
click.echo("Database data directories must never be wiped automatically (ADR-0028).")
sys.exit(1)
else:
click.echo("[check-ansible-no-state-absent-on-db] OK: no destructive operations on DB paths.")
if __name__ == "__main__": # pragma: no cover
main()
+79
View File
@@ -0,0 +1,79 @@
"""Check Ansible tasks for dangerous patterns that mask failures.
Thin wrapper around :mod:`devx.tools.ansible_checks.patterns` for
backward compatibility. The check logic lives in the subpackage; this
module preserves the CLI entry point and re-exports the internal
helpers so existing tests and imports continue to work.
Usage::
python -m devx.tools.check_ansible_patterns
python -m devx.tools.check_ansible_patterns --path ansible/roles/app_container/tasks/main.yml
Exit code 0 if no violations found, 1 otherwise.
"""
from __future__ import annotations
import sys
from pathlib import Path
import click
from devx.tools.ansible_checks.patterns import (
ALLOW_MARKER, # noqa: F401 — re-exported for backward compat
COMMAND_VALUE_KEYS, # noqa: F401
CRITICAL_TASK_KEYWORDS, # noqa: F401
LEGITIMATE_COMMAND_PREFIXES, # noqa: F401
LEGITIMATE_FAILED_WHEN_KEYWORDS, # noqa: F401
LEGITIMATE_TASK_NAME_KEYWORDS, # noqa: F401
OR_TRUE_PATTERN, # noqa: F401
REDIRECT_DEVNULL_PATTERN, # noqa: F401
REPO_ROOT,
SHELL_MODULE_KEYS, # noqa: F401
_check_file, # noqa: F401
_check_task, # noqa: F401
_check_tasks, # noqa: F401
_find_task_files, # noqa: F401
_is_legitimate_devnull, # noqa: F401
_is_legitimate_or_true, # noqa: F401
check_patterns, # noqa: F401
)
DEFAULT_ANSIBLE_DIRS: list[Path] = [
REPO_ROOT / "ansible" / "playbooks",
REPO_ROOT / "ansible" / "roles",
]
@click.command()
@click.option(
"--path",
type=click.Path(exists=True, path_type=Path),
help="Check a specific file or directory (default: ansible/playbooks + ansible/roles).",
)
@click.option(
"--ansible-dir",
"ansible_dirs",
type=click.Path(exists=True, path_type=Path),
multiple=True,
default=None,
help="Override the default ansible directories (can be repeated). Defaults to ansible/playbooks and ansible/roles.",
)
def main(path: Path | None, ansible_dirs: tuple[Path, ...]) -> None:
"""Check Ansible tasks for dangerous failure-masking patterns."""
dirs = list(ansible_dirs) if ansible_dirs else DEFAULT_ANSIBLE_DIRS
all_violations = check_patterns(path) if path else check_patterns(None, dirs)
if all_violations:
click.echo("[check-ansible-patterns] FAIL: dangerous failure-masking patterns found:")
for v in all_violations:
click.echo(f" - {v}")
click.echo(f"\nTotal: {len(all_violations)} violation(s).")
sys.exit(1)
else:
click.echo("[check-ansible-patterns] OK: no dangerous failure-masking patterns.")
if __name__ == "__main__": # pragma: no cover
main()
@@ -0,0 +1,69 @@
"""Check that Ansible ``set_fact`` tasks don't misuse ``| to_json``.
Thin wrapper around :mod:`devx.tools.ansible_checks.set_fact_to_json`
for backward compatibility. The check logic lives in the subpackage;
this module preserves the CLI entry point and re-exports the internal
helpers so existing tests and imports continue to work.
Usage::
python -m devx.tools.check_ansible_set_fact_to_json
python -m devx.tools.check_ansible_set_fact_to_json --path ansible/playbooks/deploy.yml
Exit code 0 if no misuses found, 1 otherwise.
"""
from __future__ import annotations
import sys
from pathlib import Path
import click
from devx.tools.ansible_checks.set_fact_to_json import (
REPO_ROOT,
TO_JSON_FILTERS, # noqa: F401 — re-exported for backward compat
_check_file, # noqa: F401
_check_task, # noqa: F401
_check_task_list, # noqa: F401
_check_tasks, # noqa: F401
_find_task_files, # noqa: F401
check_set_fact_to_json, # noqa: F401
)
DEFAULT_ANSIBLE_DIRS: list[Path] = [
REPO_ROOT / "ansible" / "playbooks",
REPO_ROOT / "ansible" / "roles",
]
@click.command()
@click.option(
"--path",
type=click.Path(exists=True, path_type=Path),
help="Check a specific file or directory (default: ansible/playbooks + ansible/roles).",
)
@click.option(
"--ansible-dir",
"ansible_dirs",
type=click.Path(exists=True, path_type=Path),
multiple=True,
default=None,
help="Override the default ansible directories (can be repeated). Defaults to ansible/playbooks and ansible/roles.",
)
def main(path: Path | None, ansible_dirs: tuple[Path, ...]) -> None:
"""Check that set_fact tasks don't misuse to_json."""
dirs = list(ansible_dirs) if ansible_dirs else DEFAULT_ANSIBLE_DIRS
all_errors = check_set_fact_to_json(path) if path else check_set_fact_to_json(None, dirs)
if all_errors:
click.echo("[check-ansible-set-fact-to-json] FAIL: set_fact with to_json found:")
for err in all_errors:
click.echo(f" - {err}")
sys.exit(1)
else:
click.echo("[check-ansible-set-fact-to-json] OK: no set_fact tasks misuse to_json.")
if __name__ == "__main__": # pragma: no cover
main()
+166
View File
@@ -0,0 +1,166 @@
"""Check that Docker Compose services with healthchecks have ``init: true``.
This prevents zombie process accumulation on production VMs. Without
``init: true``, Docker uses the container's PID 1 process to reap
child processes. Many images (especially those using CMD-SHELL
healthchecks with ``wget``) don't call ``wait()`` on children, causing
zombies to accumulate.
The check scans all Jinja2 docker-compose templates for services that
have a ``healthcheck:`` key but no ``init: true`` key. Since the
templates use Jinja2 syntax (not pure YAML), the check uses text-based
parsing to identify service blocks and their properties.
Usage::
python -m devx.tools.check_docker_init
python -m devx.tools.check_docker_init --path ansible/roles/observability/templates/docker-compose.yml.j2
Exit code 0 if all services with healthchecks have init: true, 1 otherwise.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
import click
REPO_ROOT = Path.cwd()
DEFAULT_TEMPLATES_DIR = REPO_ROOT / "ansible" / "roles"
def _find_compose_templates(base: Path) -> list[Path]:
"""Find all Jinja2 docker-compose templates under a base directory."""
if base.is_file():
return [base]
if not base.is_dir():
return []
results: list[Path] = []
for pattern in ("*docker-compose*", "*compose*"):
results.extend(base.rglob(f"{pattern}.yml.j2"))
results.extend(base.rglob(f"{pattern}.yaml.j2"))
# Also check exporters-compose
results.extend(base.rglob("exporters-compose*.j2"))
# Deduplicate while preserving order
seen: set[Path] = set()
unique: list[Path] = []
for p in sorted(results):
if p not in seen:
seen.add(p)
unique.append(p)
return unique
def _parse_services(content: str) -> dict[str, list[str]]:
"""Parse service blocks from a docker-compose Jinja2 template.
Returns a mapping of service_name list of lines in that service block.
"""
lines = content.splitlines()
in_services = False
services: dict[str, list[str]] = {}
current_svc: str | None = None
current_lines: list[str] = []
for line in lines:
if line.startswith("services:"):
in_services = True
continue
if not in_services:
continue
# Top-level keys (networks:, volumes:) end the services section
if re.match(r"^(networks|volumes):\s*$", line):
if current_svc is not None:
services[current_svc] = current_lines
current_svc = None
in_services = False
continue
# Service definition: exactly 2-space indent, ends with :
# Service names can contain Jinja2 variables like {{ app_name }}
# or {{ app_name }}-db. Match: 2-space indent + non-whitespace
# chars (including {{ }}, -, _, .) + optional spaces inside {{ }} + :
m = re.match(r"^ (\{\{.*?\}\}[a-zA-Z0-9_-]*|[a-zA-Z0-9_().-]+):\s*$", line)
if m:
if current_svc is not None:
services[current_svc] = current_lines
current_svc = m.group(1)
current_lines = []
elif current_svc is not None:
current_lines.append(line)
if current_svc is not None:
services[current_svc] = current_lines
return services
def _check_template(filepath: Path, repo_root: Path) -> list[str]:
"""Check a single docker-compose template for missing init: true.
Returns a list of error messages (empty if all OK).
"""
errors: list[str] = []
content = filepath.read_text(encoding="utf-8")
if "services:" not in content:
return errors
services = _parse_services(content)
for svc_name, svc_lines in services.items():
svc_text = "\n".join(svc_lines)
has_init = "init: true" in svc_text
has_healthcheck = "healthcheck:" in svc_text
# Skip services that are conditionally included (Jinja2 if blocks)
# but still check them — the healthcheck is inside the conditional
if has_healthcheck and not has_init:
try:
display_path = filepath.relative_to(repo_root)
except ValueError:
display_path = filepath
errors.append(
f"{display_path}: service '{svc_name}' has a healthcheck "
f"but no 'init: true'. Without init: true, CMD-SHELL "
f"healthchecks (wget, pgrep) spawn children that become "
f"zombies when PID 1 doesn't reap them. Add 'init: true' "
f"to enable Docker's built-in tini as PID 1."
)
return errors
@click.command()
@click.option(
"--path",
type=click.Path(exists=True, path_type=Path),
help="Check a specific file or directory (default: ansible/roles/).",
)
@click.option(
"--templates-dir",
type=click.Path(exists=True, path_type=Path),
default=None,
help="Override the default templates directory (default: ansible/roles/).",
)
def main(path: Path | None, templates_dir: Path | None) -> None:
"""Check that Docker Compose services with healthchecks have init: true."""
tdir = templates_dir or DEFAULT_TEMPLATES_DIR
files = _find_compose_templates(path) if path else _find_compose_templates(tdir)
all_errors: list[str] = []
for f in files:
errors = _check_template(f, tdir)
all_errors.extend(errors)
if all_errors:
click.echo("[check-docker-init] FAIL: services with healthchecks missing init: true:")
for err in all_errors:
click.echo(f" - {err}")
sys.exit(1)
else:
click.echo("[check-docker-init] OK: all services with healthchecks have init: true.")
if __name__ == "__main__": # pragma: no cover
main()
+65
View File
@@ -0,0 +1,65 @@
"""Validate Jinja2 expressions in Ansible files by rendering them.
Thin wrapper around :mod:`devx.tools.ansible_checks.jinja_expr` for
backward compatibility. The check logic lives in the subpackage; this
module preserves the CLI entry point and re-exports the internal
helpers so existing tests and imports continue to work.
Usage::
python -m devx.tools.check_jinja_expr
python -m devx.tools.check_jinja_expr --path ansible/playbooks/deploy-observability.yml
Exit code 0 if all renderable expressions pass, 1 if any fail.
"""
from __future__ import annotations
import sys
from pathlib import Path
import click
from devx.tools.ansible_checks.jinja_expr import (
EXPR_PATTERN, # noqa: F401 — re-exported for backward compat
MOCK_CONTEXT, # noqa: F401
_check_file, # noqa: F401
_default_ansible_dirs, # noqa: F401
_extract_expressions, # noqa: F401
_find_yaml_files, # noqa: F401
_render_expression, # noqa: F401
check_jinja_expr, # noqa: F401
)
@click.command()
@click.option(
"--path",
type=click.Path(exists=True, path_type=Path),
help="Check a specific file or directory (default: ansible/playbooks + ansible/roles).",
)
@click.option(
"--ansible-dir",
"ansible_dirs",
type=click.Path(exists=True, path_type=Path),
multiple=True,
default=None,
help="Override the default ansible directories (can be repeated). Defaults to ansible/playbooks and ansible/roles.",
)
def main(path: Path | None, ansible_dirs: tuple[Path, ...]) -> None:
"""Validate Jinja2 expressions in Ansible files."""
dirs = list(ansible_dirs) if ansible_dirs else _default_ansible_dirs()
all_violations = check_jinja_expr(path) if path else check_jinja_expr(None, dirs)
if all_violations:
click.echo("[check-jinja-expr] FAIL: invalid Jinja expressions found:")
for v in all_violations:
click.echo(f" - {v}")
click.echo("\nFix: test expressions with `ansible localhost -m debug -a 'msg={{ <expr> }}'`")
sys.exit(1)
else:
click.echo("[check-jinja-expr] OK: all Jinja expressions render correctly.")
if __name__ == "__main__": # pragma: no cover
main()
+69 -4
View File
@@ -22,18 +22,37 @@ Usage::
from __future__ import annotations
import logging
import os
import platform
import shutil
import tarfile
import tempfile
import time
import urllib.error
import urllib.request
from pathlib import Path
import click
from tenacity import (
Retrying,
before_sleep_log,
retry_if_exception_type,
stop_after_attempt,
wait_exponential,
)
TARGET_DIR = Path.home() / ".local" / "bin"
logger = logging.getLogger(__name__)
# Retry configuration for transient network failures during download.
# GitHub releases occasionally drops connections ("Remote end closed
# connection without response"). Retrying with backoff before falling
# through to the next fallback URL makes the build resilient to
# momentary network blips.
MAX_DOWNLOAD_RETRIES = 3
ACTIONLINT_VERSION = "1.7.12"
GIT_CLIFF_VERSION = "2.13.1"
@@ -64,12 +83,53 @@ def _ensure_target_dir() -> Path:
return TARGET_DIR
def _download(url: str, dest: Path) -> None:
"""Download a file from ``url`` to ``dest`` with a 60s timeout."""
def _download(url: str, dest: Path, *, _sleep=None) -> None:
"""Download a file from ``url`` to ``dest`` with retry and 60s timeout.
Retries up to ``MAX_DOWNLOAD_RETRIES`` times on transient network
errors (``URLError``, ``OSError`` from connection resets) using
exponential backoff. This handles momentary GitHub releases
connection drops that were causing CI image builds to fail.
The ``_sleep`` kwarg is for tests to avoid real sleeping; production
code should leave it as ``None`` (uses ``time.sleep``).
"""
retrying = Retrying(
stop=stop_after_attempt(MAX_DOWNLOAD_RETRIES),
wait=wait_exponential(multiplier=2, min=2, max=10),
retry=retry_if_exception_type((urllib.error.URLError, OSError, ConnectionError)),
before_sleep=before_sleep_log(logger, logging.WARNING),
sleep=_sleep if _sleep is not None else time.sleep,
reraise=True,
)
retrying(_do_download, url, dest)
def _do_download(url: str, dest: Path) -> None:
"""Single download attempt — called by :func:`_download` retry wrapper."""
with urllib.request.urlopen(url, timeout=60) as resp, open(dest, "wb") as f: # nosec B310
shutil.copyfileobj(resp, f)
def _download_with_fallback(urls: list[str], binary_name: str) -> Path:
"""Try downloading a binary from a list of URLs, falling back on failure.
Returns the path to the installed binary. Raises if all URLs fail.
"""
target_dir = _ensure_target_dir()
dest = target_dir / binary_name
errors: list[str] = []
for url in urls:
try:
_download(url, dest)
dest.chmod(0o755)
return dest
except Exception as exc: # noqa: BLE001
errors.append(f"{url}: {exc}")
click.echo(f" {binary_name}: retrying — {exc}")
raise click.ClickException(f"Failed to download {binary_name} from all URLs: {'; '.join(errors)}")
def _download_and_extract_tarball(url: str, binary_name: str) -> Path:
"""Download a tarball, extract the binary, and install it to TARGET_DIR.
@@ -164,8 +224,13 @@ def install_tea() -> bool:
click.echo("tea: already installed")
return True
arch = _arch()
url = f"https://dl.gitea.com/tea/{TEA_VERSION}/tea-{TEA_VERSION}-linux-{arch}"
dest = _download_binary(url, "tea")
# dl.gitea.com is the primary CDN, but it can return 403 from some networks.
# Fall back to the gitea.com release downloads URL.
urls = [
f"https://dl.gitea.com/tea/{TEA_VERSION}/tea-{TEA_VERSION}-linux-{arch}",
f"https://gitea.com/gitea/tea/releases/download/v{TEA_VERSION}/tea-{TEA_VERSION}-linux-{arch}",
]
dest = _download_with_fallback(urls, "tea")
click.echo(f"tea: installed to {dest}")
return True
+12 -2
View File
@@ -15,6 +15,7 @@ from pathlib import Path
import click
from dotenv import load_dotenv # pyright: ignore[reportMissingImports,reportUnknownVariableType]
from tenacity import retry, stop_after_attempt, wait_exponential
from devx.tokens import get_developer_token
@@ -56,13 +57,22 @@ def _install_pre_commit_hooks(bin_dir: str) -> None:
def _install_ansible_collections(bin_dir: str) -> None:
"""Install required Ansible Galaxy collections if requirements exist."""
"""Install required Ansible Galaxy collections if requirements exist.
Retries up to 3 times with exponential backoff to handle transient
network timeouts when contacting galaxy.ansible.com.
"""
galaxy = shutil.which("ansible-galaxy") or str(Path(bin_dir) / "ansible-galaxy")
requirements = Path("ansible/requirements.yml")
if not requirements.exists():
click.echo(" ansible/requirements.yml not found — skipping collections.")
return
_run([galaxy, "collection", "install", "-r", str(requirements)])
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=2, min=2, max=10), reraise=True)
def _do_install() -> None:
_run([galaxy, "collection", "install", "-r", str(requirements)])
_do_install()
def _configure_tea_login() -> None:
+3 -1
View File
@@ -64,9 +64,11 @@ def _install_in_image(
link.symlink_to(opt_venv)
# Build pip install command
# --no-deps: the CI image already has all dependencies pre-installed.
# We only need to install the project itself in editable mode.
spec = f".[{extras}]" if extras else "."
pip_bin = str(Path(venv_link) / "bin" / "pip")
cmd = [pip_bin, "install", "--no-cache-dir", "-e", spec]
cmd = [pip_bin, "install", "--no-cache-dir", "--no-deps", "-e", spec]
env = os.environ.copy()
try:
+4121 -3523
View File
File diff suppressed because it is too large Load Diff
+94 -6
View File
@@ -1,14 +1,26 @@
#!/usr/bin/env python3
"""Utilities for handling API response values.
"""Utilities for handling API response values and base HTTP API client.
Many APIs return boolean values as strings (``"true"``, ``"false"``)
rather than native JSON booleans. The Mattermost ``/api/v4/config/client``
endpoint is a notable example. These helpers handle both string and
boolean responses safely.
This module provides two categories of utilities:
1. **Response helpers** :func:`is_truthy` and :func:`is_falsy` handle
APIs that return boolean values as strings (``"true"``, ``"false"``)
rather than native JSON booleans.
2. **Base API client** :class:`APIClient` provides a reusable base
class for HTTP API clients with consistent timeout handling, header
propagation, and automatic raising on 4xx/5xx responses.
Usage::
from devx.utils.api import is_truthy, is_falsy
from devx.utils.api import APIClient, is_truthy
class MyClient(APIClient):
def __init__(self):
super().__init__(
base_url="https://api.example.com",
headers={"Authorization": "Bearer token"},
)
if not is_truthy(config.get("EnableOpenServer")):
raise ValueError("EnableOpenServer not enabled")
@@ -16,6 +28,82 @@ Usage::
from __future__ import annotations
import requests
class APIClient:
"""Base class for HTTP API clients.
Subclasses set ``base_url``, ``headers``, and optionally ``auth`` in
their constructor, then use :meth:`_request` or the convenience
methods (:meth:`get`, :meth:`post`, etc.) to make requests.
All requests raise :class:`requests.HTTPError` on 4xx/5xx responses
via :meth:`requests.Response.raise_for_status`.
"""
def __init__(
self,
base_url: str,
headers: dict,
timeout: int = 30,
verify: bool = True,
auth: tuple[str, str] | None = None,
) -> None:
"""Initialize the API client.
Args:
base_url: Base URL for the API (trailing slash stripped).
headers: Default headers sent with every request.
timeout: Request timeout in seconds.
verify: Whether to verify TLS certificates.
auth: Optional ``(username, password)`` tuple for basic auth.
"""
self.base_url = base_url.rstrip("/")
self.headers = headers
self.timeout = timeout
self.verify = verify
self.auth = auth
def _request(self, method: str, path: str, **kwargs) -> requests.Response:
"""Execute an HTTP request against the API.
The URL is constructed as ``{base_url}{path}``. Default timeout,
verify, auth, and headers are applied but can be overridden via
``kwargs``.
Raises:
requests.HTTPError: On 4xx/5xx response status codes.
"""
url = f"{self.base_url}{path}"
kwargs.setdefault("timeout", self.timeout)
kwargs.setdefault("verify", self.verify)
if self.auth is not None:
kwargs.setdefault("auth", self.auth)
resp = requests.request(method, url, headers=self.headers, **kwargs) # noqa: S113
resp.raise_for_status()
return resp
def get(self, path: str, **kwargs) -> requests.Response:
"""Send a GET request."""
return self._request("GET", path, **kwargs)
def post(self, path: str, **kwargs) -> requests.Response:
"""Send a POST request."""
return self._request("POST", path, **kwargs)
def put(self, path: str, **kwargs) -> requests.Response:
"""Send a PUT request."""
return self._request("PUT", path, **kwargs)
def delete(self, path: str, **kwargs) -> requests.Response:
"""Send a DELETE request."""
return self._request("DELETE", path, **kwargs)
def patch(self, path: str, **kwargs) -> requests.Response:
"""Send a PATCH request."""
return self._request("PATCH", path, **kwargs)
def is_truthy(value: str | bool | None) -> bool:
"""Check if an API config value is truthy.
+133
View File
@@ -0,0 +1,133 @@
"""Shared Jinja2 environment helpers for unit tests and template rendering.
Creating a Jinja2 Environment is expensive (filesystem scanning, template
compilation). These helpers create cached environments with
``auto_reload=False`` to skip stat() calls on every ``get_template``,
which is the single biggest speedup for template-heavy test suites.
The filters mimic Ansible builtins not available in plain Jinja2,
making it possible to render Ansible templates outside of Ansible
(e.g. in unit tests or config generation scripts).
Usage::
from devx.utils.jinja import make_env, render_template
env = make_env("/path/to/templates")
output = render_template(env, "alert-rules.yml.j2", grafana_base_url="https://grafana.example.com")
"""
from __future__ import annotations
import functools
import json
import re
import jinja2
# ---------------------------------------------------------------------------
# Filters (mimic Ansible builtins not available in plain Jinja2)
# ---------------------------------------------------------------------------
def to_json(value) -> str:
return json.dumps(value)
def to_bool(value) -> bool:
"""Mimic Ansible's |bool filter for plain Jinja2 tests."""
if isinstance(value, bool):
return value
if isinstance(value, str):
return value.lower() not in ("", "false", "0", "no", "off", "null", "none")
return bool(value)
def regex_replace(value, pattern: str, replacement: str) -> str:
"""Mimic Ansible's |regex_replace filter."""
return re.sub(pattern, replacement, str(value))
def regex_escape(value) -> str:
"""Mimic Ansible's |regex_escape filter."""
return re.escape(str(value))
def regex_search(value, pattern: str) -> str | None:
"""Mimic Ansible's |regex_search filter.
Returns the first match (group 0) or None if no match.
Ansible returns the full match string or None.
"""
m = re.search(pattern, str(value))
return m.group(0) if m else None
# ---------------------------------------------------------------------------
# Environment factory
# ---------------------------------------------------------------------------
_FILTERS = {
"to_json": to_json,
"bool": to_bool,
"regex_replace": regex_replace,
"regex_escape": regex_escape,
"regex_search": regex_search,
}
@functools.cache
def make_env(loader_path: str) -> jinja2.Environment:
"""Create a cached Jinja2 Environment with standard filters.
``auto_reload=False`` skips stat() on every get_template call
templates don't change during a test run so this is safe and
cuts ~40% off render time.
"""
env = jinja2.Environment( # nosec B701 — renders YAML/config templates, not HTML
loader=jinja2.FileSystemLoader(loader_path),
undefined=jinja2.StrictUndefined,
auto_reload=False,
cache_size=400,
)
env.filters.update(_FILTERS)
return env
@functools.cache
def make_value_env() -> jinja2.Environment:
"""Cached environment for rendering individual manifest string values."""
env = jinja2.Environment( # nosec B701 — renders config values, not HTML
undefined=jinja2.ChainableUndefined,
auto_reload=False,
)
env.filters.update(_FILTERS)
return env
# ---------------------------------------------------------------------------
# Render helpers
# ---------------------------------------------------------------------------
def render_template(env: jinja2.Environment, template_name: str, **kwargs) -> str:
"""Render a named template from a FileSystemLoader-backed env."""
return env.get_template(template_name).render(**kwargs)
def render_value(value, ctx: dict):
"""Render a single string value as a Jinja2 template if it contains expressions."""
if not isinstance(value, str):
return value
if "{{" not in value and "{%" not in value:
return value
return make_value_env().from_string(value).render(**ctx)
def render_manifest_values(obj, ctx: dict):
"""Recursively render all Jinja2 expressions in manifest string values."""
if isinstance(obj, dict):
return {k: render_manifest_values(v, ctx) for k, v in obj.items()}
if isinstance(obj, list):
return [render_manifest_values(v, ctx) for v in obj]
return render_value(obj, ctx)
+79
View File
@@ -0,0 +1,79 @@
"""User-facing output utilities combining console and log output.
Console messages are colorised via ``click.style`` for visual feedback.
The persistent log file always receives plain text (no ANSI codes).
This is a generalisation of grm's ``ui.say()`` function, extracted so
that any CLI tool can use the same pattern. The logger name and
console-level env var are configurable.
Usage::
from devx.utils.ui import say
say("Starting deployment...")
say("Error occurred", level=logging.ERROR, err=True, color="red")
"""
from __future__ import annotations
import logging
import os
import click
# Configurable env var for console verbosity — projects can override
# via :func:`configure_ui`.
_LOG_LEVEL_ENV_VAR = "DEVX_LOG_LEVEL"
_LOGGER_NAME = "devx"
def configure_ui(*, log_level_env_var: str = "DEVX_LOG_LEVEL", logger_name: str = "devx") -> None:
"""Override the env var name and logger name used by :func:`say`.
This allows downstream projects (e.g. grm) to use their own env var
names (e.g. ``GRM_LOG_LEVEL``) and logger names while still using
devx's ui module.
Args:
log_level_env_var: Environment variable name for console log level.
logger_name: Logger name for persistent log file output.
"""
global _LOG_LEVEL_ENV_VAR, _LOGGER_NAME
_LOG_LEVEL_ENV_VAR = log_level_env_var
_LOGGER_NAME = logger_name
def _console_level() -> int:
"""Return the minimum level for console output from the configured env var."""
value = os.getenv(_LOG_LEVEL_ENV_VAR, "INFO")
try:
return getattr(logging, value.upper())
except AttributeError:
return logging.INFO
def say(
msg: str,
level: int = logging.INFO,
err: bool = False,
color: str | None = None,
) -> None:
"""Output a message to the user and also log it for auditing.
Console output goes via ``click.echo`` (handles encoding, CliRunner,
Windows colorama) only when *level* is at least the configured
console log level (default ``DEVX_LOG_LEVEL``, falls back to INFO).
The same message is always sent to the configured logger so it
appears in the persistent log file regardless of console verbosity.
Args:
msg: Message to display.
level: Logging level (e.g. ``logging.INFO``, ``logging.ERROR``).
err: If True, output to stderr instead of stdout.
color: Optional ``click.style`` fg color (e.g. ``"green"``, ``"red"``).
"""
if level >= _console_level():
styled = click.style(msg, fg=color) if color else msg
click.echo(styled, err=err)
logging.getLogger(_LOGGER_NAME).log(level, msg)
+231
View File
@@ -0,0 +1,231 @@
"""Unit tests for devx.tools.ansible_checks._shared."""
from pathlib import Path
import pytest
from devx.tools.ansible_checks._shared import (
DEFAULT_ANSIBLE_DIRS,
AnsibleFileFinder,
AnsibleYAMLParser,
ViolationReporter,
)
class TestAnsibleFileFinder:
def test_find_task_files_single_yaml(self, tmp_path: Path) -> None:
f = tmp_path / "test.yml"
f.write_text("tasks: []")
assert AnsibleFileFinder.find_task_files(f) == [f]
def test_find_task_files_single_non_yaml(self, tmp_path: Path) -> None:
f = tmp_path / "test.txt"
f.write_text("hello")
assert AnsibleFileFinder.find_task_files(f) == []
def test_find_task_files_dir(self, tmp_path: Path) -> None:
(tmp_path / "a.yml").write_text("tasks: []")
(tmp_path / "b.yaml").write_text("tasks: []")
(tmp_path / "c.txt").write_text("hello")
result = AnsibleFileFinder.find_task_files(tmp_path)
assert len(result) == 2
assert all(f.suffix in (".yml", ".yaml") for f in result)
def test_find_task_files_skip_molecule(self, tmp_path: Path) -> None:
(tmp_path / "a.yml").write_text("tasks: []")
mol = tmp_path / "molecule" / "default"
mol.mkdir(parents=True)
(mol / "main.yml").write_text("tasks: []")
result = AnsibleFileFinder.find_task_files(tmp_path, skip_molecule=True)
assert len(result) == 1
assert "molecule" not in result[0].parts
def test_find_task_files_include_molecule(self, tmp_path: Path) -> None:
(tmp_path / "a.yml").write_text("tasks: []")
mol = tmp_path / "molecule" / "default"
mol.mkdir(parents=True)
(mol / "main.yml").write_text("tasks: []")
result = AnsibleFileFinder.find_task_files(tmp_path, skip_molecule=False)
assert len(result) == 2
def test_find_task_files_nonexistent(self, tmp_path: Path) -> None:
assert AnsibleFileFinder.find_task_files(tmp_path / "nonexistent") == []
def test_find_yaml_files_single_file(self, tmp_path: Path) -> None:
f = tmp_path / "test.txt"
f.write_text("hello")
# find_yaml_files accepts any single file (no suffix check)
assert AnsibleFileFinder.find_yaml_files(f) == [f]
def test_find_yaml_files_dir(self, tmp_path: Path) -> None:
(tmp_path / "a.yml").write_text("tasks: []")
(tmp_path / "sub").mkdir()
(tmp_path / "sub" / "b.yaml").write_text("tasks: []")
result = AnsibleFileFinder.find_yaml_files(tmp_path)
assert len(result) == 2
def test_find_yaml_files_skip_molecule(self, tmp_path: Path) -> None:
(tmp_path / "a.yml").write_text("tasks: []")
mol = tmp_path / "molecule" / "default"
mol.mkdir(parents=True)
(mol / "main.yml").write_text("tasks: []")
result = AnsibleFileFinder.find_yaml_files(tmp_path, skip_molecule=True)
assert len(result) == 1
def test_find_yaml_files_include_molecule(self, tmp_path: Path) -> None:
(tmp_path / "a.yml").write_text("tasks: []")
mol = tmp_path / "molecule" / "default"
mol.mkdir(parents=True)
(mol / "main.yml").write_text("tasks: []")
result = AnsibleFileFinder.find_yaml_files(tmp_path, skip_molecule=False)
assert len(result) == 2
def test_find_task_and_playbook_files(self, tmp_path: Path) -> None:
role = tmp_path / "roles" / "myrole"
(role / "tasks").mkdir(parents=True)
(role / "tasks" / "main.yml").write_text("tasks: []")
pb = tmp_path / "playbooks"
pb.mkdir()
(pb / "deploy.yml").write_text("tasks: []")
(tmp_path / "random.yml").write_text("tasks: []")
result = AnsibleFileFinder.find_task_and_playbook_files(tmp_path)
# Should find tasks/main.yml and playbooks/deploy.yml, not random.yml
names = [f.name for f in result]
assert "main.yml" in names
assert "deploy.yml" in names
assert "random.yml" not in names
def test_find_task_and_playbook_files_skip_molecule(self, tmp_path: Path) -> None:
role = tmp_path / "roles" / "myrole"
(role / "tasks").mkdir(parents=True)
(role / "tasks" / "main.yml").write_text("tasks: []")
mol = role / "molecule" / "default" / "tasks"
mol.mkdir(parents=True)
(mol / "main.yml").write_text("tasks: []")
result = AnsibleFileFinder.find_task_and_playbook_files(tmp_path, skip_molecule=True)
assert len(result) == 1
assert "molecule" not in result[0].parts
class TestAnsibleYAMLParser:
def test_parse_file_valid(self) -> None:
content = "---\n- name: test\n shell: echo hi\n"
docs = AnsibleYAMLParser.parse_file(content)
assert len(docs) == 1
assert isinstance(docs[0], list)
def test_parse_file_multi_doc(self) -> None:
content = "---\n- a\n---\n- b\n"
docs = AnsibleYAMLParser.parse_file(content)
assert len(docs) == 2
def test_parse_file_empty_docs_filtered(self) -> None:
content = "---\n- a\n---\n\n"
docs = AnsibleYAMLParser.parse_file(content)
assert len(docs) == 1
def test_parse_file_yaml_error(self) -> None:
content = "{{ invalid: ["
docs = AnsibleYAMLParser.parse_file(content)
assert docs == []
def test_iter_tasks_bare_list(self) -> None:
doc = [{"name": "task1", "shell": "echo hi"}, {"name": "task2", "shell": "echo bye"}]
tasks = list(AnsibleYAMLParser.iter_tasks(doc))
assert len(tasks) == 2
assert tasks[0][0]["name"] == "task1"
assert tasks[0][1] == 1
assert tasks[1][1] == 2
def test_iter_tasks_play_dict(self) -> None:
doc = {"hosts": "all", "tasks": [{"name": "task1", "shell": "echo hi"}]}
tasks = list(AnsibleYAMLParser.iter_tasks(doc))
assert len(tasks) == 1
assert tasks[0][0]["name"] == "task1"
def test_iter_tasks_play_with_pre_post_handlers(self) -> None:
doc = {
"hosts": "all",
"pre_tasks": [{"name": "pre", "shell": "echo pre"}],
"tasks": [{"name": "main", "shell": "echo main"}],
"post_tasks": [{"name": "post", "shell": "echo post"}],
"handlers": [{"name": "handler", "shell": "echo handler"}],
}
tasks = list(AnsibleYAMLParser.iter_tasks(doc))
assert len(tasks) == 4
names = [t[0]["name"] for t in tasks]
# Order: tasks, pre_tasks, post_tasks, handlers (as defined in _iter_play_sections)
assert names == ["main", "pre", "post", "handler"]
def test_iter_tasks_block(self) -> None:
doc = [{"name": "outer", "block": [{"name": "inner", "shell": "echo hi"}]}]
tasks = list(AnsibleYAMLParser.iter_tasks(doc))
# outer is not a play (no task sections) → yielded as bare task
# inner is yielded from block
assert len(tasks) == 2
assert tasks[0][0]["name"] == "outer"
assert tasks[1][0]["name"] == "inner"
def test_iter_tasks_block_in_play_section(self) -> None:
"""Block tasks within a play's tasks section are yielded."""
doc = {
"hosts": "all",
"tasks": [
{"name": "outer", "block": [{"name": "inner", "shell": "echo hi"}]},
],
}
tasks = list(AnsibleYAMLParser.iter_tasks(doc))
assert len(tasks) == 2
assert tasks[0][0]["name"] == "outer"
assert tasks[1][0]["name"] == "inner"
def test_iter_tasks_play_list(self) -> None:
doc = [{"hosts": "all", "tasks": [{"name": "task1", "shell": "echo hi"}]}]
tasks = list(AnsibleYAMLParser.iter_tasks(doc))
assert len(tasks) == 1
assert tasks[0][0]["name"] == "task1"
def test_iter_tasks_non_dict_items_skipped(self) -> None:
doc = ["string", 42, {"name": "task1", "shell": "echo hi"}]
tasks = list(AnsibleYAMLParser.iter_tasks(doc))
assert len(tasks) == 1
class TestViolationReporter:
def test_format_violation_with_line(self, tmp_path: Path) -> None:
result = ViolationReporter.format_violation(tmp_path / "foo.yml", tmp_path, 42, "bad")
assert result == "foo.yml:42 — bad"
def test_format_violation_without_line(self, tmp_path: Path) -> None:
result = ViolationReporter.format_violation(tmp_path / "foo.yml", tmp_path, None, "bad")
assert result == "foo.yml — bad"
def test_format_violation_not_relative(self, tmp_path: Path) -> None:
other = Path("/other/path")
result = ViolationReporter.format_violation(other, tmp_path, 1, "bad")
assert str(other) in result
assert "bad" in result
def test_report_no_violations(self, capsys: pytest.CaptureFixture[str]) -> None:
ViolationReporter.report([], "test-tool")
captured = capsys.readouterr()
assert "OK" in captured.out
assert "test-tool" in captured.out
def test_report_with_violations(self, capsys: pytest.CaptureFixture[str]) -> None:
with pytest.raises(SystemExit) as exc_info:
ViolationReporter.report(["v1", "v2"], "test-tool")
assert exc_info.value.code == 1
captured = capsys.readouterr()
assert "FAIL" in captured.out
assert "v1" in captured.out
assert "v2" in captured.out
class TestDefaultAnsibleDirs:
def test_is_tuple(self) -> None:
assert isinstance(DEFAULT_ANSIBLE_DIRS, tuple)
def test_contains_expected(self) -> None:
assert "ansible/roles" in DEFAULT_ANSIBLE_DIRS
assert "ansible/playbooks" in DEFAULT_ANSIBLE_DIRS
@@ -0,0 +1,172 @@
"""Unit tests for devx.ci.cancel_superseded_runs."""
from __future__ import annotations
import json
import urllib.error
from unittest.mock import MagicMock, patch
import pytest
import devx.ci.cancel_superseded_runs as mod
from devx.ci.cancel_superseded_runs import _api_request, cancel_run, list_running_runs, main
_HTTP_NO_CONTENT = mod._HTTP_NO_CONTENT
_PAGE_SIZE = mod._PAGE_SIZE
class TestConstants:
def test_http_no_content_is_204(self) -> None:
assert _HTTP_NO_CONTENT == 204
def test_page_size_is_50(self) -> None:
assert _PAGE_SIZE == 50
class TestApiRequest:
def test_returns_empty_for_204(self) -> None:
mock_resp = MagicMock()
mock_resp.status = _HTTP_NO_CONTENT
mock_resp.read.return_value = b""
mock_resp.__enter__ = MagicMock(return_value=mock_resp)
mock_resp.__exit__ = MagicMock(return_value=None)
with patch("urllib.request.urlopen", return_value=mock_resp):
result = _api_request("POST", "/repos/test/actions/runs/1/cancel", "tok", "https://x")
assert result == {}
def test_returns_json_for_200(self) -> None:
mock_resp = MagicMock()
mock_resp.status = 200
mock_resp.read.return_value = json.dumps({"id": 1}).encode()
mock_resp.__enter__ = MagicMock(return_value=mock_resp)
mock_resp.__exit__ = MagicMock(return_value=None)
with patch("urllib.request.urlopen", return_value=mock_resp):
result = _api_request("GET", "/repos/test/actions/runs", "tok", "https://x")
assert result == {"id": 1}
def test_http_error_raises(self) -> None:
err = urllib.error.HTTPError("x", 500, "err", {}, None)
err.read = MagicMock(return_value=b"error body")
with patch("urllib.request.urlopen", side_effect=err):
with pytest.raises(urllib.error.HTTPError):
_api_request("GET", "/repos/test/actions/runs", "tok", "https://x")
def test_url_error_raises(self) -> None:
with patch("urllib.request.urlopen", side_effect=urllib.error.URLError("fail")):
with pytest.raises(urllib.error.URLError):
_api_request("GET", "/repos/test/actions/runs", "tok", "https://x")
class TestListRunningRuns:
def test_paginates_until_empty(self) -> None:
page1 = {"workflow_runs": [{"id": 1}, {"id": 2}], "total_count": 2}
page2 = {"workflow_runs": [], "total_count": 2}
responses = iter([page1, page2])
with patch.object(mod, "_api_request", side_effect=lambda *a, **k: next(responses)):
runs = list_running_runs("owner/repo", "tok", "https://x")
assert len(runs) == 2
def test_empty_first_page(self) -> None:
with patch.object(mod, "_api_request", return_value={"workflow_runs": [], "total_count": 0}):
runs = list_running_runs("owner/repo", "tok", "https://x")
assert runs == []
def test_stops_at_page_size(self) -> None:
full_page = {"workflow_runs": [{"id": i} for i in range(_PAGE_SIZE)], "total_count": _PAGE_SIZE + 1}
half_page = {"workflow_runs": [{"id": 99}], "total_count": _PAGE_SIZE + 1}
responses = iter([full_page, half_page])
with patch.object(mod, "_api_request", side_effect=lambda *a, **k: next(responses)):
runs = list_running_runs("owner/repo", "tok", "https://x")
assert len(runs) == _PAGE_SIZE + 1
def test_uses_in_progress_status(self) -> None:
with patch.object(mod, "_api_request", return_value={"workflow_runs": [], "total_count": 0}) as mock_req:
list_running_runs("owner/repo", "tok", "https://x")
path = mock_req.call_args.args[1]
assert "status=in_progress" in path
assert "status=running" not in path
def test_accepts_bare_list(self) -> None:
with patch.object(mod, "_api_request", return_value=[{"id": 1}, {"id": 2}]):
runs = list_running_runs("owner/repo", "tok", "https://x")
assert len(runs) == 2
class TestCancelRun:
def test_success_returns_true(self) -> None:
with patch.object(mod, "_api_request", return_value={}):
assert cancel_run("owner/repo", 123, "tok", "https://x") is True
def test_http_error_returns_false(self) -> None:
with patch.object(mod, "_api_request", side_effect=urllib.error.HTTPError("x", 500, "err", {}, None)):
assert cancel_run("owner/repo", 123, "tok", "https://x") is False
class TestMain:
def test_no_token_exits_zero(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("CI_GITEA_API_TOKEN", raising=False)
monkeypatch.delenv("CI_GITEA_TOKEN", raising=False)
monkeypatch.setattr("sys.argv", ["cancel", "--repo", "o/r", "--current-run-id", "1", "--head-branch", "feat"])
assert main() == 0
def test_no_superseded_runs(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CI_GITEA_API_TOKEN", "tok")
monkeypatch.setattr("sys.argv", ["cancel", "--repo", "o/r", "--current-run-id", "10", "--head-branch", "feat"])
with patch.object(mod, "list_running_runs", return_value=[]):
assert main() == 0
def test_cancels_superseded(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CI_GITEA_API_TOKEN", "tok")
runs = [
{"id": 5, "head_branch": "feat"},
{"id": 8, "head_branch": "feat"},
{"id": 12, "head_branch": "other"},
]
monkeypatch.setattr("sys.argv", ["cancel", "--repo", "o/r", "--current-run-id", "10", "--head-branch", "feat"])
with patch.object(mod, "list_running_runs", return_value=runs):
with patch.object(mod, "cancel_run", return_value=True) as mock_cancel:
assert main() == 0
cancelled_ids = [call.args[1] for call in mock_cancel.call_args_list]
assert cancelled_ids == [5, 8]
def test_dry_run_does_not_cancel(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CI_GITEA_API_TOKEN", "tok")
runs = [{"id": 5, "head_branch": "feat"}]
monkeypatch.setattr(
"sys.argv",
["cancel", "--repo", "o/r", "--current-run-id", "10", "--head-branch", "feat", "--dry-run"],
)
with patch.object(mod, "list_running_runs", return_value=runs):
with patch.object(mod, "cancel_run", return_value=True) as mock_cancel:
assert main() == 0
assert mock_cancel.call_count == 0
def test_cancel_failure_continues(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CI_GITEA_API_TOKEN", "tok")
runs = [{"id": 5, "head_branch": "feat"}, {"id": 8, "head_branch": "feat"}]
monkeypatch.setattr("sys.argv", ["cancel", "--repo", "o/r", "--current-run-id", "10", "--head-branch", "feat"])
with patch.object(mod, "list_running_runs", return_value=runs):
with patch.object(mod, "cancel_run", side_effect=[False, True]):
assert main() == 0
def test_404_returns_zero(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CI_GITEA_API_TOKEN", "tok")
monkeypatch.setattr("sys.argv", ["cancel", "--repo", "o/r", "--current-run-id", "10", "--head-branch", "feat"])
err = urllib.error.HTTPError("x", 404, "Not Found", {}, None)
with patch.object(mod, "list_running_runs", side_effect=err):
assert main() == 0
def test_400_returns_zero(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CI_GITEA_API_TOKEN", "tok")
monkeypatch.setattr("sys.argv", ["cancel", "--repo", "o/r", "--current-run-id", "10", "--head-branch", "feat"])
err = urllib.error.HTTPError("x", 400, "Bad Request", {}, None)
with patch.object(mod, "list_running_runs", side_effect=err):
assert main() == 0
def test_500_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CI_GITEA_API_TOKEN", "tok")
monkeypatch.setattr("sys.argv", ["cancel", "--repo", "o/r", "--current-run-id", "10", "--head-branch", "feat"])
err = urllib.error.HTTPError("x", 500, "Server Error", {}, None)
with patch.object(mod, "list_running_runs", side_effect=err):
with pytest.raises(urllib.error.HTTPError):
main()
@@ -0,0 +1,419 @@
"""Unit tests for devx.ci.check_workflow_artifact_deps."""
from __future__ import annotations
import textwrap
from pathlib import Path
from click.testing import CliRunner
from devx.ci.check_workflow_artifact_deps import (
_check_workflow,
_extract_artifact_info,
_is_artifact_action,
main,
)
class TestIsArtifactAction:
def test_upload_action_gitea(self):
assert _is_artifact_action("christopherhx/gitea-upload-artifact@v4", ("upload-artifact",))
def test_upload_action_github(self):
assert _is_artifact_action("actions/upload-artifact@v4", ("upload-artifact",))
def test_download_action(self):
assert _is_artifact_action("christopherhx/gitea-download-artifact@v4", ("download-artifact",))
def test_non_artifact_action(self):
assert not _is_artifact_action("actions/checkout@v4", ("upload-artifact",))
def test_empty_string(self):
assert not _is_artifact_action("", ("upload-artifact",))
def test_case_insensitive(self):
assert _is_artifact_action("Actions/Upload-Artifact@v4", ("upload-artifact",))
class TestExtractArtifactInfo:
def test_uploads_and_downloads(self):
import yaml
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- name: Upload config
uses: christopherhx/gitea-upload-artifact@v4
with:
name: config-${{ github.run_id }}
consumer:
needs: [producer]
steps:
- name: Download config
uses: christopherhx/gitea-download-artifact@v4
with:
name: config-${{ github.run_id }}
""").strip()
wf = yaml.safe_load(workflow_yaml)
uploads, downloads = _extract_artifact_info(wf)
assert uploads == {"config-${{ github.run_id }}": ["producer"]}
assert downloads == [("consumer", "config-${{ github.run_id }}", "Download config")]
def test_no_artifacts(self):
import yaml
workflow_yaml = textwrap.dedent("""
jobs:
build:
steps:
- name: Checkout
uses: actions/checkout@v4
""").strip()
wf = yaml.safe_load(workflow_yaml)
uploads, downloads = _extract_artifact_info(wf)
assert uploads == {}
assert downloads == []
def test_multiple_uploaders_same_artifact(self):
import yaml
workflow_yaml = textwrap.dedent("""
jobs:
producer-a:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: shared
producer-b:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: shared
""").strip()
wf = yaml.safe_load(workflow_yaml)
uploads, downloads = _extract_artifact_info(wf)
assert uploads == {"shared": ["producer-a", "producer-b"]}
def test_step_without_name(self):
import yaml
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: data
consumer:
needs: [producer]
steps:
- uses: christopherhx/gitea-download-artifact@v4
with:
name: data
""").strip()
wf = yaml.safe_load(workflow_yaml)
uploads, downloads = _extract_artifact_info(wf)
assert downloads == [("consumer", "data", "")]
def test_upload_without_name_skipped(self):
import yaml
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
path: ./dist
""").strip()
wf = yaml.safe_load(workflow_yaml)
uploads, downloads = _extract_artifact_info(wf)
assert uploads == {}
class TestCheckWorkflow:
def test_valid_dependency(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- name: Upload config
uses: christopherhx/gitea-upload-artifact@v4
with:
name: config
consumer:
needs: [producer]
steps:
- name: Download config
uses: christopherhx/gitea-download-artifact@v4
with:
name: config
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
assert _check_workflow(f) == []
def test_missing_dependency(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- name: Upload config
uses: christopherhx/gitea-upload-artifact@v4
with:
name: config
consumer:
needs: [other-job]
steps:
- name: Download config
uses: christopherhx/gitea-download-artifact@v4
with:
name: config
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
errors = _check_workflow(f)
assert len(errors) == 1
assert "consumer" in errors[0]
assert "producer" in errors[0]
def test_no_needs_at_all(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: data
consumer:
steps:
- uses: christopherhx/gitea-download-artifact@v4
with:
name: data
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
errors = _check_workflow(f)
assert len(errors) == 1
assert "consumer" in errors[0]
def test_artifact_not_uploaded_in_workflow(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
consumer:
steps:
- name: Download external
uses: christopherhx/gitea-download-artifact@v4
with:
name: external-artifact
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
assert _check_workflow(f) == []
def test_multiple_uploaders_one_in_needs(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer-a:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: shared
producer-b:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: shared
consumer:
needs: [producer-a, other]
steps:
- uses: christopherhx/gitea-download-artifact@v4
with:
name: shared
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
assert _check_workflow(f) == []
def test_string_needs(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: data
consumer:
needs: producer
steps:
- uses: christopherhx/gitea-download-artifact@v4
with:
name: data
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
assert _check_workflow(f) == []
def test_needs_null(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: data
consumer:
needs: null
steps:
- uses: christopherhx/gitea-download-artifact@v4
with:
name: data
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
errors = _check_workflow(f)
assert len(errors) == 1
assert "consumer" in errors[0]
def test_invalid_yaml(self, tmp_path: Path):
f = tmp_path / "test.yml"
f.write_text("jobs: [invalid yaml: {")
errors = _check_workflow(f)
assert len(errors) == 1
assert "cannot parse YAML" in errors[0]
def test_not_a_dict(self, tmp_path: Path):
f = tmp_path / "test.yml"
f.write_text("just a string")
errors = _check_workflow(f)
assert len(errors) == 1
assert "not a valid workflow" in errors[0]
def test_no_jobs(self, tmp_path: Path):
f = tmp_path / "test.yml"
f.write_text("name: empty\non: push\n")
assert _check_workflow(f) == []
def test_continue_on_error_guard(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- name: Upload config
uses: christopherhx/gitea-upload-artifact@v4
with:
name: config
consumer:
needs: [other-job]
steps:
- name: Download config
continue-on-error: true
uses: christopherhx/gitea-download-artifact@v4
with:
name: config
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
assert _check_workflow(f) == []
def test_continue_on_error_false_still_errors(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- name: Upload config
uses: christopherhx/gitea-upload-artifact@v4
with:
name: config
consumer:
needs: [other-job]
steps:
- name: Download config
continue-on-error: false
uses: christopherhx/gitea-download-artifact@v4
with:
name: config
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
errors = _check_workflow(f)
assert len(errors) == 1
assert "consumer" in errors[0]
def test_job_with_no_steps(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
empty:
runs-on: docker
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
assert _check_workflow(f) == []
class TestMain:
def test_passes_when_valid(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: data
consumer:
needs: [producer]
steps:
- uses: christopherhx/gitea-download-artifact@v4
with:
name: data
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
runner = CliRunner()
result = runner.invoke(main, ["--workflows-dir", str(tmp_path)])
assert result.exit_code == 0
assert "OK" in result.output
def test_fails_when_missing_dep(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: data
consumer:
steps:
- uses: christopherhx/gitea-download-artifact@v4
with:
name: data
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
runner = CliRunner()
result = runner.invoke(main, ["--workflows-dir", str(tmp_path)])
assert result.exit_code == 1
assert "FAIL" in result.output
assert "consumer" in result.output
def test_specific_workflow_file(self, tmp_path: Path):
workflow_yaml = textwrap.dedent("""
jobs:
producer:
steps:
- uses: christopherhx/gitea-upload-artifact@v4
with:
name: data
consumer:
needs: [producer]
steps:
- uses: christopherhx/gitea-download-artifact@v4
with:
name: data
""").strip()
f = tmp_path / "test.yml"
f.write_text(workflow_yaml)
runner = CliRunner()
result = runner.invoke(main, ["--workflow", str(f)])
assert result.exit_code == 0
@@ -0,0 +1,356 @@
"""Unit tests for devx.ci.check_workflow_tofu_init."""
from __future__ import annotations
import textwrap
from pathlib import Path
from click.testing import CliRunner
import devx.ci.check_workflow_tofu_init as mod
from devx.ci.check_workflow_tofu_init import _check_workflow, main
def _write_workflow(tmp_path: Path, content: str) -> Path:
filepath = tmp_path / "test.yml"
filepath.write_text(textwrap.dedent(content), encoding="utf-8")
return filepath
class TestCheckWorkflow:
def test_passes_when_tofu_init_present(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
deploy:
runs-on: docker
steps:
- run: python3 scripts/create_production_deployment.py --phase tofu-init
- run: python3 scripts/preflight_deploy.py --env production
""",
)
assert _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS) == []
def test_fails_when_tofu_init_missing(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
preflight:
runs-on: docker
steps:
- run: python3 scripts/preflight_deploy.py --env production
""",
)
errors = _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS)
assert len(errors) == 1
assert "preflight" in errors[0]
assert "tofu-init" in errors[0]
def test_passes_when_direct_tofu_init(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
deploy:
runs-on: docker
steps:
- run: tofu init
- run: tofu output -json
""",
)
assert _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS) == []
def test_fails_when_direct_tofu_output_without_init(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
check:
runs-on: docker
steps:
- run: tofu output -json
""",
)
errors = _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS)
assert len(errors) == 1
assert "check" in errors[0]
def test_passes_when_no_tofu_usage(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
lint:
runs-on: docker
steps:
- run: make lint
""",
)
assert _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS) == []
def test_passes_with_staging_deployment_tofu_init(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
deploy:
runs-on: docker
steps:
- run: python3 scripts/create_staging_deployment.py --phase tofu-init
- run: python3 scripts/create_staging_deployment.py --phase deploy
""",
)
assert _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS) == []
def test_fails_with_tofu_plan_without_init(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
plan:
runs-on: docker
steps:
- run: tofu plan
""",
)
errors = _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS)
assert len(errors) == 1
assert "plan" in errors[0]
def test_fails_with_tofu_apply_without_init(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
apply:
runs-on: docker
steps:
- run: tofu apply -auto-approve
""",
)
errors = _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS)
assert len(errors) == 1
assert "apply" in errors[0]
def test_multiple_jobs_one_missing(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
good:
runs-on: docker
steps:
- run: python3 scripts/create_production_deployment.py --phase tofu-init
- run: python3 scripts/preflight_deploy.py --env production
bad:
runs-on: docker
steps:
- run: python3 scripts/preflight_deploy.py --env production
""",
)
errors = _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS)
assert len(errors) == 1
assert "bad" in errors[0]
def test_no_steps_passes(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
empty:
runs-on: docker
""",
)
assert _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS) == []
def test_destroy_orphans_does_not_require_init(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
cleanup:
runs-on: docker
steps:
- run: python3 scripts/destroy_orphans.py
""",
)
assert _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS) == []
def test_invalid_yaml_returns_error(self, tmp_path: Path) -> None:
filepath = tmp_path / "bad.yml"
filepath.write_text("jobs: [invalid yaml: {", encoding="utf-8")
errors = _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS)
assert len(errors) == 1
assert "cannot parse YAML" in errors[0]
def test_tofu_show_requires_init(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
show:
runs-on: docker
steps:
- run: tofu show -json
""",
)
errors = _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS)
assert len(errors) == 1
assert "show" in errors[0]
def test_custom_state_scripts(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
custom:
runs-on: docker
steps:
- run: python3 scripts/my_custom_script.py
""",
)
errors = _check_workflow(filepath, {"my_custom_script.py"})
assert len(errors) == 1
assert "custom" in errors[0]
def test_step_with_no_run_skipped(self, tmp_path: Path) -> None:
"""A step with no 'run' key should be skipped (line 80 continue)."""
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
deploy:
runs-on: docker
steps:
- name: Checkout
uses: actions/checkout@v4
- run: tofu init
- run: tofu output
""",
)
assert _check_workflow(filepath, mod.DEFAULT_TOFU_STATE_SCRIPTS) == []
class TestCli:
def test_passes_with_specific_workflow(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
deploy:
runs-on: docker
steps:
- run: tofu init
- run: tofu output
""",
)
runner = CliRunner()
result = runner.invoke(main, ["--workflow", str(filepath)])
assert result.exit_code == 0
assert "OK" in result.output
def test_fails_with_missing_tofu_init(self, tmp_path: Path) -> None:
filepath = _write_workflow(
tmp_path,
"""
name: Test
on: push
jobs:
preflight:
runs-on: docker
steps:
- run: python3 scripts/preflight_deploy.py --env production
""",
)
runner = CliRunner()
result = runner.invoke(main, ["--workflow", str(filepath)])
assert result.exit_code == 1
assert "FAIL" in result.output
assert "preflight" in result.output
def test_checks_all_workflows_by_default(self, tmp_path: Path) -> None:
workflows_dir = tmp_path / "workflows"
workflows_dir.mkdir()
(workflows_dir / "good.yml").write_text(
textwrap.dedent("""
name: Good
on: push
jobs:
deploy:
runs-on: docker
steps:
- run: tofu init
- run: tofu output
"""),
encoding="utf-8",
)
(workflows_dir / "bad.yml").write_text(
textwrap.dedent("""
name: Bad
on: push
jobs:
check:
runs-on: docker
steps:
- run: tofu output
"""),
encoding="utf-8",
)
runner = CliRunner()
result = runner.invoke(main, ["--workflows-dir", str(workflows_dir)])
assert result.exit_code == 1
assert "bad.yml" in result.output
assert "check" in result.output
def test_all_workflows_pass(self, tmp_path: Path) -> None:
workflows_dir = tmp_path / "workflows"
workflows_dir.mkdir()
(workflows_dir / "ok.yml").write_text(
textwrap.dedent("""
name: OK
on: push
jobs:
deploy:
runs-on: docker
steps:
- run: tofu init
- run: tofu plan
"""),
encoding="utf-8",
)
runner = CliRunner()
result = runner.invoke(main, ["--workflows-dir", str(workflows_dir)])
assert result.exit_code == 0
assert "OK" in result.output
+7
View File
@@ -150,6 +150,13 @@ class TestCiCommands:
assert result.exit_code == 0
mock_run.assert_called_once_with("devx.ci.validate_commit_msg", ["msg"])
@patch("devx.cli._run_module")
def test_ci_wait_for_checks(self, mock_run: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(cli, ["ci", "wait-for-checks", "--", "--job-name", "molecule-tests"])
assert result.exit_code == 0
mock_run.assert_called_once_with("devx.ci.wait_for_checks", ["--job-name", "molecule-tests"])
class TestToolsCommands:
@patch("devx.cli._run_module")
+57 -26
View File
@@ -1,4 +1,9 @@
"""Unit tests for scripts/ci/discover_runners.py."""
"""Unit tests for devx.ci.discover_runners (deprecated wrapper).
The wrapper re-exports from devx.molecule.discover_runners; these tests
verify backward compatibility by importing through the wrapper and
patching the canonical implementation's requests module.
"""
import json
from pathlib import Path
@@ -32,7 +37,7 @@ class TestGenerateIndices:
class TestQueryRunners:
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_returns_total_from_all_levels(self, mock_get: MagicMock) -> None:
"""Runners from repo, org, and admin levels are summed."""
responses = [
@@ -44,7 +49,7 @@ class TestQueryRunners:
result = query_runners("https://api.example.com", "token", "owner", "repo")
assert result == 6
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_skips_non_200(self, mock_get: MagicMock) -> None:
"""Non-200 responses (e.g., 403 for admin) are skipped."""
responses = [
@@ -56,7 +61,7 @@ class TestQueryRunners:
result = query_runners("https://api.example.com", "token", "owner", "repo")
assert result == 3
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_handles_request_exception(self, mock_get: MagicMock) -> None:
"""Network errors are caught and don't crash."""
mock_get.side_effect = [
@@ -67,7 +72,7 @@ class TestQueryRunners:
result = query_runners("https://api.example.com", "token", "owner", "repo")
assert result == 3
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_all_failures_return_zero(self, mock_get: MagicMock) -> None:
"""When all API calls fail, returns 0."""
mock_get.side_effect = [
@@ -78,7 +83,7 @@ class TestQueryRunners:
result = query_runners("https://api.example.com", "token", "owner", "repo")
assert result == 0
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_value_error_on_repo_level(self, mock_get: MagicMock) -> None:
"""JSON parse error on repo level is caught."""
responses = [
@@ -90,7 +95,7 @@ class TestQueryRunners:
result = query_runners("https://api.example.com", "token", "owner", "repo")
assert result == 3
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_value_error_on_org_level(self, mock_get: MagicMock) -> None:
"""JSON parse error on org level is caught."""
responses = [
@@ -102,7 +107,7 @@ class TestQueryRunners:
result = query_runners("https://api.example.com", "token", "owner", "repo")
assert result == 3
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_value_error_on_admin_level(self, mock_get: MagicMock) -> None:
"""JSON parse error on admin level is caught."""
responses = [
@@ -114,14 +119,14 @@ class TestQueryRunners:
result = query_runners("https://api.example.com", "token", "owner", "repo")
assert result == 3
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_request_exception_on_all_levels(self, mock_get: MagicMock) -> None:
"""Network errors on all levels return 0."""
mock_get.side_effect = __import__("requests").RequestException("network error")
result = query_runners("https://api.example.com", "token", "owner", "repo")
assert result == 0
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_query_runners_403_no_warning(self, mock_get: MagicMock, capsys: pytest.CaptureFixture[str]) -> None:
"""403 on instance-level runners should not produce a warning (expected without admin scope)."""
responses = [
@@ -135,7 +140,7 @@ class TestQueryRunners:
captured = capsys.readouterr()
assert "instance-level" not in captured.err
@patch("devx.ci.discover_runners.requests.get")
@patch("devx.molecule.discover_runners.requests.get")
def test_instance_level_non_403_warns(self, mock_get: MagicMock, capsys: pytest.CaptureFixture[str]) -> None:
"""Non-200, non-403 status on instance-level runners should produce a warning."""
responses = [
@@ -152,37 +157,37 @@ class TestQueryRunners:
class TestGetRunnerCount:
@patch("devx.ci.discover_runners.query_runners", return_value=5)
@patch("devx.molecule.discover_runners.query_runners", return_value=5)
def test_uses_api_count_when_positive(self, mock_query: MagicMock) -> None:
result = get_runner_count("https://api.example.com", "token", "owner", "repo")
assert result == 5
@patch("devx.ci.discover_runners.query_runners", return_value=0)
@patch("devx.molecule.discover_runners.query_runners", return_value=0)
@patch.dict("os.environ", {"MOLECULE_RUNNERS": "4"})
def test_falls_back_to_env_var(self, mock_query: MagicMock) -> None:
result = get_runner_count("https://api.example.com", "token", "owner", "repo")
assert result == 4
@patch("devx.ci.discover_runners.query_runners", return_value=0)
@patch("devx.molecule.discover_runners.query_runners", return_value=0)
@patch.dict("os.environ", {"MOLECULE_RUNNERS": "invalid"})
def test_falls_back_to_default_on_invalid_env(self, mock_query: MagicMock) -> None:
result = get_runner_count("https://api.example.com", "token", "owner", "repo")
assert result == DEFAULT_MAX_RUNNERS
@patch("devx.ci.discover_runners.query_runners", return_value=0)
@patch("devx.molecule.discover_runners.query_runners", return_value=0)
@patch.dict("os.environ", {}, clear=True)
def test_falls_back_to_default_when_no_env(self, mock_query: MagicMock) -> None:
result = get_runner_count("https://api.example.com", "token", "owner", "repo")
assert result == DEFAULT_MAX_RUNNERS
@patch("devx.ci.discover_runners.query_runners", return_value=0)
@patch("devx.molecule.discover_runners.query_runners", return_value=0)
@patch.dict("os.environ", {"MOLECULE_RUNNERS": "0"})
def test_env_var_zero_falls_back_to_default(self, mock_query: MagicMock) -> None:
"""MOLECULE_RUNNERS=0 is invalid, falls back to default."""
result = get_runner_count("https://api.example.com", "token", "owner", "repo")
assert result == DEFAULT_MAX_RUNNERS
@patch("devx.ci.discover_runners.query_runners", return_value=0)
@patch("devx.molecule.discover_runners.query_runners", return_value=0)
@patch.dict("os.environ", {}, clear=True)
def test_no_token_uses_env_var(self, mock_query: MagicMock) -> None:
"""When no token, skips API and uses env/default."""
@@ -192,7 +197,7 @@ class TestGetRunnerCount:
class TestMain:
@patch("devx.ci.discover_runners.get_runner_count", return_value=3)
@patch("devx.molecule.discover_runners.get_runner_count", return_value=3)
def test_default_output(self, mock_count: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, [])
@@ -200,28 +205,28 @@ class TestMain:
assert "count=3" in result.output
assert 'indices=["1", "2", "3"]' in result.output
@patch("devx.ci.discover_runners.get_runner_count", return_value=5)
@patch("devx.molecule.discover_runners.get_runner_count", return_value=5)
def test_count_only(self, mock_count: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--count"])
assert result.exit_code == 0
assert result.output.strip() == "5"
@patch("devx.ci.discover_runners.get_runner_count", return_value=4)
@patch("devx.molecule.discover_runners.get_runner_count", return_value=4)
def test_indices_only(self, mock_count: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--indices"])
assert result.exit_code == 0
assert json.loads(result.output.strip()) == ["1", "2", "3", "4"]
@patch("devx.ci.discover_runners.get_runner_count", return_value=1)
@patch("devx.molecule.discover_runners.get_runner_count", return_value=1)
def test_single_runner(self, mock_count: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--indices"])
assert result.exit_code == 0
assert json.loads(result.output.strip()) == ["1"]
@patch("devx.ci.discover_runners.get_runner_count", return_value=3)
@patch("devx.molecule.discover_runners.get_runner_count", return_value=3)
def test_github_output(self, mock_count: MagicMock, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
gh_file = tmp_path / "output.txt"
monkeypatch.setenv("GITHUB_OUTPUT", str(gh_file))
@@ -232,14 +237,14 @@ class TestMain:
assert "runner-count=3" in content
assert "runner-indices=" in content
@patch("devx.ci.discover_runners.get_runner_count", return_value=3)
@patch("devx.molecule.discover_runners.get_runner_count", return_value=3)
def test_github_output_no_env(self, mock_count: MagicMock, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("GITHUB_OUTPUT", raising=False)
runner = CliRunner()
result = runner.invoke(main, ["--github-output"])
assert result.exit_code != 0
@patch("devx.ci.discover_runners.get_runner_count", return_value=2)
@patch("devx.molecule.discover_runners.get_runner_count", return_value=2)
def test_explicit_owner_and_repo(self, mock_count: MagicMock) -> None:
"""When --owner and --repo are provided, env vars are not used."""
runner = CliRunner()
@@ -251,8 +256,8 @@ class TestMain:
assert "myorg" in args
assert "myrepo" in args
@patch("devx.ci.discover_runners.get_ci_token", side_effect=click.ClickException("no token"))
@patch("devx.ci.discover_runners.get_runner_count", return_value=3)
@patch("devx.molecule.discover_runners.get_ci_token", side_effect=click.ClickException("no token"))
@patch("devx.molecule.discover_runners.get_runner_count", return_value=3)
def test_missing_token_runs_without_api(self, mock_count: MagicMock, mock_token: MagicMock) -> None:
"""When no token is available, runner discovery falls back to env/default."""
runner = CliRunner()
@@ -261,3 +266,29 @@ class TestMain:
assert result.output.strip() == "3"
args, _ = mock_count.call_args
assert args[1] is None # token passed as None when missing
class TestDeprecationWrapper:
def test_re_exports_canonical_symbols(self) -> None:
"""The wrapper re-exports the canonical implementation's symbols."""
from devx.ci import discover_runners as ci_mod
from devx.molecule import discover_runners as mol_mod
assert ci_mod.query_runners is mol_mod.query_runners
assert ci_mod.get_runner_count is mol_mod.get_runner_count
assert ci_mod.generate_indices is mol_mod.generate_indices
assert ci_mod.main is mol_mod.main
assert ci_mod.DEFAULT_MAX_RUNNERS is mol_mod.DEFAULT_MAX_RUNNERS
def test_emit_deprecation_warning(self) -> None:
"""_emit_deprecation_warning issues a DeprecationWarning."""
import warnings
from devx.ci.discover_runners import _emit_deprecation_warning
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
_emit_deprecation_warning()
assert len(caught) == 1
assert issubclass(caught[0].category, DeprecationWarning)
assert "deprecated" in str(caught[0].message)
+55
View File
@@ -311,6 +311,61 @@ class TestDiscoverMultiRole:
with pytest.raises(click.ClickException):
discover_multi_role_scenarios()
def test_include_roles_filters_to_subset(self, tmp_path: Path) -> None:
roles = tmp_path / "roles"
for scenario in ["default"]:
(roles / "docker_base" / "molecule" / scenario).mkdir(parents=True)
(roles / "crowdsec" / "molecule" / scenario).mkdir(parents=True)
(roles / "app_container" / "molecule" / scenario).mkdir(parents=True)
result = discover_multi_role_scenarios(roles, include_roles=["docker_base", "crowdsec"])
assert ("docker_base", "default") in result
assert ("crowdsec", "default") in result
assert ("app_container", "default") not in result
assert len(result) == 2
def test_include_roles_case_insensitive(self, tmp_path: Path) -> None:
roles = tmp_path / "roles"
(roles / "Docker_Base" / "molecule" / "default").mkdir(parents=True)
(roles / "other" / "molecule" / "default").mkdir(parents=True)
result = discover_multi_role_scenarios(roles, include_roles=["docker_base"])
assert ("Docker_Base", "default") in result
assert len(result) == 1
def test_exclude_roles_skips_subset(self, tmp_path: Path) -> None:
roles = tmp_path / "roles"
for scenario in ["default"]:
(roles / "docker_base" / "molecule" / scenario).mkdir(parents=True)
(roles / "crowdsec" / "molecule" / scenario).mkdir(parents=True)
(roles / "app_container" / "molecule" / scenario).mkdir(parents=True)
result = discover_multi_role_scenarios(roles, exclude_roles=["docker_base", "crowdsec"])
assert ("docker_base", "default") not in result
assert ("crowdsec", "default") not in result
assert ("app_container", "default") in result
assert len(result) == 1
def test_exclude_roles_case_insensitive(self, tmp_path: Path) -> None:
roles = tmp_path / "roles"
(roles / "Docker_Base" / "molecule" / "default").mkdir(parents=True)
(roles / "other" / "molecule" / "default").mkdir(parents=True)
result = discover_multi_role_scenarios(roles, exclude_roles=["docker_base"])
assert ("Docker_Base", "default") not in result
assert ("other", "default") in result
assert len(result) == 1
def test_include_and_exclude_combined(self, tmp_path: Path) -> None:
roles = tmp_path / "roles"
for scenario in ["default"]:
(roles / "docker_base" / "molecule" / scenario).mkdir(parents=True)
(roles / "crowdsec" / "molecule" / scenario).mkdir(parents=True)
(roles / "app_container" / "molecule" / scenario).mkdir(parents=True)
result = discover_multi_role_scenarios(
roles, include_roles=["docker_base", "crowdsec", "app_container"], exclude_roles=["crowdsec"]
)
assert ("docker_base", "default") in result
assert ("crowdsec", "default") not in result
assert ("app_container", "default") in result
assert len(result) == 2
def test_default_roles_root_constant(self) -> None:
assert Path("ansible/roles") == DEFAULT_ROLES_ROOT
+100
View File
@@ -0,0 +1,100 @@
"""Unit tests for devx.i18n."""
from __future__ import annotations
import pytest
import devx.i18n as i18n_mod
from devx.i18n import _, configure_i18n
class TestTranslate:
def test_returns_english_by_default(self) -> None:
with pytest.MonkeyPatch().context() as mp:
mp.delenv("DEVX_LANG", raising=False)
assert _("Running tests") == "Running tests"
def test_returns_key_when_missing(self) -> None:
with pytest.MonkeyPatch().context() as mp:
mp.delenv("DEVX_LANG", raising=False)
assert _("nonexistent.key.xyz") == "nonexistent.key.xyz"
def test_formats_kwargs(self) -> None:
# Find a key with format placeholders
for key, translations in i18n_mod.TRANSLATIONS.items():
en = translations.get("en", "")
if "{" in en:
with pytest.MonkeyPatch().context() as mp:
mp.delenv("DEVX_LANG", raising=False)
result = _(key, **dict.fromkeys(_extract_format_keys(en), "x"))
assert "{" not in result
return
pytest.skip("No key with format placeholders found")
def test_invalid_lang_falls_back_to_english(self) -> None:
with pytest.MonkeyPatch().context() as mp:
mp.setenv("DEVX_LANG", "fr")
assert _("Running tests") == "Running tests"
def test_bulgarian_translation(self) -> None:
with pytest.MonkeyPatch().context() as mp:
mp.setenv("DEVX_LANG", "bg")
# Find a key that has a Bulgarian translation
for key, translations in i18n_mod.TRANSLATIONS.items():
if "bg" in translations:
result = _(key)
assert result == translations["bg"]
return
pytest.skip("No Bulgarian translation found")
class TestConfigureI18n:
def test_custom_lang_env_var(self) -> None:
configure_i18n(lang_env_var="GRM_LANG")
try:
with pytest.MonkeyPatch().context() as mp:
mp.setenv("GRM_LANG", "bg")
mp.delenv("DEVX_LANG", raising=False)
# Find a key with Bulgarian translation
for key, translations in i18n_mod.TRANSLATIONS.items():
if "bg" in translations:
assert _(key) == translations["bg"]
return
pytest.skip("No Bulgarian translation found")
finally:
configure_i18n() # Reset to defaults
def test_custom_translations_path_env_var(self, tmp_path) -> None:
custom_translations = {"custom.key": {"en": "Custom Value", "bg": "Персонализирано"}}
custom_file = tmp_path / "custom.json"
custom_file.write_text(__import__("json").dumps(custom_translations))
configure_i18n(translations_path_env_var="GRM_TRANSLATIONS_PATH")
try:
# Use i18n_mod.TRANSLATIONS (not a stale import) — other tests
# may call importlib.reload(devx.i18n), replacing the dict object.
translations = i18n_mod.TRANSLATIONS
original = dict(translations)
translations.update(custom_translations)
try:
with pytest.MonkeyPatch().context() as mp:
mp.setenv("GRM_TRANSLATIONS_PATH", str(custom_file))
assert _("custom.key") == "Custom Value"
finally:
translations.clear()
translations.update(original)
finally:
configure_i18n() # Reset to defaults
def test_reset_to_defaults(self) -> None:
configure_i18n(lang_env_var="GRM_LANG")
configure_i18n() # Reset
assert i18n_mod._lang_env_var == "DEVX_LANG"
assert i18n_mod._translations_path_env_var == "DEVX_TRANSLATIONS_PATH"
def _extract_format_keys(template: str) -> list[str]:
"""Extract {key} format placeholders from a template string."""
import re
return re.findall(r"\{(\w+)\}", template)
+89
View File
@@ -1,6 +1,7 @@
from __future__ import annotations
import platform
import urllib.error
from pathlib import Path
from unittest.mock import patch
@@ -68,6 +69,67 @@ class TestDownload:
mock_urlopen.assert_called_once()
assert dest.read_bytes() == b"data"
def test_download_retries_on_transient_error(self, tmp_path: Path) -> None:
"""Download retries on URLError then succeeds."""
dest = tmp_path / "file.bin"
call_count = [0]
class _FakeResponse:
def __init__(self) -> None:
self._sent = False
def __enter__(self) -> _FakeResponse:
return self
def __exit__(self, *args: object) -> None:
pass
def read(self, n: int = -1) -> bytes:
if self._sent:
return b""
self._sent = True
return b"data"
def _flaky_urlopen(url: str, timeout: int = 60):
call_count[0] += 1
if call_count[0] < 2:
raise urllib.error.URLError("Remote end closed connection")
return _FakeResponse()
with patch("urllib.request.urlopen", side_effect=_flaky_urlopen):
install_tools._download("https://example.com/file", dest, _sleep=lambda _: None)
assert call_count[0] == 2
assert dest.read_bytes() == b"data"
def test_download_fails_after_max_retries(self, tmp_path: Path) -> None:
"""Download raises after MAX_DOWNLOAD_RETRIES attempts."""
dest = tmp_path / "file.bin"
call_count = [0]
def _always_fail(url: str, timeout: int = 60):
call_count[0] += 1
raise urllib.error.URLError("Remote end closed connection")
with patch("urllib.request.urlopen", side_effect=_always_fail):
with pytest.raises(urllib.error.URLError):
install_tools._download("https://example.com/file", dest, _sleep=lambda _: None)
assert call_count[0] == install_tools.MAX_DOWNLOAD_RETRIES
assert not dest.exists()
def test_download_no_retry_on_non_transient_error(self, tmp_path: Path) -> None:
"""Download does not retry on non-network errors (e.g. ValueError)."""
dest = tmp_path / "file.bin"
call_count = [0]
def _fail_with_value_error(url: str, timeout: int = 60):
call_count[0] += 1
raise ValueError("not a network error")
with patch("urllib.request.urlopen", side_effect=_fail_with_value_error):
with pytest.raises(ValueError):
install_tools._download("https://example.com/file", dest, _sleep=lambda _: None)
assert call_count[0] == 1
class TestDownloadBinary:
def test_download(self, tmp_path: Path) -> None:
@@ -233,6 +295,33 @@ class TestInstallTea:
assert install_tools.install_tea() is True
assert (tmp_path / "tea").exists()
def test_install_fallback_to_second_url(self, tmp_path: Path) -> None:
"""First URL fails (403), second URL succeeds."""
call_count = [0]
def _download_side_effect(url: str, dest: Path) -> None:
call_count[0] += 1
if call_count[0] == 1:
raise OSError("HTTP Error 403: Forbidden")
Path(dest).write_bytes(b"binary")
with patch.object(install_tools, "_is_installed", return_value=False):
with patch.object(install_tools, "TARGET_DIR", tmp_path):
with patch.object(platform, "machine", return_value="x86_64"):
with patch.object(install_tools, "_download", side_effect=_download_side_effect):
assert install_tools.install_tea() is True
assert (tmp_path / "tea").exists()
assert call_count[0] == 2
def test_install_all_urls_fail(self, tmp_path: Path) -> None:
"""All URLs fail — should raise ClickException."""
with patch.object(install_tools, "_is_installed", return_value=False):
with patch.object(install_tools, "TARGET_DIR", tmp_path):
with patch.object(platform, "machine", return_value="x86_64"):
with patch.object(install_tools, "_download", side_effect=OSError("403 Forbidden")):
with pytest.raises(ClickException, match="Failed to download tea"):
install_tools.install_tea()
class TestInstallHadolint:
def test_already_installed(self) -> None:
+192
View File
@@ -0,0 +1,192 @@
"""Unit tests for devx.molecule.molecule_changed.
Verifies that the script correctly detects changed roles and maps
them to make targets.
"""
from __future__ import annotations
from unittest.mock import patch
from click.testing import CliRunner
from devx.molecule.molecule_changed import (
detect_changed_roles,
get_changed_files,
main,
roles_to_targets,
)
def test_detect_role_change():
"""A file in ansible/roles/<role>/ maps to that role."""
files = ["ansible/roles/docker_base/tasks/main.yml"]
roles = detect_changed_roles(files)
assert "docker_base" in roles
def test_detect_playbook_change():
"""A playbook change maps to its included roles."""
files = ["ansible/playbooks/deploy-observability.yml"]
roles = detect_changed_roles(files)
assert "observability" in roles
assert "docker_base" in roles
assert "zitadel" in roles
def test_detect_shared_infra_triggers_all():
"""ansible.cfg change triggers all roles."""
files = ["ansible/ansible.cfg"]
roles = detect_changed_roles(files)
assert len(roles) == 10 # all roles
def test_detect_no_ansible_changes():
"""Non-Ansible files don't trigger any roles."""
files = ["scripts/molecule_changed.py", "Makefile"]
roles = detect_changed_roles(files)
assert len(roles) == 0
def test_roles_to_targets():
"""Role names map to make targets."""
targets = roles_to_targets({"docker_base", "zitadel"})
assert "molecule-docker-base" in targets
assert "molecule-zitadel" in targets
def test_roles_to_targets_unknown_role():
"""Unknown roles are silently skipped."""
targets = roles_to_targets({"docker_base", "unknown_role"})
assert targets == ["molecule-docker-base"]
def test_main_no_changes():
"""When no files changed, outputs message to stderr."""
with patch("devx.molecule.molecule_changed.get_changed_files", return_value=[]):
runner = CliRunner()
result = runner.invoke(main, ["--print-targets"])
assert result.exit_code == 0
assert "No changed files" in result.output
def test_main_print_targets():
"""--print-targets outputs make targets."""
with patch(
"devx.molecule.molecule_changed.get_changed_files",
return_value=["ansible/roles/docker_base/tasks/main.yml"],
):
runner = CliRunner()
result = runner.invoke(main, ["--print-targets"])
assert result.exit_code == 0
assert "molecule-docker-base" in result.output
def test_main_print_roles():
"""--print-roles outputs role names."""
with patch(
"devx.molecule.molecule_changed.get_changed_files",
return_value=["ansible/roles/zitadel/tasks/main.yml"],
):
runner = CliRunner()
result = runner.invoke(main, ["--print-roles"])
assert result.exit_code == 0
assert "zitadel" in result.output
def test_main_no_ansible_changes():
"""When only non-Ansible files changed, outputs no scenarios message."""
with patch(
"devx.molecule.molecule_changed.get_changed_files",
return_value=["scripts/molecule_changed.py"],
):
runner = CliRunner()
result = runner.invoke(main, ["--print-targets"])
assert result.exit_code == 0
assert "No molecule scenarios" in result.output
def test_get_changed_files_with_mock():
"""get_changed_files returns files from git diff."""
with patch("devx.molecule.molecule_changed._run_git", return_value="file1\nfile2\n"):
files = get_changed_files("origin/master")
assert files == ["file1", "file2"]
def test_get_changed_files_falls_back_to_master():
"""When base ref has no diff, falls back to master."""
calls: list[list[str]] = []
def mock_git(args):
calls.append(args)
# First call (origin/master) returns empty, second (master) returns files
if "origin/master...HEAD" in args[2]:
return ""
return "ansible/roles/docker_base/tasks/main.yml\n"
with patch("devx.molecule.molecule_changed._run_git", side_effect=mock_git):
files = get_changed_files("origin/master")
assert files == ["ansible/roles/docker_base/tasks/main.yml"]
assert len(calls) == 2
def test_get_changed_files_empty():
"""When no changes in either ref, returns empty list."""
with patch("devx.molecule.molecule_changed._run_git", return_value=""):
files = get_changed_files("origin/master")
assert files == []
def test_detect_molecule_shared_path():
"""ansible/molecule/ change triggers all roles."""
files = ["ansible/molecule/Dockerfile"]
roles = detect_changed_roles(files)
assert len(roles) == 10
def test_detect_requirements_yml_triggers_all():
"""ansible/requirements.yml change triggers all roles."""
files = ["ansible/requirements.yml"]
roles = detect_changed_roles(files)
assert len(roles) == 10
def test_detect_configure_oidc_playbook():
"""configure-oidc.yml maps to sso_config and app_container."""
files = ["ansible/playbooks/configure-oidc.yml"]
roles = detect_changed_roles(files)
assert "sso_config" in roles
assert "app_container" in roles
def test_detect_prepare_vms_playbook():
"""prepare-vms.yml maps to all base roles."""
files = ["ansible/playbooks/prepare-vms.yml"]
roles = detect_changed_roles(files)
assert "docker_base" in roles
assert "app_hardening" in roles
assert "storage" in roles
assert "disk_cleanup" in roles
assert "crowdsec" in roles
def test_detect_deploy_customer_playbook():
"""deploy-customer.yml maps to its roles."""
files = ["ansible/playbooks/deploy-customer.yml"]
roles = detect_changed_roles(files)
assert "app_container" in roles
assert "docker_base" in roles
assert "app_hardening" in roles
assert "sso_config" in roles
def test_main_default_base():
"""main() with no --base uses origin/master."""
with patch(
"devx.molecule.molecule_changed.get_changed_files",
return_value=["ansible/roles/zitadel/tasks/main.yml"],
) as mock:
runner = CliRunner()
result = runner.invoke(main, ["--print-roles"])
assert result.exit_code == 0
mock.assert_called_once_with("origin/master")
+43
View File
@@ -114,6 +114,49 @@ class TestInstallAnsibleCollections:
_install_ansible_collections(".venv/bin")
mock_run.assert_not_called()
@patch("tenacity.nap.time.sleep")
@patch("devx.tools.setup.shutil.which", return_value="/usr/local/bin/ansible-galaxy")
@patch("devx.tools.setup._run")
def test_retries_on_transient_failure(
self, mock_run: MagicMock, mock_which: MagicMock, mock_sleep: MagicMock, tmp_path: Path
) -> None:
"""ansible-galaxy install should retry on transient network errors."""
import subprocess as _subprocess
req = tmp_path / "ansible" / "requirements.yml"
req.parent.mkdir(parents=True)
req.write_text("collections: []")
# First call fails (timeout), second succeeds
mock_run.side_effect = [
_subprocess.CalledProcessError(1, ["ansible-galaxy", "collection", "install"]),
None,
]
with patch("devx.tools.setup.Path") as mock_path:
mock_path.return_value.exists.return_value = True
mock_path.return_value.__str__ = lambda _: str(req)
_install_ansible_collections(".venv/bin")
assert mock_run.call_count == 2
@patch("tenacity.nap.time.sleep")
@patch("devx.tools.setup.shutil.which", return_value="/usr/local/bin/ansible-galaxy")
@patch("devx.tools.setup._run")
def test_exhausts_retries_then_raises(
self, mock_run: MagicMock, mock_which: MagicMock, mock_sleep: MagicMock, tmp_path: Path
) -> None:
"""After 3 attempts, the error should propagate."""
import subprocess as _subprocess
req = tmp_path / "ansible" / "requirements.yml"
req.parent.mkdir(parents=True)
req.write_text("collections: []")
mock_run.side_effect = _subprocess.CalledProcessError(1, ["ansible-galaxy"])
with patch("devx.tools.setup.Path") as mock_path:
mock_path.return_value.exists.return_value = True
mock_path.return_value.__str__ = lambda _: str(req)
with pytest.raises(_subprocess.CalledProcessError):
_install_ansible_collections(".venv/bin")
assert mock_run.call_count == 3
class TestConfigureTeaLogin:
@patch("devx.tools.setup.shutil.which", return_value=None)
+1
View File
@@ -52,6 +52,7 @@ class TestInstallInImage:
mock_run.assert_called_once()
cmd = mock_run.call_args[0][0]
assert "--no-cache-dir" in cmd
assert "--no-deps" in cmd
assert "-e" in cmd
assert "." in cmd
# No extras → spec is "."
+103
View File
@@ -0,0 +1,103 @@
"""Unit tests for devx.tools.check_alert_rules."""
from __future__ import annotations
from pathlib import Path
from unittest.mock import MagicMock, patch
from click.testing import CliRunner
from devx.tools.check_alert_rules import main
class TestMain:
def test_skip_when_promtool_not_found(self, tmp_path: Path):
"""Should exit 0 and print skip message when promtool is not on PATH."""
with patch("shutil.which", return_value=None):
runner = CliRunner()
result = runner.invoke(main, ["--template-path", str(tmp_path)])
assert result.exit_code == 0
assert "promtool not found" in result.output
def test_validates_rules_successfully(self, tmp_path: Path):
"""Should exit 0 when promtool reports SUCCESS."""
(tmp_path / "alert-rules.yml.j2").write_text("groups: []")
mock_result = MagicMock()
mock_result.returncode = 0
mock_result.stdout = "Checking /tmp/test.yml\n SUCCESS: 60 rules found\n"
mock_result.stderr = ""
with patch("shutil.which", return_value="/usr/bin/promtool"):
with patch("subprocess.run", return_value=mock_result):
runner = CliRunner()
result = runner.invoke(main, ["--template-path", str(tmp_path)])
assert result.exit_code == 0
def test_fails_on_promtool_error(self, tmp_path: Path):
"""Should exit non-zero when promtool reports an error."""
(tmp_path / "alert-rules.yml.j2").write_text("groups: []")
mock_result = MagicMock()
mock_result.returncode = 1
mock_result.stdout = ""
mock_result.stderr = "Error: invalid template function 'default'\n"
with patch("shutil.which", return_value="/usr/bin/promtool"):
with patch("subprocess.run", return_value=mock_result):
runner = CliRunner()
result = runner.invoke(main, ["--template-path", str(tmp_path)])
assert result.exit_code != 0
def test_uses_correct_template_path(self, tmp_path: Path):
"""Should render the specified template from the given path."""
(tmp_path / "alert-rules.yml.j2").write_text("groups: []")
captured_args = []
def fake_run(args, **kwargs):
captured_args.append(args)
mock = MagicMock()
mock.returncode = 0
mock.stdout = "SUCCESS"
mock.stderr = ""
return mock
with patch("shutil.which", return_value="/usr/bin/promtool"):
with patch("subprocess.run", side_effect=fake_run):
runner = CliRunner()
runner.invoke(main, ["--template-path", str(tmp_path)])
assert captured_args[0][0] == "promtool"
assert captured_args[0][1] == "check"
assert captured_args[0][2] == "rules"
def test_custom_template_name(self, tmp_path: Path):
"""Should render a custom template name."""
(tmp_path / "custom-rules.yml.j2").write_text("groups: []")
mock_result = MagicMock()
mock_result.returncode = 0
mock_result.stdout = "SUCCESS"
mock_result.stderr = ""
with patch("shutil.which", return_value="/usr/bin/promtool"):
with patch("subprocess.run", return_value=mock_result):
runner = CliRunner()
result = runner.invoke(
main, ["--template-path", str(tmp_path), "--template-name", "custom-rules.yml.j2"]
)
assert result.exit_code == 0
def test_template_vars_passed(self, tmp_path: Path):
"""Should pass template variables to the render call."""
(tmp_path / "alert-rules.yml.j2").write_text("grafana: {{ grafana_base_url }}\ngroups: []")
mock_result = MagicMock()
mock_result.returncode = 0
mock_result.stdout = "SUCCESS"
mock_result.stderr = ""
with patch("shutil.which", return_value="/usr/bin/promtool"):
with patch("subprocess.run", return_value=mock_result):
runner = CliRunner()
result = runner.invoke(
main,
[
"--template-path",
str(tmp_path),
"--var",
"grafana_base_url=https://grafana.test.example.com",
],
)
assert result.exit_code == 0
@@ -0,0 +1,367 @@
"""Unit tests for devx.tools.check_ansible_no_log."""
from __future__ import annotations
from pathlib import Path
from click.testing import CliRunner
from devx.tools.check_ansible_no_log import _check_task, check_directory, main
def _make_task(name: str, action: str, value: str, **extra: object) -> dict:
"""Build a minimal task dict for testing."""
task: dict = {"name": name, action: value}
task.update(extra)
return task
class TestCheckTask:
def test_task_with_secret_and_no_log_passes(self):
task = _make_task(
"Safe task",
"ansible.builtin.shell",
"echo {{ _secrets.mattermost_admin_password }}",
no_log=True,
)
assert _check_task(task, Path("test.yml"), 1) == []
def test_task_with_secret_and_no_no_log_fails(self):
task = _make_task(
"Unsafe task",
"ansible.builtin.shell",
"echo {{ _secrets.mattermost_admin_password }}",
)
violations = _check_task(task, Path("test.yml"), 1)
assert len(violations) == 1
assert "no_log" in violations[0]
def test_task_without_secret_passes(self):
task = _make_task(
"Normal task",
"ansible.builtin.shell",
"echo hello world",
)
assert _check_task(task, Path("test.yml"), 1) == []
def test_task_with_jinja_no_log_passes(self):
task = _make_task(
"Safe task with jinja no_log",
"ansible.builtin.shell",
"echo {{ _secrets.mattermost_admin_password }}",
no_log="{{ not (debug_mode | default(false) | bool) }}",
)
assert _check_task(task, Path("test.yml"), 1) == []
def test_task_with_password_in_name_only_no_false_positive(self):
"""Task name contains 'password' but no secret value — should not flag."""
task = _make_task(
"Configure passwdqc in common-password",
"ansible.builtin.lineinfile",
"password required pam_passwdqc.so min=disabled,disabled,16,12,8",
)
assert _check_task(task, Path("test.yml"), 1) == []
def test_task_with_password_in_module_param_no_false_positive(self):
"""Module param named 'password' but value is a literal — no Jinja."""
task = {
"name": "Set user password",
"ansible.builtin.user": {
"name": "deploy",
"password_lock": True,
},
}
assert _check_task(task, Path("test.yml"), 1) == []
def test_task_with_vault_password_variable_fails(self):
task = _make_task(
"Unsafe vault task",
"ansible.builtin.shell",
"echo {{ vault_zitadel_db_password }}",
)
violations = _check_task(task, Path("test.yml"), 1)
assert len(violations) == 1
def test_task_with_nested_dict_secret_fails(self):
"""Secrets in nested dict values (e.g. set_fact) should be caught."""
task = {
"name": "Set secrets",
"ansible.builtin.set_fact": {
"db_password": "{{ vault_db_password }}",
"api_key": "{{ vault_api_key }}",
},
}
violations = _check_task(task, Path("test.yml"), 1)
assert len(violations) == 1
def test_task_with_no_log_none_passes(self):
"""no_log: None should count as not set (flagged)."""
task = _make_task(
"Unsafe task",
"ansible.builtin.shell",
"echo {{ _secrets.db_password }}",
no_log=None,
)
violations = _check_task(task, Path("test.yml"), 1)
assert len(violations) == 1
def test_task_with_secret_in_list_value_fails(self):
"""Secrets inside list values should be caught."""
task = {
"name": "Task with list secret",
"ansible.builtin.set_fact": {
"items": ["{{ _secrets.api_key }}", "normal_value"],
},
}
violations = _check_task(task, Path("test.yml"), 1)
assert len(violations) == 1
def test_task_with_api_key_secret_fails(self):
"""api_key in Jinja expression should be caught."""
task = _make_task(
"Unsafe task",
"ansible.builtin.shell",
"echo {{ my_api_key }}",
)
violations = _check_task(task, Path("test.yml"), 1)
assert len(violations) == 1
def test_task_with_secret_in_jinja_fails(self):
"""_secret in Jinja expression should be caught."""
task = _make_task(
"Unsafe task",
"ansible.builtin.shell",
"echo {{ my_secret }}",
)
violations = _check_task(task, Path("test.yml"), 1)
assert len(violations) == 1
def test_task_with_access_token_fails(self):
"""access_token in Jinja expression should be caught."""
task = _make_task(
"Unsafe task",
"ansible.builtin.shell",
"echo {{ my_access_token }}",
)
violations = _check_task(task, Path("test.yml"), 1)
assert len(violations) == 1
def test_task_with_non_secret_non_dict_non_list_value(self):
"""Non-str, non-dict, non-list values (e.g. int) should not crash."""
task = _make_task(
"Task with int",
"ansible.builtin.shell",
"echo hello",
some_int=42,
)
assert _check_task(task, Path("test.yml"), 1) == []
class TestCheckDirectory:
def test_clean_directory_passes(self, tmp_path: Path):
"""A directory with no secret-handling tasks should pass."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text(
"- name: Normal task\n ansible.builtin.shell: echo hello\n changed_when: false\n"
)
assert check_directory(role_dir) == []
def test_unsafe_task_is_caught(self, tmp_path: Path):
"""A task with secrets but no no_log should be flagged."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text(
"- name: Unsafe task\n ansible.builtin.shell: echo {{ _secrets.db_password }}\n changed_when: false\n"
)
violations = check_directory(role_dir)
assert len(violations) == 1
assert "Unsafe task" in violations[0]
def test_molecule_files_are_skipped(self, tmp_path: Path):
"""Molecule test files should not be scanned."""
role_dir = tmp_path / "roles" / "test_role"
mol_dir = role_dir / "molecule" / "default" / "tasks"
mol_dir.mkdir(parents=True)
(mol_dir / "main.yml").write_text(
"- name: Unsafe task in molecule\n ansible.builtin.shell: echo {{ _secrets.db_password }}\n"
)
assert check_directory(role_dir) == []
def test_playbook_format_is_parsed(self, tmp_path: Path):
"""Playbook files (list of plays with 'hosts') should be parsed."""
pb_dir = tmp_path / "playbooks"
pb_dir.mkdir(parents=True)
(pb_dir / "test.yml").write_text(
"---\n"
"- name: Test play\n"
" hosts: all\n"
" tasks:\n"
" - name: Unsafe task\n"
" ansible.builtin.shell: echo {{ _secrets.db_password }}\n"
)
violations = check_directory(tmp_path)
assert len(violations) == 1
assert "Unsafe task" in violations[0]
def test_invalid_yaml_is_skipped(self, tmp_path: Path):
"""Invalid YAML files should be skipped, not crash."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text("{{ invalid yaml: [")
assert check_directory(role_dir) == []
def test_empty_yaml_doc_is_skipped(self, tmp_path: Path):
"""Empty YAML documents (None) should be skipped."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text("---\n")
assert check_directory(role_dir) == []
def test_non_dict_non_list_doc_is_skipped(self, tmp_path: Path):
"""YAML docs that are neither dict nor list should be skipped."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text("just a string\n")
assert check_directory(role_dir) == []
def test_task_file_with_non_dict_task_skipped(self, tmp_path: Path):
"""Non-dict items in a task list should be skipped."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text(
"- just a string\n- name: Safe task\n ansible.builtin.shell: echo hello\n"
)
assert check_directory(role_dir) == []
def test_secret_in_list_value_is_caught(self, tmp_path: Path):
"""Secrets inside list values should be caught."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text(
"- name: Task with list secret\n"
" ansible.builtin.set_fact:\n"
" items:\n"
' - "{{ _secrets.api_key }}"\n'
" - normal_value\n"
)
violations = check_directory(role_dir)
assert len(violations) == 1
def test_single_play_dict_format(self, tmp_path: Path):
"""A playbook that's a bare dict (not list of plays) should be parsed."""
pb_dir = tmp_path / "playbooks"
pb_dir.mkdir(parents=True)
(pb_dir / "test.yml").write_text(
"---\n"
"name: Single play\n"
"hosts: all\n"
"tasks:\n"
" - name: Unsafe task\n"
" ansible.builtin.shell: echo {{ _secrets.db_password }}\n"
)
violations = check_directory(tmp_path)
assert len(violations) == 1
def test_play_with_non_dict_play_skipped(self, tmp_path: Path):
"""Non-dict plays in a playbook list should be skipped."""
pb_dir = tmp_path / "playbooks"
pb_dir.mkdir(parents=True)
# First play is valid (makes is_plays=True), second is a non-dict
(pb_dir / "test.yml").write_text(
"---\n"
"- name: Safe play\n"
" hosts: all\n"
" tasks:\n"
" - name: Safe task\n"
" ansible.builtin.shell: echo hello\n"
'- "just a string as second play"\n'
)
assert check_directory(tmp_path) == []
def test_play_with_non_list_tasks_skipped(self, tmp_path: Path):
"""Plays where tasks is not a list should be skipped."""
pb_dir = tmp_path / "playbooks"
pb_dir.mkdir(parents=True)
(pb_dir / "test.yml").write_text('---\n- name: Play with bad tasks\n hosts: all\n tasks: "not a list"\n')
assert check_directory(tmp_path) == []
def test_play_with_non_dict_task_in_playbook(self, tmp_path: Path):
"""Non-dict tasks in a playbook should be skipped."""
pb_dir = tmp_path / "playbooks"
pb_dir.mkdir(parents=True)
(pb_dir / "test.yml").write_text(
"---\n"
"- name: Play\n"
" hosts: all\n"
" tasks:\n"
' - "just a string"\n'
" - name: Safe task\n"
" ansible.builtin.shell: echo hello\n"
)
assert check_directory(tmp_path) == []
def test_yaml_file_with_oserror_skipped(self, tmp_path: Path):
"""YAML files that can't be opened should be skipped."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
# Create a file that will cause OSError when opened
# (use a directory with .yml extension)
bad_file = role_dir / "tasks" / "main.yml"
bad_file.mkdir()
assert check_directory(role_dir) == []
class TestMain:
def test_main_passes_on_clean_dir(self, tmp_path: Path):
"""main() should exit 0 on a clean directory."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text(
"- name: Normal task\n ansible.builtin.shell: echo hello\n changed_when: false\n"
)
runner = CliRunner()
result = runner.invoke(main, ["--path", str(role_dir)])
assert result.exit_code == 0
assert "OK" in result.output or "no_log" in result.output
def test_main_fails_on_unsafe_dir(self, tmp_path: Path):
"""main() should exit 1 when violations are found."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text(
"- name: Unsafe task\n ansible.builtin.shell: echo {{ _secrets.db_password }}\n"
)
runner = CliRunner()
result = runner.invoke(main, ["--path", str(role_dir)])
assert result.exit_code == 1
assert "Unsafe task" in result.output
def test_main_returns_2_on_missing_dir(self, tmp_path: Path):
"""main() should exit 2 when the directory doesn't exist."""
runner = CliRunner()
result = runner.invoke(main, ["--path", str(tmp_path / "nonexistent")])
assert result.exit_code == 2
def test_main_with_ansible_dir_option(self, tmp_path: Path):
"""main() --ansible-dir should work like --path."""
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text(
"- name: Unsafe task\n ansible.builtin.shell: echo {{ _secrets.db_password }}\n"
)
runner = CliRunner()
result = runner.invoke(main, ["--ansible-dir", str(tmp_path)])
assert result.exit_code == 1
def test_main_no_path_no_ansible_dir_uses_default(self, tmp_path: Path, monkeypatch):
"""main() with no args uses DEFAULT_ANSIBLE_DIR."""
import devx.tools.check_ansible_no_log as mod
role_dir = tmp_path / "roles" / "test_role"
(role_dir / "tasks").mkdir(parents=True)
(role_dir / "tasks" / "main.yml").write_text("- name: Normal task\n ansible.builtin.shell: echo hello\n")
monkeypatch.setattr(mod, "DEFAULT_ANSIBLE_DIR", tmp_path)
runner = CliRunner()
result = runner.invoke(main, [])
assert result.exit_code == 0
@@ -0,0 +1,198 @@
"""Unit tests for devx.tools.check_ansible_no_state_absent_on_db."""
from __future__ import annotations
from pathlib import Path
from click.testing import CliRunner
from devx.tools.check_ansible_no_state_absent_on_db import _check_file, _find_task_files, main
class TestCheckFile:
def test_clean_file_no_db_paths(self, tmp_path: Path):
"""A file with no DB paths should produce no violations."""
p = tmp_path / "test.yml"
p.write_text("- name: Safe task\n ansible.builtin.file:\n path: /opt/app/data\n state: directory\n")
assert _check_file(p, tmp_path) == []
def test_state_absent_on_zitadel_db_fails(self, tmp_path: Path):
"""state: absent on zitadel-db path should be flagged."""
p = tmp_path / "test.yml"
p.write_text(
"- name: Dangerous wipe\n ansible.builtin.file:\n path: /opt/postgres/zitadel-db\n state: absent\n"
)
violations = _check_file(p, tmp_path)
assert len(violations) >= 1
assert "state" in violations[0].lower() or "absent" in violations[0].lower()
def test_state_absent_on_var_lib_postgresql_fails(self, tmp_path: Path):
"""state: absent on /var/lib/postgresql/data should be flagged."""
p = tmp_path / "test.yml"
p.write_text(
"- name: Dangerous wipe\n ansible.builtin.file:\n path: /var/lib/postgresql/data\n state: absent\n"
)
violations = _check_file(p, tmp_path)
assert len(violations) >= 1
def test_state_absent_on_app_db_fails(self, tmp_path: Path):
"""state: absent on any *-db path should be flagged."""
p = tmp_path / "test.yml"
p.write_text(
"- name: Dangerous wipe\n ansible.builtin.file:\n path: /opt/postgres/gitea-db\n state: absent\n"
)
violations = _check_file(p, tmp_path)
assert len(violations) >= 1
def test_state_absent_with_pg_upgrade_context_passes(self, tmp_path: Path):
"""state: absent near DB path with upgrade-postgres context should pass."""
p = tmp_path / "test.yml"
p.write_text(
"- name: PG upgrade — remove old data\n"
" ansible.builtin.file:\n"
" path: /opt/postgres/zitadel-db\n"
" state: absent\n"
" when: pg_version_changed | default(false)\n"
)
violations = _check_file(p, tmp_path)
assert violations == []
def test_state_absent_with_pg_version_context_passes(self, tmp_path: Path):
"""state: absent near DB path with PG_VERSION context should pass."""
p = tmp_path / "test.yml"
p.write_text(
"- name: PG upgrade\n"
" ansible.builtin.file:\n"
" path: /opt/postgres/zitadel-db\n"
" state: absent\n"
" when: PG_VERSION is defined\n"
)
violations = _check_file(p, tmp_path)
assert violations == []
def test_state_absent_with_allow_marker_passes(self, tmp_path: Path):
"""state: absent with lint:allow-state-absent comment should pass."""
p = tmp_path / "test.yml"
p.write_text(
"# lint:allow-state-absent\n"
"- name: Intentional wipe\n"
" ansible.builtin.file:\n"
" path: /opt/postgres/zitadel-db\n"
" state: absent\n"
)
violations = _check_file(p, tmp_path)
assert violations == []
def test_state_present_on_db_path_passes(self, tmp_path: Path):
"""state: present (not absent) on DB path should pass."""
p = tmp_path / "test.yml"
p.write_text(
"- name: Safe task\n ansible.builtin.file:\n path: /opt/postgres/zitadel-db\n state: directory\n"
)
assert _check_file(p, tmp_path) == []
def test_nonexistent_file_returns_empty(self):
"""A nonexistent file should return no violations."""
assert _check_file(Path("/nonexistent/path/file.yml"), Path.cwd()) == []
def test_rm_rf_db_fails(self, tmp_path: Path):
"""rm -rf on a DB path should be flagged."""
p = tmp_path / "test.yml"
p.write_text("- name: Dangerous wipe\n ansible.builtin.shell: rm -rf /opt/postgres/zitadel-db\n")
violations = _check_file(p, tmp_path)
assert len(violations) >= 1
def test_relative_path_outside_repo(self, tmp_path: Path):
"""Files outside repo_root use the full path in display."""
p = tmp_path / "test.yml"
p.write_text(
"- name: Dangerous wipe\n ansible.builtin.file:\n path: /opt/postgres/zitadel-db\n state: absent\n"
)
violations = _check_file(p, Path("/other/repo"))
assert len(violations) >= 1
assert str(tmp_path) in violations[0] or "test.yml" in violations[0]
class TestFindTaskFiles:
def test_find_yml_files_in_directory(self, tmp_path: Path):
"""Should find .yml files in a directory."""
(tmp_path / "tasks").mkdir()
(tmp_path / "tasks" / "main.yml").write_text("[]")
(tmp_path / "tasks" / "other.yaml").write_text("[]")
files = _find_task_files(tmp_path)
assert len(files) == 2
def test_skip_molecule_files(self, tmp_path: Path):
"""Should skip files in molecule directories."""
(tmp_path / "molecule").mkdir()
(tmp_path / "molecule" / "test.yml").write_text("[]")
(tmp_path / "main.yml").write_text("[]")
files = _find_task_files(tmp_path)
assert len(files) == 1
assert "molecule" not in files[0].parts
def test_single_file_input(self, tmp_path: Path):
"""Should return the file itself if it's a .yml file."""
f = tmp_path / "test.yml"
f.write_text("[]")
files = _find_task_files(f)
assert files == [f]
def test_nonexistent_path_returns_empty(self):
"""A path that is neither a file nor a dir should return []."""
files = _find_task_files(Path("/nonexistent/path/that/does/not/exist"))
assert files == []
def test_non_yaml_file_skipped(self, tmp_path: Path):
"""Non-YAML files should not be included."""
f = tmp_path / "readme.txt"
f.write_text("not yaml")
assert _find_task_files(f) == []
class TestMain:
def test_main_no_violations_exit_zero(self, tmp_path: Path):
"""main() with a clean file should exit 0."""
f = tmp_path / "clean.yml"
f.write_text("- name: Safe task\n ansible.builtin.file:\n path: /opt/app\n state: directory\n")
result = CliRunner().invoke(main, ["--path", str(f)])
assert result.exit_code == 0
assert "OK" in result.output
def test_main_with_violations_exit_one(self, tmp_path: Path):
"""main() with a state: absent on a DB path should exit 1."""
f = tmp_path / "dangerous.yml"
f.write_text(
"- name: Dangerous wipe\n ansible.builtin.file:\n path: /opt/postgres/zitadel-db\n state: absent\n"
)
result = CliRunner().invoke(main, ["--path", str(f)])
assert result.exit_code == 1
assert "FAIL" in result.output
def test_main_path_to_clean_file(self, tmp_path: Path):
"""main() --path pointing to a specific clean file should exit 0."""
f = tmp_path / "tasks.yml"
f.write_text("- name: Safe\n ansible.builtin.file:\n path: /opt/app\n state: directory\n")
result = CliRunner().invoke(main, ["--path", str(f)])
assert result.exit_code == 0
def test_main_default_dirs_no_violations(self, tmp_path: Path, monkeypatch):
"""main() with no --path scans default dirs and exits 0."""
import devx.tools.check_ansible_no_state_absent_on_db as mod
(tmp_path / "clean.yml").write_text(
"- name: Safe\n ansible.builtin.file:\n path: /opt/app\n state: directory\n"
)
monkeypatch.setattr(mod, "DEFAULT_ANSIBLE_DIRS", [tmp_path])
result = CliRunner().invoke(main, [])
assert result.exit_code == 0
assert "OK" in result.output
def test_main_custom_ansible_dirs(self, tmp_path: Path):
"""main() --ansible-dir should work."""
f = tmp_path / "dangerous.yml"
f.write_text(
"- name: Dangerous wipe\n ansible.builtin.file:\n path: /opt/postgres/zitadel-db\n state: absent\n"
)
result = CliRunner().invoke(main, ["--ansible-dir", str(tmp_path)])
assert result.exit_code == 1
@@ -0,0 +1,340 @@
"""Unit tests for devx.tools.check_ansible_patterns."""
from __future__ import annotations
from pathlib import Path
from click.testing import CliRunner
from devx.tools.check_ansible_patterns import (
_check_file,
_check_task,
_check_tasks,
_find_task_files,
_is_legitimate_devnull,
_is_legitimate_or_true,
main,
)
def _make_task(name: str, action: str, value: str, **extra: object) -> dict:
"""Build a minimal task dict for testing."""
task: dict = {"name": name, action: value}
task.update(extra)
return task
class TestIsLegitimateOrTrue:
def test_cleanup_task_name_is_legitimate(self):
assert _is_legitimate_or_true("docker rm old-container", "Remove old container")
def test_prune_task_name_is_legitimate(self):
assert _is_legitimate_or_true("docker image prune -f", "Prune unused images")
def test_docker_rm_command_is_legitimate(self):
assert _is_legitimate_or_true("docker rm -f mycontainer", "Some task")
def test_provision_task_is_not_legitimate(self):
assert not _is_legitimate_or_true("curl -X POST https://api/app || true", "Provision OIDC client")
def test_sync_task_name_is_legitimate(self):
assert _is_legitimate_or_true("psql -c 'ALTER USER' || true", "Sync PostgreSQL password")
class TestCheckTask:
def test_or_true_on_provision_task_fails(self, tmp_path: Path):
task = _make_task(
"Provision OIDC client",
"ansible.builtin.shell",
"curl -X POST https://zitadel/api || true",
)
violations = _check_task(task, tmp_path / "test.yml", 1, tmp_path)
assert len(violations) >= 1
assert "|| true" in violations[0]
def test_or_true_on_cleanup_task_passes(self, tmp_path: Path):
task = _make_task(
"Remove old container",
"ansible.builtin.shell",
"docker rm -f old-container || true",
)
assert _check_task(task, tmp_path / "test.yml", 1, tmp_path) == []
def test_failed_when_false_on_provision_fails(self, tmp_path: Path):
task = _make_task(
"Provision OIDC client",
"ansible.builtin.shell",
"curl -X POST https://zitadel/api",
failed_when=False,
)
violations = _check_task(task, tmp_path / "test.yml", 1, tmp_path)
assert any("failed_when" in v for v in violations)
def test_failed_when_false_on_stop_passes(self, tmp_path: Path):
task = _make_task(
"Stop ZITADEL containers",
"ansible.builtin.shell",
"docker stop zitadel",
failed_when=False,
)
assert _check_task(task, tmp_path / "test.yml", 1, tmp_path) == []
def test_failed_when_false_on_check_passes(self, tmp_path: Path):
task = _make_task(
"Check if ZITADEL is running",
"ansible.builtin.shell",
"docker inspect zitadel",
failed_when=False,
)
assert _check_task(task, tmp_path / "test.yml", 1, tmp_path) == []
def test_allow_marker_in_name_passes(self, tmp_path: Path):
task = _make_task(
"Provision OIDC #lint:allow-failure-masking",
"ansible.builtin.shell",
"curl -X POST https://zitadel/api || true",
failed_when=False,
)
assert _check_task(task, tmp_path / "test.yml", 1, tmp_path) == []
def test_safe_task_no_violations(self, tmp_path: Path):
task = _make_task(
"Create directory",
"ansible.builtin.file",
"path=/opt/app state=directory",
)
assert _check_task(task, tmp_path / "test.yml", 1, tmp_path) == []
def test_relative_path_outside_repo(self, tmp_path: Path):
"""Files outside repo_root use the full path in display."""
task = _make_task(
"Provision OIDC",
"ansible.builtin.shell",
"curl || true",
)
other_dir = Path("/tmp/other")
violations = _check_task(task, other_dir / "test.yml", 1, tmp_path)
assert len(violations) >= 1
class TestCheckFile:
def test_clean_file_passes(self, tmp_path: Path):
p = tmp_path / "test.yml"
p.write_text("- name: Safe task\n ansible.builtin.file:\n path: /opt/app\n state: directory\n")
assert _check_file(p, tmp_path) == []
def test_dangerous_pattern_detected(self, tmp_path: Path):
p = tmp_path / "test.yml"
p.write_text(
"- name: Provision OIDC\n"
" ansible.builtin.shell: |\n"
" curl -X POST https://api/app || true\n"
" failed_when: false\n"
)
violations = _check_file(p, tmp_path)
assert len(violations) >= 1
def test_file_level_allow_marker_passes(self, tmp_path: Path):
p = tmp_path / "test.yml"
p.write_text(
"# lint:allow-failure-masking\n"
"- name: Provision OIDC\n"
" ansible.builtin.shell: |\n"
" curl -X POST https://api/app || true\n"
" failed_when: false\n"
)
assert _check_file(p, tmp_path) == []
def test_nonexistent_file_returns_empty(self):
assert _check_file(Path("/nonexistent/path/file.yml"), Path.cwd()) == []
def test_yaml_parse_error_returns_empty(self, tmp_path: Path):
p = tmp_path / "test.yml"
p.write_text("name: Provision OIDC\n shell: curl || true\n: invalid: [")
assert _check_file(p, tmp_path) == []
def test_dict_doc_playbook_with_tasks(self, tmp_path: Path):
p = tmp_path / "playbook.yml"
p.write_text(
"- hosts: all\n"
" tasks:\n"
" - name: Provision OIDC\n"
" ansible.builtin.shell: curl -X POST https://api/app || true\n"
)
violations = _check_file(p, tmp_path)
assert any("|| true" in v for v in violations)
def test_dict_doc_with_pre_tasks_and_post_tasks(self, tmp_path: Path):
p = tmp_path / "playbook.yml"
p.write_text(
"- hosts: all\n"
" pre_tasks:\n"
" - name: Provision secret\n"
" ansible.builtin.shell: curl -X POST https://api/app || true\n"
" post_tasks:\n"
" - name: Provision OIDC\n"
" ansible.builtin.shell: curl -X POST https://api/app || true\n"
" handlers:\n"
" - name: Provision password\n"
" ansible.builtin.shell: curl -X POST https://api/app || true\n"
)
violations = _check_file(p, tmp_path)
assert len(violations) >= 3
def test_block_tasks_in_list_item(self, tmp_path: Path):
p = tmp_path / "tasks.yml"
p.write_text(
"- name: Outer task\n"
" block:\n"
" - name: Provision OIDC\n"
" ansible.builtin.shell: curl -X POST https://api/app || true\n"
" - name: Provision secret\n"
" ansible.builtin.shell: curl -X POST https://api/app || true\n"
)
violations = _check_file(p, tmp_path)
assert any("|| true" in v for v in violations)
def test_empty_doc_skipped(self, tmp_path: Path):
p = tmp_path / "test.yml"
p.write_text("---\nnull\n---\n- name: Provision OIDC\n ansible.builtin.shell: curl || true\n")
violations = _check_file(p, tmp_path)
assert any("|| true" in v for v in violations)
def test_pure_dict_doc_with_tasks(self, tmp_path: Path):
p = tmp_path / "playbook.yml"
p.write_text(
"hosts: all\n"
"tasks:\n"
" - name: Provision OIDC\n"
" ansible.builtin.shell: curl -X POST https://api/app || true\n"
)
violations = _check_file(p, tmp_path)
assert any("|| true" in v for v in violations)
class TestIsLegitimateDevnull:
def test_cleanup_task_is_legitimate(self):
assert _is_legitimate_devnull("docker rm old-container 2>/dev/null", "Remove old container")
def test_provision_task_is_not_legitimate(self):
assert not _is_legitimate_devnull("curl -X POST https://api/app 2>/dev/null", "Provision OIDC client")
class TestCheckTasks:
def test_tasks_section_checked(self, tmp_path: Path):
doc = {
"tasks": [
{"name": "Provision OIDC", "ansible.builtin.shell": "curl || true"},
],
}
errors: list[str] = []
_check_tasks(doc, tmp_path / "test.yml", errors, tmp_path)
assert any("|| true" in e for e in errors)
def test_block_inside_tasks_section(self, tmp_path: Path):
doc = {
"tasks": [
{
"name": "Outer",
"block": [
{"name": "Provision secret", "ansible.builtin.shell": "curl || true"},
],
},
],
}
errors: list[str] = []
_check_tasks(doc, tmp_path / "test.yml", errors, tmp_path)
assert any("|| true" in e for e in errors)
def test_non_list_section_ignored(self, tmp_path: Path):
doc = {"tasks": "not a list"}
errors: list[str] = []
_check_tasks(doc, tmp_path / "test.yml", errors, tmp_path)
assert errors == []
def test_non_dict_task_ignored(self, tmp_path: Path):
doc = {"tasks": ["just a string"]}
errors: list[str] = []
_check_tasks(doc, tmp_path / "test.yml", errors, tmp_path)
assert errors == []
class TestFindTaskFiles:
def test_single_file(self, tmp_path: Path):
p = tmp_path / "main.yml"
p.write_text("- name: test\n")
assert _find_task_files(p) == [p]
def test_single_yaml_file(self, tmp_path: Path):
p = tmp_path / "main.yaml"
p.write_text("- name: test\n")
assert _find_task_files(p) == [p]
def test_non_yaml_file_returns_empty(self, tmp_path: Path):
p = tmp_path / "main.txt"
p.write_text("hello\n")
assert _find_task_files(p) == []
def test_directory_finds_yaml_files(self, tmp_path: Path):
(tmp_path / "a.yml").write_text("- name: a\n")
(tmp_path / "sub").mkdir()
(tmp_path / "sub" / "b.yaml").write_text("- name: b\n")
(tmp_path / "ignore.txt").write_text("nope\n")
result = _find_task_files(tmp_path)
names = {f.name for f in result}
assert names == {"a.yml", "b.yaml"}
def test_directory_skips_molecule(self, tmp_path: Path):
(tmp_path / "a.yml").write_text("- name: a\n")
(tmp_path / "molecule").mkdir()
(tmp_path / "molecule" / "scenario.yml").write_text("- name: mol\n")
result = _find_task_files(tmp_path)
assert all("molecule" not in f.parts for f in result)
def test_nonexistent_path_returns_empty(self):
assert _find_task_files(Path("/nonexistent/path/xyz")) == []
class TestMain:
def test_main_clean_file_exit_zero(self, tmp_path: Path):
p = tmp_path / "clean.yml"
p.write_text("- name: Safe task\n ansible.builtin.file:\n path: /opt/app\n state: directory\n")
result = CliRunner().invoke(main, ["--path", str(p)])
assert result.exit_code == 0
assert "OK" in result.output
def test_main_violation_exit_one(self, tmp_path: Path):
p = tmp_path / "bad.yml"
p.write_text(
"- name: Provision OIDC\n"
" ansible.builtin.shell: curl -X POST https://api/app || true\n"
" failed_when: false\n"
)
result = CliRunner().invoke(main, ["--path", str(p)])
assert result.exit_code == 1
assert "FAIL" in result.output
def test_main_directory(self, tmp_path: Path):
(tmp_path / "clean.yml").write_text(
"- name: Safe task\n ansible.builtin.file:\n path: /opt\n state: directory\n"
)
result = CliRunner().invoke(main, ["--path", str(tmp_path)])
assert result.exit_code == 0
def test_main_default_dirs(self, tmp_path: Path, monkeypatch):
import devx.tools.check_ansible_patterns as mod
(tmp_path / "clean.yml").write_text(
"- name: Safe task\n ansible.builtin.file:\n path: /opt\n state: directory\n"
)
monkeypatch.setattr(mod, "DEFAULT_ANSIBLE_DIRS", [tmp_path])
result = CliRunner().invoke(main, [])
assert result.exit_code == 0
assert "OK" in result.output
def test_main_custom_ansible_dirs(self, tmp_path: Path):
(tmp_path / "bad.yml").write_text(
"- name: Provision OIDC\n ansible.builtin.shell: curl -X POST https://api/app || true\n"
)
result = CliRunner().invoke(main, ["--ansible-dir", str(tmp_path)])
assert result.exit_code == 1
@@ -0,0 +1,358 @@
"""Unit tests for devx.tools.check_ansible_set_fact_to_json."""
from __future__ import annotations
import textwrap
from pathlib import Path
from click.testing import CliRunner
from devx.tools.check_ansible_set_fact_to_json import (
_check_file,
_check_task,
_check_task_list,
_find_task_files,
main,
)
class TestFindTaskFiles:
def test_single_file(self, tmp_path: Path):
f = tmp_path / "tasks.yml"
f.write_text("tasks: []")
assert _find_task_files(f) == [f]
def test_directory_recursive(self, tmp_path: Path):
(tmp_path / "sub").mkdir()
f1 = tmp_path / "a.yml"
f2 = tmp_path / "sub" / "b.yml"
f1.write_text("tasks: []")
f2.write_text("tasks: []")
result = _find_task_files(tmp_path)
assert f1 in result
assert f2 in result
def test_nonexistent_path(self, tmp_path: Path):
assert _find_task_files(tmp_path / "nonexistent") == []
def test_non_yaml_file_skipped(self, tmp_path: Path):
f = tmp_path / "readme.txt"
f.write_text("not yaml")
assert _find_task_files(f) == []
class TestCheckTask:
def test_set_fact_with_to_json_flagged(self, tmp_path: Path):
task = {"name": "Set targets", "set_fact": {"customer_hosts": "{{ targets | to_json }}"}}
errors: list[str] = []
_check_task(task, tmp_path / "test.yml", errors, tmp_path)
assert len(errors) == 1
assert "customer_hosts" in errors[0]
assert "to_json" in errors[0]
def test_set_fact_without_to_json_ok(self, tmp_path: Path):
task = {"name": "Set targets", "set_fact": {"customer_hosts": "{{ targets }}"}}
errors: list[str] = []
_check_task(task, tmp_path / "test.yml", errors, tmp_path)
assert errors == []
def test_ansible_builtin_set_fact(self, tmp_path: Path):
task = {"name": "Set targets", "ansible.builtin.set_fact": {"my_list": "{{ items | to_nice_json }}"}}
errors: list[str] = []
_check_task(task, tmp_path / "test.yml", errors, tmp_path)
assert len(errors) == 1
assert "to_nice_json" in errors[0]
def test_non_set_fact_task_ignored(self, tmp_path: Path):
task = {"name": "Render config", "copy": {"content": "{{ data | to_json }}"}}
errors: list[str] = []
_check_task(task, tmp_path / "test.yml", errors, tmp_path)
assert errors == []
def test_cacheable_key_ignored(self, tmp_path: Path):
task = {"name": "Set fact", "set_fact": {"my_var": "{{ value }}", "cacheable": True}}
errors: list[str] = []
_check_task(task, tmp_path / "test.yml", errors, tmp_path)
assert errors == []
def test_unnamed_task(self, tmp_path: Path):
task = {"set_fact": {"my_var": "{{ value | to_json }}"}}
errors: list[str] = []
_check_task(task, tmp_path / "test.yml", errors, tmp_path)
assert len(errors) == 1
assert "(unnamed)" in errors[0]
def test_set_fact_not_dict_ignored(self, tmp_path: Path):
task = {"set_fact": "not a dict"}
errors: list[str] = []
_check_task(task, tmp_path / "test.yml", errors, tmp_path)
assert errors == []
def test_to_json_no_spaces(self, tmp_path: Path):
task = {"set_fact": {"my_var": "{{ items|to_json }}"}}
errors: list[str] = []
_check_task(task, tmp_path / "test.yml", errors, tmp_path)
assert len(errors) == 1
class TestCheckTaskList:
def test_block_tasks_checked(self, tmp_path: Path):
tasks = [{"name": "Block", "block": [{"name": "Set in block", "set_fact": {"x": "{{ y | to_json }}"}}]}]
errors: list[str] = []
_check_task_list(tasks, tmp_path / "test.yml", errors, tmp_path)
assert len(errors) == 1
assert "x" in errors[0]
def test_non_dict_task_ignored(self, tmp_path: Path):
tasks = ["just a string", 42, None]
errors: list[str] = []
_check_task_list(tasks, tmp_path / "test.yml", errors, tmp_path)
assert errors == []
class TestCheckFile:
def test_playbook_with_set_fact_to_json(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Deploy
hosts: all
tasks:
- name: Set targets
ansible.builtin.set_fact:
customer_hosts: "{{ targets | to_json }}"
""").strip()
f = tmp_path / "playbook.yml"
f.write_text(content)
errors = _check_file(f, tmp_path)
assert len(errors) == 1
assert "customer_hosts" in errors[0]
def test_playbook_without_set_fact(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Deploy
hosts: all
tasks:
- name: Debug
ansible.builtin.debug:
msg: "hello"
""").strip()
f = tmp_path / "playbook.yml"
f.write_text(content)
assert _check_file(f, tmp_path) == []
def test_role_tasks_file(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Set config
set_fact:
my_data: "{{ data | to_json }}"
- name: Copy config
copy:
content: "{{ config | to_json }}"
dest: /etc/config.json
""").strip()
f = tmp_path / "main.yml"
f.write_text(content)
errors = _check_file(f, tmp_path)
assert len(errors) == 1
assert "my_data" in errors[0]
def test_pre_tasks_and_post_tasks(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Play
hosts: all
pre_tasks:
- name: Pre set
set_fact:
pre_var: "{{ x | to_json }}"
post_tasks:
- name: Post set
set_fact:
post_var: "{{ y | to_json }}"
""").strip()
f = tmp_path / "playbook.yml"
f.write_text(content)
errors = _check_file(f, tmp_path)
assert len(errors) == 2
def test_handlers_checked(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Play
hosts: all
handlers:
- name: Restart service
set_fact:
restart_data: "{{ data | to_json }}"
""").strip()
f = tmp_path / "playbook.yml"
f.write_text(content)
errors = _check_file(f, tmp_path)
assert len(errors) == 1
def test_invalid_yaml(self, tmp_path: Path):
f = tmp_path / "bad.yml"
f.write_text("tasks: [invalid: {")
errors = _check_file(f, tmp_path)
assert len(errors) == 1
assert "cannot parse YAML" in errors[0]
def test_non_dict_doc_skipped(self, tmp_path: Path):
f = tmp_path / "list.yml"
f.write_text("- just\n- a\n- list\n")
assert _check_file(f, tmp_path) == []
def test_multi_doc_yaml(self, tmp_path: Path):
content = textwrap.dedent("""
---
- name: Play 1
hosts: all
tasks:
- name: Set in play 1
set_fact:
var1: "{{ x | to_json }}"
---
- name: Play 2
hosts: all
tasks:
- name: Set in play 2
set_fact:
var2: "{{ y }}"
""").strip()
f = tmp_path / "multi.yml"
f.write_text(content)
errors = _check_file(f, tmp_path)
assert len(errors) == 1
assert "var1" in errors[0]
def test_dict_doc_role_tasks_file(self, tmp_path: Path):
content = textwrap.dedent("""
tasks:
- name: Set var
set_fact:
my_var: "{{ value | to_json }}"
""").strip()
f = tmp_path / "main.yml"
f.write_text(content)
errors = _check_file(f, tmp_path)
assert len(errors) == 1
assert "my_var" in errors[0]
def test_play_with_roles_key(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Play
hosts: all
roles:
- role: my_role
tasks:
- name: Set in role
set_fact:
role_var: "{{ x | to_json }}"
""").strip()
f = tmp_path / "playbook.yml"
f.write_text(content)
errors = _check_file(f, tmp_path)
assert len(errors) == 1
assert "role_var" in errors[0]
def test_bare_task_in_list_with_block(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Outer task
set_fact:
outer: "{{ x | to_json }}"
- name: Block
block:
- name: Inner task
set_fact:
inner: "{{ y | to_json }}"
""").strip()
f = tmp_path / "tasks.yml"
f.write_text(content)
errors = _check_file(f, tmp_path)
assert len(errors) == 2
def test_check_file_os_error(self, tmp_path: Path, monkeypatch) -> None:
"""OSError reading a file returns empty errors (not a crash)."""
f = tmp_path / "playbook.yml"
f.write_text("- name: ok\n set_fact:\n x: 1\n")
def _raise(*args, **kwargs):
raise OSError("disk error")
monkeypatch.setattr(Path, "read_text", _raise)
errors = _check_file(f, tmp_path)
assert errors == []
class TestMain:
def test_passes_when_clean(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Play
hosts: all
tasks:
- name: Set var
set_fact:
my_var: "{{ value }}"
""").strip()
f = tmp_path / "playbook.yml"
f.write_text(content)
runner = CliRunner()
result = runner.invoke(main, ["--path", str(f)])
assert result.exit_code == 0
assert "OK" in result.output
def test_fails_when_to_json_found(self, tmp_path: Path):
content = textwrap.dedent("""
- name: Play
hosts: all
tasks:
- name: Set var
set_fact:
my_var: "{{ value | to_json }}"
""").strip()
f = tmp_path / "playbook.yml"
f.write_text(content)
runner = CliRunner()
result = runner.invoke(main, ["--path", str(f)])
assert result.exit_code == 1
assert "FAIL" in result.output
assert "my_var" in result.output
def test_directory_scan(self, tmp_path: Path):
(tmp_path / "good.yml").write_text(
textwrap.dedent("""
- name: Play
hosts: all
tasks:
- name: Set
set_fact:
x: "{{ y }}"
""").strip()
)
(tmp_path / "bad.yml").write_text(
textwrap.dedent("""
- name: Play
hosts: all
tasks:
- name: Set
set_fact:
x: "{{ y | to_json }}"
""").strip()
)
runner = CliRunner()
result = runner.invoke(main, ["--path", str(tmp_path)])
assert result.exit_code == 1
assert "bad.yml" in result.output
def test_custom_ansible_dirs(self, tmp_path: Path):
(tmp_path / "playbook.yml").write_text(
textwrap.dedent("""
- name: Play
hosts: all
tasks:
- name: Set
set_fact:
x: "{{ y | to_json }}"
""").strip()
)
runner = CliRunner()
result = runner.invoke(main, ["--ansible-dir", str(tmp_path)])
assert result.exit_code == 1
assert "playbook.yml" in result.output
+291
View File
@@ -0,0 +1,291 @@
"""Unit tests for devx.tools.check_docker_init."""
from __future__ import annotations
import textwrap
from pathlib import Path
from click.testing import CliRunner
from devx.tools.check_docker_init import _check_template, _find_compose_templates, _parse_services, main
class TestFindComposeTemplates:
def test_finds_docker_compose_templates(self, tmp_path: Path):
(tmp_path / "docker-compose.observability.yml.j2").write_text("services:")
(tmp_path / "docker-compose.service.yml.j2").write_text("services:")
result = _find_compose_templates(tmp_path)
assert len(result) == 2
def test_finds_exporters_compose(self, tmp_path: Path):
(tmp_path / "exporters-compose.yml.j2").write_text("services:")
result = _find_compose_templates(tmp_path)
assert len(result) == 1
assert "exporters-compose" in str(result[0])
def test_finds_compose_yaml_templates(self, tmp_path: Path):
(tmp_path / "compose.yaml.j2").write_text("services:")
result = _find_compose_templates(tmp_path)
assert len(result) == 1
def test_single_file(self, tmp_path: Path):
f = tmp_path / "docker-compose.test.yml.j2"
f.write_text("services:")
result = _find_compose_templates(f)
assert result == [f]
def test_nonexistent_path(self, tmp_path: Path):
assert _find_compose_templates(tmp_path / "nonexistent") == []
def test_deduplicates(self, tmp_path: Path):
(tmp_path / "docker-compose.yml.j2").write_text("services:")
result = _find_compose_templates(tmp_path)
assert len(result) == 1
def test_recursive(self, tmp_path: Path):
(tmp_path / "sub").mkdir()
(tmp_path / "sub" / "docker-compose.yml.j2").write_text("services:")
result = _find_compose_templates(tmp_path)
assert len(result) == 1
class TestParseServices:
def test_basic_services(self):
content = textwrap.dedent("""
services:
web:
image: nginx
healthcheck:
test: ["CMD", "curl", "localhost"]
db:
image: postgres
networks:
default:
""").strip()
services = _parse_services(content)
assert "web" in services
assert "db" in services
assert any("image: nginx" in line for line in services["web"])
def test_jinja2_service_names(self):
content = textwrap.dedent("""
services:
{{ app_name }}:
image: {{ app_image }}
healthcheck:
test: ["CMD", "curl"]
{{ app_name }}-db:
image: postgres
networks:
traefik:
""").strip()
services = _parse_services(content)
assert "{{ app_name }}" in services
assert "{{ app_name }}-db" in services
def test_no_services_section(self):
content = "version: '3'\nvolumes:\n data:"
assert _parse_services(content) == {}
def test_service_at_end_of_file(self):
content = textwrap.dedent("""
services:
web:
image: nginx
""").strip()
services = _parse_services(content)
assert "web" in services
def test_volumes_ends_services(self):
content = textwrap.dedent("""
services:
web:
image: nginx
volumes:
data:
""").strip()
services = _parse_services(content)
assert "web" in services
assert "data" not in services
class TestCheckTemplate:
def test_service_with_healthcheck_and_init_ok(self, tmp_path: Path):
content = textwrap.dedent("""
services:
web:
image: nginx
init: true
healthcheck:
test: ["CMD", "curl", "localhost"]
networks:
default:
""").strip()
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
assert _check_template(f, tmp_path) == []
def test_service_with_healthcheck_no_init_flagged(self, tmp_path: Path):
content = textwrap.dedent("""
services:
web:
image: nginx
healthcheck:
test: ["CMD", "curl", "localhost"]
networks:
default:
""").strip()
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
errors = _check_template(f, tmp_path)
assert len(errors) == 1
assert "web" in errors[0]
assert "init: true" in errors[0]
def test_service_without_healthcheck_ok(self, tmp_path: Path):
content = textwrap.dedent("""
services:
web:
image: nginx
networks:
default:
""").strip()
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
assert _check_template(f, tmp_path) == []
def test_multiple_services_some_missing(self, tmp_path: Path):
content = textwrap.dedent("""
services:
good:
image: nginx
init: true
healthcheck:
test: ["CMD", "curl"]
bad:
image: redis
healthcheck:
test: ["CMD", "redis-cli", "ping"]
networks:
default:
""").strip()
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
errors = _check_template(f, tmp_path)
assert len(errors) == 1
assert "bad" in errors[0]
assert "good" not in errors[0]
def test_no_services_section(self, tmp_path: Path):
content = "version: '3'\nvolumes:\n data:"
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
assert _check_template(f, tmp_path) == []
def test_jinja2_conditional_service(self, tmp_path: Path):
content = textwrap.dedent("""
services:
{% if backup_enabled %}
backup:
image: backup
healthcheck:
test: ["CMD-SHELL", "pgrep backup"]
{% endif %}
networks:
default:
""").strip()
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
errors = _check_template(f, tmp_path)
assert len(errors) == 1
assert "backup" in errors[0]
def test_relative_path_in_error(self, tmp_path: Path):
content = textwrap.dedent("""
services:
web:
image: nginx
healthcheck:
test: ["CMD"]
networks:
default:
""").strip()
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
errors = _check_template(f, tmp_path)
assert len(errors) == 1
assert "docker-compose.yml.j2" in errors[0]
assert str(tmp_path) not in errors[0]
class TestMain:
def test_passes_when_all_ok(self, tmp_path: Path):
content = textwrap.dedent("""
services:
web:
image: nginx
init: true
healthcheck:
test: ["CMD"]
networks:
default:
""").strip()
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
runner = CliRunner()
result = runner.invoke(main, ["--path", str(f)])
assert result.exit_code == 0
assert "OK" in result.output
def test_fails_when_missing_init(self, tmp_path: Path):
content = textwrap.dedent("""
services:
web:
image: nginx
healthcheck:
test: ["CMD"]
networks:
default:
""").strip()
f = tmp_path / "docker-compose.yml.j2"
f.write_text(content)
runner = CliRunner()
result = runner.invoke(main, ["--path", str(f)])
assert result.exit_code == 1
assert "FAIL" in result.output
assert "web" in result.output
def test_default_dir(self, tmp_path: Path):
(tmp_path / "docker-compose.good.yml.j2").write_text(
textwrap.dedent("""
services:
web:
image: nginx
init: true
healthcheck:
test: ["CMD"]
networks:
default:
""").strip()
)
(tmp_path / "docker-compose.bad.yml.j2").write_text(
textwrap.dedent("""
services:
db:
image: postgres
healthcheck:
test: ["CMD"]
networks:
default:
""").strip()
)
runner = CliRunner()
result = runner.invoke(main, ["--templates-dir", str(tmp_path)])
assert result.exit_code == 1
assert "db" in result.output
def test_no_templates_found(self, tmp_path: Path):
runner = CliRunner()
result = runner.invoke(main, ["--templates-dir", str(tmp_path)])
assert result.exit_code == 0
assert "OK" in result.output
+264
View File
@@ -0,0 +1,264 @@
"""Unit tests for devx.tools.check_jinja_expr.
Verifies that the check correctly validates Jinja2 expressions,
catches reversed strftime filter arguments (the OBL-INFRA-508 bug),
and passes on valid expressions.
"""
from __future__ import annotations
from pathlib import Path
from unittest.mock import patch
from click.testing import CliRunner
from devx.tools.check_jinja_expr import (
_check_file,
_default_ansible_dirs,
_extract_expressions,
_render_expression,
main,
)
def test_render_valid_expression():
"""Valid Jinja expression renders without error."""
ok, _ = _render_expression("'%Y-%m-%dT%H:%M:%S+00:00' | strftime(1735689600)")
assert ok
def test_render_reversed_strftime_args():
"""Reversed strftime filter args are detected as an error."""
ok, msg = _render_expression("(now().timestamp() | int + 3600) | strftime('%Y-%m-%dT%H:%M:%S+00:00')")
assert not ok
assert "reversed" in msg.lower()
def test_render_correct_strftime_args():
"""Correct strftime filter args pass."""
ok, _ = _render_expression("'%Y-%m-%dT%H:%M:%S+00:00' | strftime((now().timestamp() | int) + 3600)")
assert ok
def test_render_unknown_filter():
"""Unknown filter is reported as an error."""
ok, msg = _render_expression("'test' | nonexistent_filter")
assert not ok
assert "filter" in msg.lower()
def test_extract_skips_go_templates():
"""Go template syntax ({{.Field}}) is not extracted."""
content = "cmd: docker inspect --format '{{.State.Running}}' container"
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_single_char():
"""Single-character fragments are not extracted."""
content = 'value: "{{ \' }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_multiline():
"""Multi-line expressions are skipped."""
content = 'value: "{{\n something\n}}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_unbalanced():
"""Expressions with unbalanced braces (from partial capture) are skipped."""
content = "value: \"{{ default({'k': {}}, true) }}\""
expressions = _extract_expressions(content)
# The regex captures {{ default({'k': {}} — unbalanced parens
# because the inner }} terminates the match early.
# All extracted expressions should have balanced braces.
for expr in expressions:
assert expr.count("{") == expr.count("}")
def test_extract_valid_expression():
"""Valid Jinja expressions are extracted."""
content = "value: \"{{ my_var | default('x') }}\""
expressions = _extract_expressions(content)
assert "my_var | default('x')" in expressions
def test_main_passes_on_clean_file(tmp_path: Path) -> None:
"""A file with valid expressions passes."""
test_file = tmp_path / "tasks.yml"
test_file.write_text("value: \"{{ my_var | default('x') }}\"\nother: \"{{ '%Y' | strftime(1735689600) }}\"\n")
runner = CliRunner()
result = runner.invoke(main, ["--path", str(test_file)])
assert result.exit_code == 0
def test_main_no_violations_empty_dir(tmp_path: Path) -> None:
"""An empty directory passes."""
runner = CliRunner()
result = runner.invoke(main, ["--path", str(tmp_path)])
assert result.exit_code == 0
def test_main_catches_reversed_strftime(tmp_path: Path) -> None:
"""A file with reversed strftime args is flagged."""
test_file = tmp_path / "test.yml"
test_file.write_text("value: \"{{ (now().timestamp() | int + 3600) | strftime('%Y-%m-%dT%H:%M:%S+00:00') }}\"\n")
runner = CliRunner()
result = runner.invoke(main, ["--path", str(test_file)])
assert result.exit_code == 1
assert "reversed" in result.output.lower()
def test_render_skips_undefined_var():
"""Undefined variables are skipped (MockDict returns mock for missing keys)."""
ok, _ = _render_expression("nonexistent_var_in_mock | upper")
assert ok
def test_render_skips_other_errors():
"""Non-filter errors from missing mocks are skipped."""
ok, _ = _render_expression("some_undefined.attr.method()")
assert ok
def test_extract_skips_backtick():
"""Backtick fragments are skipped (caught by single-char check)."""
content = 'value: "{{ ` }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_network_settings():
"""Expressions with .NetworkSettings. patterns are skipped."""
content = 'value: "{{ foo.NetworkSettings.IPAddress }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_unbalanced_parens():
"""Expressions with unbalanced parens are skipped."""
content = 'value: "{{ foo(bar }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_unbalanced_braces():
"""Expressions with unbalanced braces are skipped."""
content = 'value: "{{ foo{bar }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_unbalanced_brackets():
"""Expressions with unbalanced brackets are skipped."""
content = 'value: "{{ foo[0 }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_control_flow():
"""Control flow fragments starting with % are skipped."""
content = 'value: "{{ % if x }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_check_file_outside_repo(tmp_path: Path) -> None:
"""Files outside REPO_ROOT are handled (no relative_to error)."""
test_file = tmp_path / "test.yml"
test_file.write_text("value: \"{{ (now().timestamp() | int + 3600) | strftime('%Y-%m-%dT%H:%M:%S+00:00') }}\"\n")
violations = _check_file(test_file, Path("/other/repo"))
assert len(violations) == 1
assert "reversed" in violations[0].lower()
def test_render_mock_dict_missing_key():
"""MockDict returns a mock for missing keys (no UndefinedError)."""
ok, _ = _render_expression("undefined_var.some_attr | upper")
assert ok
def test_render_syntax_error():
"""Syntax errors are reported as failures."""
ok, msg = _render_expression("{{ invalid syntax +")
assert not ok
assert "Syntax error" in msg
def test_render_unknown_filter_error():
"""Unknown filters are reported as failures (not skipped)."""
ok, msg = _render_expression("'test' | nonexistent_filter")
assert not ok
assert "filter" in msg.lower()
def test_render_generic_exception_skipped():
"""Non-filter exceptions from missing mocks are skipped."""
# replace() with no args triggers TypeError (missing required args)
# which is not a filter-not-found or strftime error — should be skipped.
ok, msg = _render_expression("my_var | replace")
assert ok
assert "Skipped" in msg
def test_default_ansible_dirs():
"""_default_ansible_dirs returns playbooks and roles paths."""
dirs = _default_ansible_dirs()
assert Path.cwd() / "ansible" / "playbooks" in dirs
assert Path.cwd() / "ansible" / "roles" in dirs
def test_main_default_dirs(tmp_path: Path) -> None:
"""Running with no --path scans default dirs (uses small temp fixture)."""
(tmp_path / "playbooks").mkdir()
(tmp_path / "roles").mkdir()
(tmp_path / "playbooks" / "test.yml").write_text("value: \"{{ my_var | default('x') }}\"\n")
with patch(
"devx.tools.check_jinja_expr._default_ansible_dirs",
return_value=[tmp_path / "playbooks", tmp_path / "roles"],
):
runner = CliRunner()
result = runner.invoke(main, [])
assert result.exit_code == 0
def test_main_custom_ansible_dirs(tmp_path: Path) -> None:
"""--ansible-dir option works."""
(tmp_path / "test.yml").write_text("value: \"{{ my_var | default('x') }}\"\n")
runner = CliRunner()
result = runner.invoke(main, ["--ansible-dir", str(tmp_path)])
assert result.exit_code == 0
def test_extract_skips_println():
"""Expressions with 'println' (Go template) are skipped."""
content = 'value: "{{ println something }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_extract_skips_state_dot():
"""Expressions with .State. patterns are skipped."""
content = 'value: "{{ foo.State.Running }}"'
expressions = _extract_expressions(content)
assert len(expressions) == 0
def test_render_now_with_format():
"""now() with a format argument works."""
ok, _ = _render_expression("now('%Y-%m-%d')")
assert ok
def test_find_yaml_files_skips_molecule(tmp_path: Path) -> None:
"""Molecule directories are excluded from file search."""
from devx.tools.check_jinja_expr import _find_yaml_files
(tmp_path / "tasks.yml").write_text("value: test\n")
(tmp_path / "molecule").mkdir()
(tmp_path / "molecule" / "test.yml").write_text("value: test\n")
files = _find_yaml_files(tmp_path)
assert all("molecule" not in f.parts for f in files)
+171
View File
@@ -0,0 +1,171 @@
"""Unit tests for devx.utils.api.APIClient."""
from __future__ import annotations
from unittest.mock import MagicMock, patch
import pytest
import requests
from devx.utils.api import APIClient
class TestAPIClient:
@patch("devx.utils.api.requests.request")
def test_get(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {"Authorization": "Bearer token"})
result = client.get("/users")
assert result is mock_resp
mock_req.assert_called_once_with(
"GET",
"https://api.example.com/users",
headers={"Authorization": "Bearer token"},
timeout=30,
verify=True,
)
@patch("devx.utils.api.requests.request")
def test_post(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {})
client.post("/users", json={"name": "alice"})
mock_req.assert_called_once_with(
"POST",
"https://api.example.com/users",
headers={},
timeout=30,
verify=True,
json={"name": "alice"},
)
@patch("devx.utils.api.requests.request")
def test_put(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {})
client.put("/users/1", json={"name": "bob"})
mock_req.assert_called_once_with(
"PUT",
"https://api.example.com/users/1",
headers={},
timeout=30,
verify=True,
json={"name": "bob"},
)
@patch("devx.utils.api.requests.request")
def test_delete(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {})
client.delete("/users/1")
mock_req.assert_called_once_with(
"DELETE",
"https://api.example.com/users/1",
headers={},
timeout=30,
verify=True,
)
@patch("devx.utils.api.requests.request")
def test_patch(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {})
client.patch("/users/1", json={"name": "carol"})
mock_req.assert_called_once_with(
"PATCH",
"https://api.example.com/users/1",
headers={},
timeout=30,
verify=True,
json={"name": "carol"},
)
@patch("devx.utils.api.requests.request")
def test_auth_tuple(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {}, auth=("admin", "pass"))
client.get("/data")
mock_req.assert_called_once_with(
"GET",
"https://api.example.com/data",
headers={},
timeout=30,
verify=True,
auth=("admin", "pass"),
)
@patch("devx.utils.api.requests.request")
def test_custom_timeout_and_verify(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {}, timeout=60, verify=False)
client.get("/data")
mock_req.assert_called_once_with(
"GET",
"https://api.example.com/data",
headers={},
timeout=60,
verify=False,
)
@patch("devx.utils.api.requests.request")
def test_raises_on_error(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.side_effect = requests.HTTPError("500")
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {})
with pytest.raises(requests.HTTPError):
client.get("/fail")
@patch("devx.utils.api.requests.request")
def test_strips_trailing_slash(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com/", {})
client.get("/users")
mock_req.assert_called_once_with(
"GET",
"https://api.example.com/users",
headers={},
timeout=30,
verify=True,
)
@patch("devx.utils.api.requests.request")
def test_kwargs_override_defaults(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {}, timeout=30)
client.get("/slow", timeout=120)
mock_req.assert_called_once_with(
"GET",
"https://api.example.com/slow",
headers={},
timeout=120,
verify=True,
)
@patch("devx.utils.api.requests.request")
def test_no_auth_when_not_set(self, mock_req):
mock_resp = MagicMock()
mock_resp.raise_for_status.return_value = None
mock_req.return_value = mock_resp
client = APIClient("https://api.example.com", {})
client.get("/data")
call_kwargs = mock_req.call_args.kwargs
assert "auth" not in call_kwargs
+161
View File
@@ -0,0 +1,161 @@
"""Unit tests for devx.utils.jinja."""
from __future__ import annotations
import jinja2
from devx.utils.jinja import (
make_env,
make_value_env,
regex_escape,
regex_replace,
regex_search,
render_manifest_values,
render_template,
render_value,
to_bool,
to_json,
)
class TestFilters:
def test_to_json(self):
assert to_json({"a": 1}) == '{"a": 1}'
def test_to_json_list(self):
assert to_json([1, 2]) == "[1, 2]"
def test_to_bool_true(self):
assert to_bool(True) is True
def test_to_bool_false(self):
assert to_bool(False) is False
def test_to_bool_string_true(self):
assert to_bool("yes") is True
def test_to_bool_string_false(self):
assert to_bool("false") is False
def test_to_bool_empty_string(self):
assert to_bool("") is False
def test_to_bool_none(self):
assert to_bool(None) is False
def test_to_bool_int(self):
assert to_bool(1) is True
assert to_bool(0) is False
def test_regex_replace(self):
assert regex_replace("hello world", "world", "there") == "hello there"
def test_regex_replace_with_pattern(self):
assert regex_replace("abc123", r"\d+", "X") == "abcX"
def test_regex_escape(self):
assert regex_escape("a.b*c") == "a\\.b\\*c"
def test_regex_search_found(self):
assert regex_search("hello world", r"world") == "world"
def test_regex_search_not_found(self):
assert regex_search("hello", r"world") is None
def test_regex_search_group(self):
assert regex_search("abc123", r"\d+") == "123"
class TestMakeEnv:
def test_returns_environment(self, tmp_path):
(tmp_path / "test.j2").write_text("hello {{ name }}")
env = make_env(str(tmp_path))
assert isinstance(env, jinja2.Environment)
def test_has_filters(self, tmp_path):
env = make_env(str(tmp_path))
assert "to_json" in env.filters
assert "bool" in env.filters
assert "regex_replace" in env.filters
assert "regex_escape" in env.filters
assert "regex_search" in env.filters
def test_cached(self, tmp_path):
env1 = make_env(str(tmp_path))
env2 = make_env(str(tmp_path))
assert env1 is env2
def test_auto_reload_disabled(self, tmp_path):
env = make_env(str(tmp_path))
assert env.auto_reload is False
def test_strict_undefined(self, tmp_path):
env = make_env(str(tmp_path))
assert env.undefined is jinja2.StrictUndefined
class TestMakeValueEnv:
def test_returns_environment(self):
env = make_value_env()
assert isinstance(env, jinja2.Environment)
def test_chainable_undefined(self):
env = make_value_env()
assert env.undefined is jinja2.ChainableUndefined
def test_cached(self):
assert make_value_env() is make_value_env()
def test_has_filters(self):
env = make_value_env()
assert "to_json" in env.filters
class TestRenderTemplate:
def test_renders_named_template(self, tmp_path):
(tmp_path / "test.j2").write_text("hello {{ name }}")
env = make_env(str(tmp_path))
assert render_template(env, "test.j2", name="world") == "hello world"
def test_renders_with_filters(self, tmp_path):
(tmp_path / "test.j2").write_text("{{ data | to_json }}")
env = make_env(str(tmp_path))
assert render_template(env, "test.j2", data={"a": 1}) == '{"a": 1}'
class TestRenderValue:
def test_renders_string_with_expressions(self):
assert render_value("hello {{ name }}", {"name": "world"}) == "hello world"
def test_passes_through_non_string(self):
assert render_value(42, {}) == 42
def test_passes_through_string_without_expressions(self):
assert render_value("plain text", {}) == "plain text"
def test_passes_through_none(self):
assert render_value(None, {}) is None
class TestRenderManifestValues:
def test_renders_dict_values(self):
result = render_manifest_values({"key": "{{ value }}"}, {"value": "rendered"})
assert result == {"key": "rendered"}
def test_renders_list_values(self):
result = render_manifest_values(["{{ a }}", "{{ b }}"], {"a": "1", "b": "2"})
assert result == ["1", "2"]
def test_renders_nested(self):
result = render_manifest_values({"outer": {"inner": "{{ x }}"}}, {"x": "yes"})
assert result == {"outer": {"inner": "yes"}}
def test_passes_through_non_string(self):
result = render_manifest_values({"n": 42, "b": True, "l": [1, 2]}, {})
assert result == {"n": 42, "b": True, "l": [1, 2]}
def test_empty_dict(self):
assert render_manifest_values({}, {}) == {}
def test_empty_list(self):
assert render_manifest_values([], {}) == []
+101
View File
@@ -0,0 +1,101 @@
"""Unit tests for devx.utils.ui."""
from __future__ import annotations
import logging
from unittest.mock import patch
import pytest
import devx.utils.ui as ui_mod
from devx.utils.ui import _console_level, configure_ui, say
class TestConsoleLevel:
def test_default_is_info(self) -> None:
with patch.dict("os.environ", {}, clear=True):
assert _console_level() == logging.INFO
def test_env_override(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("DEVX_LOG_LEVEL", "DEBUG")
assert _console_level() == logging.DEBUG
def test_invalid_fallback(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("DEVX_LOG_LEVEL", "VERBOSE")
assert _console_level() == logging.INFO
def test_custom_env_var(self, monkeypatch: pytest.MonkeyPatch) -> None:
configure_ui(log_level_env_var="GRM_LOG_LEVEL")
try:
monkeypatch.setenv("GRM_LOG_LEVEL", "DEBUG")
monkeypatch.delenv("DEVX_LOG_LEVEL", raising=False)
assert _console_level() == logging.DEBUG
finally:
configure_ui()
class TestSay:
def test_echoes_to_console(self) -> None:
with patch("devx.utils.ui.click.echo") as mock_echo:
say("hello")
mock_echo.assert_called_once_with("hello", err=False)
def test_logs_at_info_level(self) -> None:
with (
patch("devx.utils.ui.click.echo"),
patch("devx.utils.ui.logging.getLogger") as mock_get_logger,
):
mock_logger = mock_get_logger.return_value
say("hello")
mock_logger.log.assert_called_once_with(logging.INFO, "hello")
def test_passes_level_and_err(self) -> None:
with (
patch("devx.utils.ui.click.echo") as mock_echo,
patch("devx.utils.ui.logging.getLogger") as mock_get_logger,
):
mock_logger = mock_get_logger.return_value
say("error msg", level=logging.ERROR, err=True)
mock_echo.assert_called_once_with("error msg", err=True)
mock_logger.log.assert_called_once_with(logging.ERROR, "error msg")
def test_suppresses_console_below_level(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("DEVX_LOG_LEVEL", "WARNING")
with (
patch("devx.utils.ui.click.echo") as mock_echo,
patch("devx.utils.ui.logging.getLogger") as mock_get_logger,
):
mock_logger = mock_get_logger.return_value
say("debug msg", level=logging.DEBUG)
mock_echo.assert_not_called()
mock_logger.log.assert_called_once_with(logging.DEBUG, "debug msg")
def test_color_applied(self) -> None:
with (
patch("devx.utils.ui.click.echo") as mock_echo,
patch("devx.utils.ui.click.style") as mock_style,
):
mock_style.return_value = "styled-output"
say("success", color="green")
mock_style.assert_called_once_with("success", fg="green")
mock_echo.assert_called_once_with("styled-output", err=False)
def test_custom_logger_name(self) -> None:
configure_ui(logger_name="grm")
try:
with (
patch("devx.utils.ui.click.echo"),
patch("devx.utils.ui.logging.getLogger") as mock_get_logger,
):
say("hello")
mock_get_logger.assert_called_with("grm")
finally:
configure_ui()
class TestConfigureUi:
def test_reset_to_defaults(self) -> None:
configure_ui(log_level_env_var="GRM_LOG_LEVEL", logger_name="grm")
configure_ui()
assert ui_mod._LOG_LEVEL_ENV_VAR == "DEVX_LOG_LEVEL"
assert ui_mod._LOGGER_NAME == "devx"
+272
View File
@@ -0,0 +1,272 @@
"""Unit tests for devx.ci.wait_for_checks."""
from unittest.mock import MagicMock, patch
import pytest
from click.testing import CliRunner
from devx.ci.wait_for_checks import (
main,
poll_until_complete,
query_job_status,
)
def _mock_response(status_code: int = 200, json_data: object | None = None) -> MagicMock:
m = MagicMock()
m.status_code = status_code
if json_data is None:
m.json.side_effect = ValueError("no json")
else:
m.json.return_value = json_data
return m
class TestQueryJobStatus:
@patch("devx.ci.wait_for_checks.requests.get")
def test_returns_matching_jobs(self, mock_get: MagicMock) -> None:
"""Jobs whose name starts with the prefix are returned."""
mock_get.side_effect = [
_mock_response(200, [{"id": 1}, {"id": 2}]),
_mock_response(200, [{"name": "molecule-tests (1)", "status": "completed", "conclusion": "success"}]),
_mock_response(200, [{"name": "other-job", "status": "completed", "conclusion": "success"}]),
]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert len(result) == 1
assert result[0]["name"] == "molecule-tests (1)"
assert result[0]["status"] == "completed"
assert result[0]["conclusion"] == "success"
@patch("devx.ci.wait_for_checks.requests.get")
def test_no_matching_jobs(self, mock_get: MagicMock) -> None:
"""When no job names match the prefix, returns empty list."""
mock_get.side_effect = [
_mock_response(200, [{"id": 1}]),
_mock_response(200, [{"name": "other-job", "status": "completed", "conclusion": "success"}]),
]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert result == []
@patch("devx.ci.wait_for_checks.requests.get")
def test_api_error_returns_empty(self, mock_get: MagicMock) -> None:
"""Network errors on the runs endpoint return an empty list."""
import requests
mock_get.side_effect = requests.ConnectionError("down")
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert result == []
@patch("devx.ci.wait_for_checks.requests.get")
def test_non_200_returns_empty(self, mock_get: MagicMock) -> None:
"""Non-200 on runs endpoint returns empty list."""
mock_get.side_effect = [_mock_response(500, {"message": "err"})]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert result == []
@patch("devx.ci.wait_for_checks.requests.get")
def test_jobs_as_dict_with_jobs_key(self, mock_get: MagicMock) -> None:
"""Jobs endpoint returning {'jobs': [...]} dict is handled."""
mock_get.side_effect = [
_mock_response(200, [{"id": 1}]),
_mock_response(
200, {"jobs": [{"name": "molecule-tests (1)", "status": "in_progress", "conclusion": None}]}
),
]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert len(result) == 1
assert result[0]["status"] == "in_progress"
@patch("devx.ci.wait_for_checks.requests.get")
def test_runs_as_dict_with_runs_key(self, mock_get: MagicMock) -> None:
"""Runs endpoint returning {'runs': [...]} dict is handled."""
mock_get.side_effect = [
_mock_response(200, {"runs": [{"id": 1}]}),
_mock_response(200, [{"name": "molecule-tests (1)", "status": "completed", "conclusion": "success"}]),
]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert len(result) == 1
@patch("devx.ci.wait_for_checks.requests.get")
def test_jobs_endpoint_error_skips_run(self, mock_get: MagicMock) -> None:
"""A failed jobs query for one run doesn't abort the whole call."""
mock_get.side_effect = [
_mock_response(200, [{"id": 1}, {"id": 2}]),
_mock_response(500, {"message": "err"}),
_mock_response(200, [{"name": "molecule-tests (1)", "status": "completed", "conclusion": "success"}]),
]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert len(result) == 1
@patch("devx.ci.wait_for_checks.requests.get")
def test_run_without_id_skipped(self, mock_get: MagicMock) -> None:
"""Runs missing an 'id' field are skipped."""
mock_get.side_effect = [
_mock_response(200, [{"foo": "bar"}, {"id": 1}]),
_mock_response(200, [{"name": "molecule-tests (1)", "status": "completed", "conclusion": "success"}]),
]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert len(result) == 1
@patch("devx.ci.wait_for_checks.requests.get")
def test_jobs_query_exception_skips_run(self, mock_get: MagicMock) -> None:
"""A ConnectionError on the jobs endpoint for one run is skipped."""
import requests
mock_get.side_effect = [
_mock_response(200, [{"id": 1}, {"id": 2}]),
requests.ConnectionError("down"),
_mock_response(200, [{"name": "molecule-tests (1)", "status": "completed", "conclusion": "success"}]),
]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert len(result) == 1
@patch("devx.ci.wait_for_checks.requests.get")
def test_jobs_json_value_error_skips_run(self, mock_get: MagicMock) -> None:
"""A ValueError (bad JSON) on the jobs endpoint is skipped."""
mock_get.side_effect = [
_mock_response(200, [{"id": 1}]),
_mock_response(200), # json raises ValueError by default
]
result = query_job_status("https://api", "tok", "o/r", "molecule-tests")
assert result == []
class TestPollUntilComplete:
@patch("devx.ci.wait_for_checks.time.sleep")
@patch("devx.ci.wait_for_checks.time.monotonic")
@patch("devx.ci.wait_for_checks.query_job_status")
def test_all_jobs_succeed(self, mock_query: MagicMock, mock_mono: MagicMock, mock_sleep: MagicMock) -> None:
"""All jobs completed with success → returns 0."""
mock_query.return_value = [{"name": "molecule-tests (1)", "status": "completed", "conclusion": "success"}]
mock_mono.side_effect = [0.0, 0.0]
code = poll_until_complete("https://api", "tok", "o/r", "molecule-tests", timeout=100, interval=10)
assert code == 0
@patch("devx.ci.wait_for_checks.time.sleep")
@patch("devx.ci.wait_for_checks.time.monotonic")
@patch("devx.ci.wait_for_checks.query_job_status")
def test_job_fails(self, mock_query: MagicMock, mock_mono: MagicMock, mock_sleep: MagicMock) -> None:
"""A job with non-success conclusion → returns 1."""
mock_query.return_value = [{"name": "molecule-tests (1)", "status": "completed", "conclusion": "failure"}]
mock_mono.side_effect = [0.0, 0.0]
code = poll_until_complete("https://api", "tok", "o/r", "molecule-tests", timeout=100, interval=10)
assert code == 1
@patch("devx.ci.wait_for_checks.time.sleep")
@patch("devx.ci.wait_for_checks.time.monotonic")
@patch("devx.ci.wait_for_checks.query_job_status")
def test_no_require_success(self, mock_query: MagicMock, mock_mono: MagicMock, mock_sleep: MagicMock) -> None:
"""With require_success=False, a failed job returns 0."""
mock_query.return_value = [{"name": "molecule-tests (1)", "status": "completed", "conclusion": "failure"}]
mock_mono.side_effect = [0.0, 0.0]
code = poll_until_complete(
"https://api", "tok", "o/r", "molecule-tests", timeout=100, interval=10, require_success=False
)
assert code == 0
@patch("devx.ci.wait_for_checks.time.sleep")
@patch("devx.ci.wait_for_checks.time.monotonic")
@patch("devx.ci.wait_for_checks.query_job_status")
def test_timeout(self, mock_query: MagicMock, mock_mono: MagicMock, mock_sleep: MagicMock) -> None:
"""Jobs never complete → returns 2 after timeout."""
mock_query.return_value = [{"name": "molecule-tests (1)", "status": "in_progress", "conclusion": None}]
# monotonic calls: deadline=0, while-check=0 (enter), sleep-calc=0, while-check=200 (exit)
mock_mono.side_effect = [0.0, 0.0, 0.0, 200.0]
code = poll_until_complete("https://api", "tok", "o/r", "molecule-tests", timeout=100, interval=10)
assert code == 2
@patch("devx.ci.wait_for_checks.time.sleep")
@patch("devx.ci.wait_for_checks.time.monotonic")
@patch("devx.ci.wait_for_checks.query_job_status")
def test_no_jobs_found_timeout(self, mock_query: MagicMock, mock_mono: MagicMock, mock_sleep: MagicMock) -> None:
"""No matching jobs at all → returns 3."""
mock_query.return_value = []
# monotonic calls: deadline=0, while-check=0 (enter), sleep-calc=0, while-check=200 (exit)
mock_mono.side_effect = [0.0, 0.0, 0.0, 200.0]
code = poll_until_complete("https://api", "tok", "o/r", "molecule-tests", timeout=100, interval=10)
assert code == 3
@patch("devx.ci.wait_for_checks.time.sleep")
@patch("devx.ci.wait_for_checks.time.monotonic")
@patch("devx.ci.wait_for_checks.query_job_status")
def test_in_progress_then_success(self, mock_query: MagicMock, mock_mono: MagicMock, mock_sleep: MagicMock) -> None:
"""First poll in_progress, second poll success → returns 0."""
mock_query.side_effect = [
[{"name": "molecule-tests (1)", "status": "in_progress", "conclusion": None}],
[{"name": "molecule-tests (1)", "status": "completed", "conclusion": "success"}],
]
# monotonic: deadline=0, while=0 (enter), sleep-calc=0, while=5 (enter), success→return
mock_mono.side_effect = [0.0, 0.0, 0.0, 5.0]
code = poll_until_complete("https://api", "tok", "o/r", "molecule-tests", timeout=100, interval=10)
assert code == 0
class TestMain:
@patch("devx.ci.wait_for_checks.get_ci_token", return_value="tok")
@patch("devx.ci.wait_for_checks.poll_until_complete", return_value=0)
def test_success_exit_code(self, mock_poll: MagicMock, mock_token: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests", "--repo", "o/r"])
assert result.exit_code == 0
@patch("devx.ci.wait_for_checks.get_ci_token", return_value="tok")
@patch("devx.ci.wait_for_checks.poll_until_complete", return_value=1)
def test_failure_exit_code(self, mock_poll: MagicMock, mock_token: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests", "--repo", "o/r"])
assert result.exit_code == 1
@patch("devx.ci.wait_for_checks.get_ci_token", return_value="tok")
@patch("devx.ci.wait_for_checks.poll_until_complete", return_value=2)
def test_timeout_exit_code(self, mock_poll: MagicMock, mock_token: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests", "--repo", "o/r"])
assert result.exit_code == 2
@patch("devx.ci.wait_for_checks.get_ci_token", return_value="tok")
@patch("devx.ci.wait_for_checks.poll_until_complete", return_value=3)
def test_api_error_exit_code(self, mock_poll: MagicMock, mock_token: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests", "--repo", "o/r"])
assert result.exit_code == 3
def test_missing_job_name(self) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--repo", "o/r"])
assert result.exit_code != 0
def test_missing_repo(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("GITHUB_REPOSITORY", raising=False)
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests"])
assert result.exit_code != 0
@patch("devx.ci.wait_for_checks.get_ci_token", return_value="tok")
@patch("devx.ci.wait_for_checks.poll_until_complete", return_value=0)
def test_repo_from_env(self, mock_poll: MagicMock, mock_token: MagicMock, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("GITHUB_REPOSITORY", "oblachno-oss/grm")
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests"])
assert result.exit_code == 0
args, kwargs = mock_poll.call_args
assert args[2] == "oblachno-oss/grm"
@patch("devx.ci.wait_for_checks.get_ci_token", return_value="tok")
@patch("devx.ci.wait_for_checks.poll_until_complete", return_value=0)
def test_invalid_timeout(self, mock_poll: MagicMock, mock_token: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests", "--repo", "o/r", "--timeout", "0"])
assert result.exit_code != 0
@patch("devx.ci.wait_for_checks.get_ci_token", return_value="tok")
@patch("devx.ci.wait_for_checks.poll_until_complete", return_value=0)
def test_invalid_interval(self, mock_poll: MagicMock, mock_token: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests", "--repo", "o/r", "--poll-interval", "0"])
assert result.exit_code != 0
@patch("devx.ci.wait_for_checks.get_ci_token", side_effect=__import__("click").ClickException("no token"))
def test_no_token_exit_3(self, mock_token: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(main, ["--job-name", "molecule-tests", "--repo", "o/r"])
assert result.exit_code == 3