oblachno-oss/devx

Public Access

Fork 0

Table of Contents

Architecture

Package structure
Core modules

__init__.py
cli.py
config.py
exceptions.py
i18n.py
api_clients.py
gitea_cli.py
opentofu.py

CI/CD modules (devx.ci)

release.py
publish.py
auto_merge.py
classify_changes.py
pr_review.py
sync_wiki.py
push_badges.py
notify_failure.py
post_merge.py
validate_commit_msg.py
detect_release_commit.py
check_translations.py
doc_coverage.py
discover_runners.py
distribute_files.py
integration_guard.py

Developer tools (devx.tools)

setup.py
install_tools.py
check_test_speed.py
check_test_isolation.py
configure_repo.py
generate_badges.py
generate_cliff_config.py
install_checkmake.py

Molecule modules (devx.molecule)

distribute_molecule.py
molecule_all.py
molecule/discover_runners.py
start_docker.py
platforms.py

Design principles
Import rules
Data flow

PR lifecycle
Post-merge flow
Publish flow
Badge generation flow

tea CLI integration
Version source

Architecture

devx is a reusable Python package providing development and CI/CD tools for oblachno-oss projects. It is self-contained under src/devx/ and never imports from scripts outside the package.

Package structure

src/devx/
├── __init__.py              # Version (single source of truth, read by setuptools)
├── cli.py                   # Click-based CLI entry point (devx command)
├── config.py                # Configuration system (DEVX_ env vars, .env loading)
├── api_clients.py           # GiteaClient, VikunjaClient — HTTP API wrappers
├── gitea_cli.py             # TeaCLI — wrapper around tea CLI with JSON parsing
├── i18n.py                  # Translation system (JSON-based, translations.json)
├── exceptions.py            # Custom exception types (DevxError, APIError)
├── opentofu.py              # OpenTofu output helpers
├── translations.json        # Translation strings (en, bg, de, ru, zh, pl)
├── ci/                      # CI/CD automation modules (run by workflows)
│   ├── __init__.py
│   ├── _shared.py           # Shared utilities (get_latest_tag)
│   ├── release.py           # Automated versioning, tagging, changelog
│   ├── publish.py           # Build and publish to Gitea PyPI registry
│   ├── auto_merge.py        # Squash-merge PRs with task ID validation
│   ├── classify_changes.py  # User-facing vs workflow-only change detection
│   ├── detect_release_commit.py  # Detect release commits on master
│   ├── validate_commit_msg.py    # Conventional commit validation
│   ├── pr_review.py         # Automated PR review
│   ├── post_merge.py        # Vikunja task updates after merge
│   ├── sync_wiki.py         # Sync documentation to Gitea wiki
│   ├── push_badges.py       # Generate and push quality badges
│   ├── notify_failure.py    # Create Gitea issues on CI failures
│   ├── distribute_files.py  # Distribute files across parallel runners
│   ├── integration_guard.py # Run pytest with cross-runner fail-fast
│   ├── discover_runners.py  # Dynamic Gitea runner discovery
│   ├── check_translations.py  # Translation completeness check
│   └── doc_coverage.py      # Documentation coverage check
├── tools/                   # Developer tooling modules (run locally or by CI)
│   ├── __init__.py
│   ├── setup.py             # Environment setup (venv, deps, hooks, tea login)
│   ├── install_tools.py     # Install actionlint, git-cliff, act_runner, tea
│   ├── check_test_speed.py  # Measure unit test execution time
│   ├── check_test_isolation.py  # Pytest plugin: detect un-hermetic test patterns
│   ├── configure_repo.py    # Branch protection and label setup
│   ├── generate_badges.py   # Badge SVG generation
│   ├── generate_cliff_config.py  # Generate cliff.toml with correct prefix
│   └── install_checkmake.py # Install checkmake (Makefile linter)
└── molecule/                # Optional molecule testing helpers (Ansible projects)
    ├── __init__.py
    ├── discover_runners.py      # Dynamic Gitea runner discovery
    ├── distribute_molecule.py   # Distribute scenarios across runners
    ├── molecule_all.py          # Run all molecule scenarios locally
    ├── start_docker.py          # Ensure Docker is available for molecule
    └── platforms.py             # Supported molecule platforms

Core modules

`init.py`

Contains only __version__, the single source of truth for the package version. Read by setuptools via dynamic = ["version"] in pyproject.toml. Updated automatically by devx.ci.release during the release process. Treated as infrastructure (not user-facing) by the change classifier since it is a release artifact, not user code.

`cli.py`

Click-based CLI entry point. Provides three command groups: devx ci, devx tools, and devx molecule. Each subcommand delegates to the corresponding module via _run_module(), which imports the module, sets sys.argv, and calls its main() function. This design keeps all logic in the modules themselves — cli.py is purely a router.

The CLI is registered as a console script via pyproject.toml:

[project.scripts]
devx = "devx.cli:cli"

`config.py`

Shared configuration constants for all devx modules. All defaults can be overridden via environment variables with the DEVX_ prefix. Provides:

GITEA_API_URL / VIKUNJA_API_URL — API endpoints
REPO_OWNER — repository owner (must be set per-project)
TASK_PREFIX / TASK_ID_RE — task ID prefix and regular expression (for example, DEVX-N)
VIKUNJA_PROJECT_ID — Vikunja project for task tracking
DEFAULT_TIMEOUT, DEFAULT_PER_PAGE — HTTP client defaults
MAX_RETRIES, RETRY_BACKOFF_BASE, RETRY_STATUS_CODES — retry config
CONVENTIONAL_RE — conventional commit format regular expression

`exceptions.py`

Custom exception hierarchy:

DevxError — base exception for all devx errors
APIError(DevxError) — raised when a REST API call returns an HTTP error. Carries status (HTTP status code) and message (error message).

`i18n.py`

Simple i18n system using a JSON translations file (translations.json). Supports six languages: en, bg, de, pl, ru, zh. The _() function wraps user-facing strings for translation.

Projects can extend translations by setting DEVX_TRANSLATIONS_PATH to a custom JSON file. Keys from the project's file are merged on top of devx's built-in translations, allowing projects to override, or add keys without modifying the package.

`api_clients.py`

Reusable HTTP API clients with connection pooling and retry logic. Both clients retry on transient errors (429, 5xx, connection errors) with exponential backoff (2s, 4s, 8s).

GiteaClient — Gitea REST API wrapper:

Branch protection (get, create, update)
Labels (list, create, add to issues)
Issues (create, list)
Pull requests (get commits, merge, create review)
Releases (list, create idempotent)
Actions (list runs, list jobs, get job logs)
Actions variables (get, set idempotent)
Wiki pages (list, fetch, create, update, delete)

VikunjaClient — Vikunja REST API wrapper:

Tasks (list project tasks, get, update, mark done)
Comments (create)

`gitea_cli.py`

Thin Python wrapper around the tea Gitea CLI tool. Parses JSON output for structured data. Used by CI scripts for Gitea API operations that tea handles well, avoiding hand-rolled HTTP requests.

TeaCLI operations:

create_issue() — Create issues with labels
list_labels() / create_label() / add_label() — Label management
create_pr() / merge_pr() / review_pr() — Pull request operations
create_release() / list_releases() — Release management
list_branches() — Branch listing

Operations NOT supported via tea (still use GiteaClient):

Wiki page management
Commit status checks
Runner discovery
PR file/commit listing (tea has limited support)
Branch protection with detailed config

`opentofu.py`

OpenTofu output helpers for CI/CD deployment scripts. Provides reusable functions for extracting values from tofu output in a structured way, eliminating duplicated subprocess.run boilerplate:

get_tofu_output(output_name, cwd, env) — Run tofu output -json and return parsed JSON
get_tofu_vm_ip(output_name, vm_name, cwd, env) — Extract a VM's IP address
get_tofu_vm_field(output_name, vm_name, field, cwd, env) — Extract a VM field

CI/CD modules (`devx.ci`)

Modules in this package are run by Gitea Actions workflows. They may import from devx.api_clients, devx.config, devx.gitea_cli, and devx.i18n.

`release.py`

Automated release using git-cliff. Calculates the next semver version from conventional commits since the last tag, updates __version__ in __init__.py and CHANGELOG.md, runs lint, and tests to verify the release is healthy, commits with release: vX.Y.Z [skip ci], creates an annotated tag, and pushes both to master.

Idempotent: if there are no new conventional commits since the last tag, it exits without doing anything. If the tag already exists, it skips tag creation and only pushes. Includes a --verify mode that checks tag/version/changelog alignment without making changes.

`publish.py`

Builds the Python package with python -m build, publishes to a Gitea PyPI registry (or standard PyPI if PYPI_TOKEN is set), and creates a Gitea release with git-cliff-generated notes. Supports --skip-build for non-Python repos that only need a Gitea release.

`auto_merge.py`

Auto-merges a PR when all CI checks pass. Reads the task ID from the branch name, validates the PR title format against the Vikunja task title, extracts the conventional commit message from PR commits, and squash-merges with title {PREFIX}-N <conventional commit>.

If the head branch is behind master (HTTP 405), it automatically pulls master, rebases, force-pushes, and retries the merge.

`classify_changes.py`

Classifies git changes between two refs as user-facing or workflow-only. Uses a layered rule system configured in pyproject.toml under [tool.devx.classify]:

User-facing overrides (highest priority — safety override)
Infrastructure overrides (explicit per-file)
Infrastructure patterns (DEFAULT_INFRASTRUCTURE + project-specific)
Default: user-facing (safe default — any unknown file triggers release)

Also supports custom tags (orthogonal to release impact) for CI conditional execution (for example, ansible tag to trigger molecule tests).

`pr_review.py`

Automated PR review. Fetches the PR diff via the Gitea API and runs a series of checks, posting a structured review with COMMENT (no issues) or REQUEST_CHANGES (issues found):

Architecture compliance (no subprocess in CLI, no hardcoded URLs)
Best practices (no print(), no bare except, no TODO/FIXME, no functions > 50 lines)
Security (no hardcoded secrets, no shell=True, no eval/exec)
i18n (no raw strings in click.echo() without _() wrapper)
Resource management (no open() without with, no Popen() without cleanup)
Documentation (source changes must include doc updates)
Test coverage (source changes must include test updates)
Commit conventions (conventional commit format on PR commits)

`sync_wiki.py`

Syncs documentation from docs/ to the Gitea wiki via the API. Reads docs/mapping.json to map file paths to wiki page titles, then creates or updates pages. Supports --dry-run, --verify (check content), and --strict (full integrity check: page count, missing pages, stale pages, content match).

`push_badges.py`

Generates SVG badge files using devx.tools.generate_badges, pushes them to an orphan badges branch, and updates README.md and docs/index.md on master with cache-busting raw/commit/<sha>/badge.svg URLs (Gitea caches raw/branch/ URLs for 6 hours). Fetches latest master before generating badges so the version badge reflects the current state. Supports --retries for retrying on git push failures.

`notify_failure.py`

Creates a Gitea issue when a CI workflow fails. Uses the tea CLI for issue creation with failure labels. Supports --auto-login to configure the tea CLI login profile from CI_GITEA_TOKEN and DEVX_GITEA_API_URL before creating the issue.

`post_merge.py`

Updates the Vikunja task after a merge to master. Extracts the task ID from the commit message, marks the task as done, and posts a comment with the merge SHA.

`validate_commit_msg.py`

Validates commit messages. On feature branches: conventional commits only (no {PREFIX}-N prefix). On master: must have {PREFIX}-N prefix from auto-merge, followed by a conventional commit message.

`detect_release_commit.py`

Detects whether the latest git commit is a release commit (release: vX.Y.Z [skip ci]). Writes is-release=true or is-release=false to $GITHUB_OUTPUT for use in CI workflow conditionals.

`check_translations.py`

Validates translation files against the Python source code. Checks for missing keys (used in code but not in translations), dead keys (defined but not used), and missing languages (a key exists but is missing one of the five supported languages). Supports checking additional translation sets via --translations.

`doc_coverage.py`

Checks documentation coverage for CLI commands and major modules. Parses Click commands from cli.py and verifies each has documentation in docs/user/cli-commands.md. Checks that core modules are documented in architecture.md and CI scripts in ci-cd-workflow.md. Supports --fail-on-missing to enforce 100% coverage.

`discover_runners.py`

Discovers available Gitea Actions runners at three levels: repository, organization, and instance (administrator). Falls back to the MOLECULE_RUNNERS repo variable or DEFAULT_MAX_RUNNERS (3). Outputs runner count or a JSON index array for use as a dynamic matrix in Gitea Actions.

`distribute_files.py`

Distributes files matching a glob pattern across N parallel runners (round-robin). Writes the assigned file list for the current runner to $GITHUB_ENV. Used for splitting test suites across CI runners.

`integration_guard.py`

Runs pytest with cross-runner failure detection. A background thread polls the Gitea API. If any other integration-tests matrix runner reports failure, the current pytest subprocess is killed and this runner exits early.

Developer tools (`devx.tools`)

Modules in this package are run locally or by CI setup jobs. They may import from devx.api_clients, devx.config, and devx.gitea_cli.

`setup.py`

Project setup: installs Python dependencies (editable mode with extras), Ansible Galaxy collections (if ansible/requirements.yml exists in the target repo), pre-commit hooks (pre-commit, commit-msg, pre-push), and configures the tea CLI login profile from .env. Supports --extras to specify dependency groups, --no-pre-commit to skip hook installation, and --no-tea-login to skip tea configuration.

`install_tools.py`

Installs CI/CD development tools that are not Python packages: actionlint, git-cliff, act_runner, and tea. Each tool is installed to ~/.local/bin if not already on PATH. Idempotent: skips tools that are already available. Supports --tool to install specific tools and --list to show status.

`check_test_speed.py`

Runs unit tests and enforces execution-time budgets. Two quality gates: total suite time must not exceed --max-seconds (default: 10s), and no individual test may exceed --max-single-seconds (default: 0.5s, 0 to off). Runs make test-unit with PYTEST_ADDOPTS=--durations=0.

`check_test_isolation.py`

Pytest plugin (auto-discovered via pytest11 entry point) that statically analyzes test files for un-hermetic patterns causing slow or flaky tests: unpatched subprocess.run/time.sleep calls, known subprocess-spawning helpers called without @patch, and excessive loop iterations (>100). Also available as a standalone CLI for CI gates and pre-commit hooks. See ADR-0001 for design rationale.

`configure_repo.py`

Configures repository branch protection and labels via the Gitea REST API. Sets up master branch protection (required status checks, block on rejected reviews, block on outdated branch) and creates standard labels. Status check contexts are read from DEVX_STATUS_CHECKS or default to CI / validate (pull_request).

`generate_badges.py`

Generates self-contained SVG badge files from project metrics. Runs pytest-cov, doc-coverage, lint checks, and version extraction, then writes SVG files that can be served as static files from the Gitea raw file API. Badges generated: coverage, tests, docs, quality, version, python.

`generate_cliff_config.py`

Generates a cliff.toml configuration file with the correct task ID prefix preprocessor. Eliminates the need to manually duplicate and maintain cliff.toml across repos that use devx. Supports --prefix to set the task ID prefix and --force to overwrite an existing file.

`install_checkmake.py`

Installs checkmake (Makefile linter) if not already present. Tries go install first if Go is available, otherwise downloads the latest pre-built Linux binary from the official GitHub releases.

Molecule modules (`devx.molecule`)

Optional modules for projects with Ansible roles. Requires the molecule extra (pip install devx[molecule]).

`distribute_molecule.py`

Distributes molecule (scenario, platform) pairs across N parallel runners. Discovers scenarios under ansible/roles/*/molecule/ and crosses them with the supported OS platform matrix. Supports --roles-root for multi-role repositories, --list to list scenarios, and --list-platforms to list platforms.

`molecule_all.py`

Runs all molecule scenarios on all supported OS platforms sequentially. Intended for local development; CI uses the parallel matrix instead.

`molecule/discover_runners.py`

Discovers available Gitea Actions runners for molecule tests. Same logic as devx.ci.discover_runners but intended for molecule-specific workflows.

`start_docker.py`

Ensures Docker is available for molecule tests in CI. Verifies Docker is accessible and sets DOCKER_HOST explicitly. If the host socket is not available, tries the rootless socket, then starts a local dockerd with the vfs storage driver (requires privileged container).

`platforms.py`

Single source of truth for the supported OS platform matrix. Each entry maps a short name to (image, command). Uses the project's pre-built molecule-test-base image with sleep infinity (not systemd) to avoid cgroup v2 failures. Supports loading custom platforms from a JSON file.

Design principles

Self-contained package — src/devx/ never imports from scripts outside the package. This allows devx to be installed and used as a dependency without requiring a specific repo layout in the consumer.
Module-based invocation — All tools invoked via python -m devx.ci.*, python -m devx.tools.*, or python -m devx.molecule.*. The devx CLI is a thin router that delegates to module main() functions.
PYTHONPATH: src — Workflows set PYTHONPATH: src (not .:src since there are no scripts at repo root). The src directory is the sole import root.
Config via env vars — DEVX_* environment variables with .env file fallback. Projects override defaults via environment or .env, never by editing package code.
100% test coverage — enforced by --cov-fail-under=100 in pytest.
i18n by default — all user-facing strings wrapped in _() for translation. Five languages supported out of the box.
Safe-by-default classification — any file that doesn't match an infrastructure pattern defaults to user-facing, triggering a release. This prevents new file types from accidentally skipping releases.
Secrets via environment — secrets are passed via environment variables, never on the command line.

Import rules

src/devx/ is self-contained — the package never imports from outside src/
CI modules (devx.ci.*) may import from devx.api_clients, devx.config, devx.gitea_cli, devx.i18n
Tool modules (devx.tools.*) may import from devx.api_clients, devx.config, devx.gitea_cli
Cross-module imports within devx.ci.* or devx.tools.* are allowed but must be documented (for example, release.py imports from classify_changes.py)

Data flow

PR lifecycle

Developer creates Vikunja task (DEVX-N)
    │
    ▼
Developer creates branch (DEVX-N-short-description)
    │
    ▼
Developer commits (conventional commits, no DEVX-N prefix)
    │
    ▼
Developer pushes and creates PR (title: "DEVX-N: <vikunja task title>")
    │
    ▼
CI workflow (ci.yml) triggers:
    │
    ├── validate (single job: quality + detect-changes +
    │            release-dry-run + pr-review + pre-merge validation)
    │       ├── quality steps (lint, tests, coverage, test speed, doc coverage,
    │       │                 translation check, dependency scan, workflow dry-run)
    │       ├── detect-changes (classify_changes.py → user-facing or workflow-only)
    │       │       └── if user-facing → release-dry-run (release.py --dry-run)
    │       ├── pre-merge validation (check_auto_merge_ready.py)
    │       └── pr-review (pr_review.py → posts COMMENT or REQUEST_CHANGES)
    │
    └── auto-merge (auto_merge.py)
            ├── validate PR title format
            ├── validate PR title matches Vikunja task title
            ├── extract conventional commit message from PR commits
            ├── squash-merge with "DEVX-N <conventional commit>" title
            └── push to master
                    │
                    ▼
            Post-merge workflow triggers (see below)

Post-merge flow

Push to master (squash-merge commit: "DEVX-N <conventional commit>")
    │
    ▼
Post-merge workflow (post-merge.yml) triggers:
    │
    ├── detect-and-configure (single job)
    │       ├── configure-repo (configure_repo.py)
    │       ├── detect-type (detect_release_commit.py)
    │       │       └── is-release? → skip all steps except badges
    │       └── validate-commit-msg (validate_commit_msg.py --branch master)
    │
    └── release-and-maintain (needs detect-and-configure)
            ├── release (release.py) [skip if release commit or workflow-only]
            │       ├── classify_changes.py → skip if workflow-only
            │       ├── git-cliff → calculate next version
            │       ├── update __version__ in __init__.py
            │       ├── update CHANGELOG.md
            │       ├── run make lint-ruff && make pytest-cov
            │       ├── commit "release: vX.Y.Z [skip ci]"
            │       ├── create annotated tag vX.Y.Z
            │       └── push commit + tag to master
            │               │
            │               ▼
            │       publish (publish.py) [if release created a tag]
            │               ├── build package (python -m build)
            │               ├── publish to Gitea PyPI registry (twine upload)
            │               │   OR publish to standard PyPI (if PYPI_TOKEN set)
            │               │   OR skip publish (if --skip-build)
            │               └── create Gitea release with git-cliff notes
            │
            ├── sync-wiki (sync_wiki.py --strict) [skip if automated]
            │       └── sync docs/ to Gitea wiki with integrity check
            │
            ├── vikunja (post_merge.py) [skip if automated]
            │       ├── extract task ID from commit message
            │       ├── mark Vikunja task as done
            │       └── post comment with merge SHA
            │
            └── badges (push_badges.py) [ALWAYS runs, even on release commits]
                    ├── fetch latest master
                    ├── generate_badges.py → SVG files
                    ├── push to orphan badges branch
                    └── update README.md + docs/index.md with cache-busting URLs

Publish flow

Within release-and-maintain job (after release step creates a tag):
    │
    ├── install build, twine, git-cliff, tea
    ├── configure tea login
    ├── checkout release tag
    │
    └── publish (publish.py)
            ├── build package (python -m build)
            ├── publish to Gitea PyPI registry (twine upload)
            │   OR publish to standard PyPI (if PYPI_TOKEN set)
            │   OR skip publish (if --skip-build)
            └── create Gitea release with git-cliff notes

Badge generation flow

push_badges.py:
    │
    ├── fetch_latest_master() → git fetch + reset --hard origin/master
    │
    ├── generate_badges() → devx.tools.generate_badges
    │       ├── run pytest-cov → parse coverage %
    │       ├── run pytest → parse test count
    │       ├── run doc_coverage → parse doc coverage %
    │       ├── run lint → quality status
    │       ├── read __version__ from __init__.py
    │       └── write SVG files to .badges/
    │
    ├── push_to_badges_branch()
    │       ├── git checkout --orphan badges
    │       ├── git rm -rf .
    │       ├── copy SVG files to root
    │       ├── git commit "Update badges [skip ci]"
    │       ├── git push origin badges --force
    │       └── return commit SHA
    │
    └── update_readme_with_badge_sha()
            ├── git checkout master
            ├── replace raw/branch/badges/ URLs with raw/commit/<sha>/ URLs
            ├── git commit "chore: update badge URLs [skip ci]"
            └── git push origin master

tea CLI integration

The tea Gitea CLI tool is used for Gitea API interactions where tea provides reliable, official support. It is installed by python -m devx.tools.install_tools and configured by python -m devx.tools.setup (login profile from .env CI_GITEA_TOKEN).

devx.gitea_cli.TeaCLI wraps tea with JSON output parsing. Operations that tea does not support (wiki management, commit status, runner discovery, detailed branch protection) fall back to GiteaClient (direct HTTP).

Version source

The version source is __version__ in src/devx/__init__.py, read by setuptools via dynamic = ["version"] in pyproject.toml. The release script updates this file, commits it, and tags the commit. This ensures the package version, git tag, and changelog always stay aligned.

Architecture

Package structure

Core modules

__init__.py

cli.py

config.py

exceptions.py

i18n.py

api_clients.py

gitea_cli.py

opentofu.py

CI/CD modules (devx.ci)

release.py

publish.py

auto_merge.py

classify_changes.py

pr_review.py

sync_wiki.py

push_badges.py

notify_failure.py

post_merge.py

validate_commit_msg.py

detect_release_commit.py

check_translations.py

doc_coverage.py

discover_runners.py

distribute_files.py

integration_guard.py

Developer tools (devx.tools)

setup.py

install_tools.py

check_test_speed.py

check_test_isolation.py

configure_repo.py

generate_badges.py

generate_cliff_config.py

install_checkmake.py

Molecule modules (devx.molecule)

distribute_molecule.py

molecule_all.py

molecule/discover_runners.py

start_docker.py

platforms.py