devx/AGENTS.md

AGENTS.md — Project Conventions for devx

Build & Test Commands

make setup           # Create venv, install deps, set up hooks, install CI tools
make install-tools   # Install actionlint, git-cliff, act_runner, tea, hadolint to ~/.local/bin
make lint-all        # ruff + pyright + bandit + actionlint + lint-dockerfiles
make pytest-cov      # Unit tests with 100% coverage enforcement
make test-unit       # Unit tests without coverage
make workflow-lint   # Static lint of .gitea/workflows/*.yml (actionlint)
make workflow-dryrun # Dry-run all workflows in Docker (act_runner exec --dryrun)
make workflow-check  # workflow-lint + workflow-dryrun
make clean           # Remove caches, build artifacts, coverage data

make setup automatically installs all development tools:

• Python deps via python -m devx.tools.setup (pip install -e .[dev], pre-commit hooks)
• actionlint, git-cliff, act_runner, tea, hadolint via python -m devx.tools.install_tools (CI/CD tools to ~/.local/bin)
• tea CLI login via python -m devx.tools.setup (configures tea login from .env CI_GITEA_TOKEN)

Workflow Verification (Before Push)

Workflow YAML files (.gitea/workflows/*.yml) are verified with two tools:

1. actionlint — Static linter that catches syntax errors, invalid expressions, unknown keys, type mismatches, and shellcheck issues. Config: .gitea/actionlint.yaml (registers custom docker runner label). Installed automatically by make setup via python -m devx.tools.install_tools.

2. act_runner exec --dryrun — Gitea's own runner in dry-run mode. Validates job dependencies, step ordering, and Docker image selection without starting containers. Installed automatically by make setup.

Both run via make workflow-check and are part of make lint-all. The pre-commit hook runs actionlint automatically when workflow files change. The CI quality job runs make setup-quality then make lint-all. CI also runs a best-effort make workflow-dryrun step (skipped if act_runner is not installed in the CI Docker image).

Architecture

devx is a reusable Python package providing development and CI/CD tools for oblachno-oss projects.

Package Structure

src/devx/
├── __init__.py          # Version (single source of truth, read by setuptools)
├── cli.py               # Click-based CLI entry point (devx command)
├── config.py            # Configuration system (DEVX_ env vars, .env loading)
├── api_clients.py       # GiteaClient, VikunjaClient — HTTP API wrappers
├── gitea_cli.py         # TeaCLI — wrapper around tea CLI with JSON parsing
├── i18n.py              # Translation system (gettext-based, translations.json)
├── exceptions.py        # Custom exception types
├── translations.json    # Translation strings (en, bg)
├── ci/                  # CI/CD automation modules (run by workflows)
│   ├── release.py       # Automated versioning, tagging, changelog
│   ├── publish.py       # Build and publish to Gitea PyPI registry (--skip-build for non-Python repos)
│   ├── auto_merge.py    # Squash-merge PRs with task ID validation
│   ├── check_auto_merge_ready.py  # Pre-merge validation gate (branch, PR title, Vikunja, behind-master)
│   ├── _shared.py      # Shared utilities (get_latest_tag)
│   ├── classify_changes.py  # User-facing vs workflow-only change detection
│   ├── detect_release_commit.py  # Detect release commits on master
│   ├── validate_commit_msg.py    # Conventional commit validation
│   ├── pr_review.py     # Automated PR review + manual reviews (--event, --body, --checklist-confirmed)
│   ├── post_merge.py    # Vikunja task updates after merge
│   ├── sync_wiki.py     # Sync documentation to Gitea wiki
│   ├── push_badges.py   # Generate and push quality badges (--retries for retry on git push failures)
│   ├── notify_failure.py    # Create Gitea issues on CI failures (--auto-login)
│   ├── distribute_files.py  # Distribute files across parallel runners (LPT scheduling)
│   ├── distribute_items.py  # Distribute generic items (VMs, hosts) across parallel runners (LPT)
│   ├── integration_guard.py # Run pytest with cross-runner fail-fast
│   ├── check_translations.py  # Translation completeness check
│   ├── doc_coverage.py  # Documentation coverage check
│   └── lint_docs.py  # Documentation linter (structure, links, headings)
├── tools/               # Developer tooling modules (run locally or by CI)
│   ├── setup.py         # Environment setup (venv, deps, hooks)
│   ├── install_tools.py # Install actionlint, git-cliff, act_runner, tea, hadolint
│   ├── install_checkmake.py  # Install checkmake (Makefile linter)
│   ├── build_image.py   # Build and push Docker images to Gitea registry
│   ├── clean_images.py  # Clean up old Docker image versions from Gitea registry
│   ├── check_test_speed.py  # Measure unit test execution time
│   ├── check_mutable_globals.py  # Detect module-level mutable globals (test isolation bugs)
│   ├── check_pyproject_deps.py   # Validate pyproject.toml deps have documentation comments
│   ├── check_test_coverage.py    # Ensure changed files have corresponding tests (configurable rules)
│   ├── check_agent_docs.py       # Validate docs for stale file references (configurable patterns)
│   ├── configure_repo.py    # Branch protection and label setup
│   ├── generate_badges.py   # Badge SVG generation
│   ├── create_task.py       # Create Vikunja tasks
│   ├── create_pr.py         # Create PRs with auto-derived title from Vikunja
│   ├── pr_status.py         # Check CI status for a PR/commit (--wait polls)
│   ├── pr_logs.py           # Fetch logs for failed CI jobs
│   └── pr_label.py          # Add labels to PRs (idempotent)
├── opentofu.py          # OpenTofu output helpers (get_tofu_output, get_tofu_vm_ip, get_tofu_vm_field)
└── molecule/            # Optional molecule testing helpers (for Ansible projects)
    ├── discover_runners.py    # Dynamic Gitea runner discovery
    ├── distribute_molecule.py # Distribute molecule scenarios across runners (LPT scheduling, --roles-root for multi-role)
    ├── molecule_ci_guard.py   # Run molecule with cross-runner fail-fast (--roles-root)
    ├── molecule_all.py        # Run all molecule scenarios locally
    └── platforms.py           # Supported molecule platforms

Key Design Principles

• Self-contained package — src/devx/ never imports from scripts outside the package
• Module-based invocation — All tools invoked via python -m devx.ci.* or python -m devx.tools.*
• PYTHONPATH: src — Workflows set PYTHONPATH: src (NOT .:src since there are no scripts at repo root)
• Config via env vars — DEVX_* environment variables with .env file fallback

PR Workflow (Mandatory)

Every change to master goes through this workflow. No exceptions.

Branch Protection (Required Gitea Settings)

Branch protection and labels are automatically configured by python -m devx.tools.configure_repo, which runs as a configure-repo job in the post-merge workflow on every push to master.

The following rules are enforced for master:

• Require pull request: No direct pushes to master
• Require approval review: At least 1 APPROVE review before merge
• Require status checks: CI quality must pass
• Block force pushes: No history rewriting on master

1. Create Vikunja Task

Create a task in Vikunja to get a DEVX-N identifier.

2. Create Branch

git checkout master && git pull
git checkout -b DEVX-N-short-description

3. Implement Changes

• Write code following conventions below
• Write/update tests (100% coverage required)
• Update documentation (CHANGELOG, README, AGENTS.md as needed)

4. Commit (Conventional Commits)

Branch commits use conventional commit format (no DEVX-N: prefix):

feat: add new feature
fix: resolve bug
docs: update README

5. Push and Create PR

• PR title format: DEVX-N: <vikunja task title> (must match the Vikunja task title exactly)
• PR body: summary of changes, Closes DEVX-N
• Add ready-to-merge label only after review is complete

6. Review the PR

Automated review (CI pr-review job): Every PR triggers an automated review via python -m devx.ci.pr_review. This job posts a review with COMMENT (no issues) or REQUEST_CHANGES (issues found):

• Architecture compliance (no subprocess in CLI, no hardcoded URLs)
• Best practices (no print(), no bare except, no TODO/FIXME, no functions > 50 lines)
• Security (no hardcoded secrets, no shell=True, no eval/exec)
• i18n (no raw strings in click.echo() without _() wrapper)
• Resource management (no open() without with, no Popen() without cleanup)
• Documentation (source changes must include doc updates)
• Test coverage (source changes must include test updates)
• Commit conventions (conventional commit format on PR commits)

The agent must address all REQUEST_CHANGES issues before proceeding.

7. Address Review Comments

Fix each comment one by one, commit, and push. Re-review until satisfied.

8. Approve and Merge

Once all checklist items are verified and comments are addressed, approve the PR. Then add the ready-to-merge label. The auto-merge workflow will:

1. Validate PR title format (DEVX-N: <vikunja task title>) and match against Vikunja task title
2. Check that at least one substantive APPROVE review exists
3. Wait for all CI checks to pass (including the pr-review job)
4. Squash-merge with title: DEVX-N: <conventional commit message>
5. The post-merge workflow marks the Vikunja task as done
6. The release workflow automatically versions, tags, and publishes

Important

: Never manually merge PRs via the API. Always use the auto-merge workflow by adding the ready-to-merge label.

Automated Release Pipeline

After a PR is merged to master, the post-merge workflow (.gitea/workflows/post-merge.yml) runs automatically:

1. detect-type — Checks if the commit is a regular merge or a release commit (release: vX.Y.Z). All subsequent jobs skip for release commits (except badges).

2. release — Runs python -m devx.ci.release which:

   • Checks for user-facing changes via python -m devx.ci.classify_changes
   • Uses git-cliff to calculate the next semver version from conventional commits
   • Updates __version__ in src/devx/__init__.py (single source of truth)
   • Updates CHANGELOG.md with the new version section
   • Runs make lint-ruff and make pytest-cov to verify the release is healthy
   • Commits with release: vX.Y.Z [skip ci] prefix
   • Creates an annotated tag vX.Y.Z on the release commit
   • Pushes both the commit and tag to master

3. sync-wiki — Syncs documentation to the Gitea wiki. Runs for ALL non-release commits (not just when release succeeds), so docs-only changes still update the wiki.

4. badges — Generates and pushes quality badge SVGs to the badges branch. Uses if: always() so it runs on every push, including release commits.

5. vikunja — Marks the corresponding Vikunja task as done. Runs for ALL non-release commits (not just when release succeeds), so infrastructure-only changes still update the task tracker.

6. publish — Runs after release succeeds (needs: release). Builds and publishes the package to the Gitea PyPI registry. Gets the tag from the release job's tag output (written via GITHUB_OUTPUT).

Smart CI: User-Facing vs Workflow-Only Changes

Not all changes require a new release. The project classifies changes using python -m devx.ci.classify_changes:

Workflow-only paths (infrastructure → no release needed):

• .gitea/** — Gitea Actions workflows
• tests/** — Test files
• AGENTS.md, README.md, CHANGELOG.md — Project docs
• Makefile, cliff.toml, .pre-commit-config.yaml — Config
• .env.example, .gitignore — Config
• hooks/** — Git hooks

User-facing paths (tool changes → release needed) — everything else:

• src/devx/** — Python package source (except __init__.py)
• pyproject.toml — Package metadata
• Any new file type not in the allowlist

Script Separation and Import Rules

Directory Layout

Directory	Purpose	Release impact
src/devx/	User-facing devx package	Changes trigger release
src/devx/ci/	CI/CD automation (run by workflows)	Part of package
src/devx/tools/	Developer tooling (run locally or by CI)	Part of package
tests/	Test files	Workflow-only (no release)

Import Rules

1. src/devx/ is self-contained — the package never imports from outside src/
2. CI modules (devx.ci.*) may import from devx.api_clients, devx.config, devx.gitea_cli, devx.i18n
3. Tool modules (devx.tools.*) may import from devx.api_clients, devx.config, devx.gitea_cli
4. Cross-module imports within devx.ci.* or devx.tools.* are allowed but must be documented

PYTHONPATH Configuration

All workflows use PYTHONPATH: src — devx has no scripts at the repo root, so .:src is not needed. The src directory is the sole import root.

- name: Run script
  env:
    PYTHONPATH: src
  run: python -m devx.ci.example

tea CLI Integration

The tea Gitea CLI tool is used for Gitea API interactions. It is installed by python -m devx.tools.install_tools and configured by python -m devx.tools.setup (login profile from .env CI_GITEA_TOKEN).

devx.gitea_cli.TeaCLI — Python wrapper around tea CLI with JSON output parsing:

• create_issue() — Create issues with labels
• list_labels() / create_label() / add_label() — Label management
• create_pr() / merge_pr() / review_pr() — Pull request operations
• create_release() / list_releases() — Release management

git-cliff Commit Preprocessing

Merge commits on master have the format DEVX-N: <conventional commit>. The cliff.toml includes a commit_preprocessors entry that strips the DEVX-N prefix before parsing. This ensures all merged work appears in the changelog.

Version Bumping Rules (git-cliff)

Commit type	Version bump
feat:	minor (0.X.0)
fix:	patch (0.0.X)
feat!: or BREAKING CHANGE	minor (pre-1.0)
chore:, ci:, docs:	no bump (excluded by cliff.toml)

The version source is __version__ in src/devx/__init__.py, read by setuptools via dynamic = ["version"] in pyproject.toml.

Title Format Summary

What	Format	Example
Branch name	DEVX-N-short-description	DEVX-12-add-release-script
Branch commits	<conventional commit>	feat: add release script
PR title	DEVX-N: <vikunja task title>	DEVX-12: Add release automation
Merge commit	DEVX-N: <conventional commit>	DEVX-12: feat: add release script

Task ID Resolution

auto_merge resolves the task ID solely from the branch name (e.g. DEVX-12-fix-foo → DEVX-12). Branch names must include the task ID prefix — there is no .taskid file fallback. If a stale .taskid file exists in the repo, a deprecation warning is printed advising its removal.

Workflow auto-merge Job and always()

When auto-merge depends on a job that can be skipped (e.g. molecule-tests), the if: condition MUST include always() && at the start. Without it, Gitea Actions skips auto-merge when any dependency is skipped, even if the condition explicitly allows result == 'skipped'.

auto-merge:
  needs: [quality, detect-changes, pr-review, molecule-tests]
  if: >-
    always() &&
    github.event_name == 'pull_request' &&
    needs.quality.result == 'success' &&
    needs.pr-review.result == 'success' &&
    (needs.molecule-tests.result == 'success' || needs.molecule-tests.result == 'skipped')

LPT Test Distribution Algorithm

distribute_molecule and distribute_files use LPT (Longest Processing Time first) scheduling instead of naive round-robin. This produces a more balanced distribution when test items have varying costs:

1. Weight estimation: Each item is assigned a weight:
   • Molecule scenarios: heuristic by name (nextcloud=10, gitea=8, binary=2, default=3). See _SCENARIO_WEIGHTS in distribute_molecule.py.
   • Integration test files: weight by file size in bytes (as a proxy for test runtime).
2. LPT assignment: Items are sorted by weight (descending), then each is assigned to the runner with the least total weight.

This ensures heavy scenarios (e.g. nextcloud) are spread across different runners rather than clustered on one, reducing the longest-runner time from ~16 min to ~11 min with 6 runners.

Config System

devx uses environment variables with .env file fallback for configuration.

DEVX_ Environment Variables

Variable	Default	Description
DEVX_GITEA_API_URL	https://git.oblachno.oblachno.fyi/api/v1	Gitea API base URL
DEVX_VIKUNJA_API_URL	https://work.oblachno.oblachno.fyi/api/v1	Vikunja API base URL
DEVX_REPO_OWNER	(none — must be set)	Repository owner for API calls
DEVX_REPO_NAME	(none — must be set)	Repository name (or owner/repo)
DEVX_TASK_PREFIX	DEVX	Task ID prefix (GRM, OBL-INFRA, etc.)
DEVX_VIKUNJA_PROJECT_ID	6	Vikunja project ID
DEVX_LANG	en	Language for i18n (en, bg)
CI_GITEA_TOKEN	(from .env)	Gitea API token
VIKUNJA_TOKEN	(from .env)	Vikunja API token

Per-Project Overrides

Projects using devx can override the default API URLs and language by setting DEVX_* environment variables or entries in their .env file. The config system loads .env automatically via python-dotenv.

pyproject.toml [tool.devx] Configuration

In addition to DEVX_ env vars, several devx tools read configuration from the [tool.devx] section in pyproject.toml. This allows per-project customization without environment variables.

Base config ([tool.devx]):

• task_prefix — Task ID prefix (e.g. "DEVX", "GRM", "OBL-INFRA")
• vikunja_project_id — Vikunja project ID
• repo_owner / repo_name — Gitea repository coordinates
• gitea_api_url / vikunja_api_url — API endpoints

Tool-specific config:

• [tool.devx.check_mutable_globals] — scan_dirs, skip_dirs, known_safe
• [tool.devx.check_test_coverage] — rules (source_pattern → test_paths mapping), skip_patterns
• [tool.devx.check_agent_docs] — scan_dirs, deleted_files, deprecated_patterns, legitimate_indicators

devx.mak — Shared Makefile Fragment

devx.mak provides common Makefile targets that projects can include via -include $(DEVX_MAK). This eliminates Makefile duplication across projects.

Available targets (all prefixed with devx-):

Target	Purpose
devx-create-task	Create a Vikunja task
devx-create-pr	Create a PR with auto-derived title
devx-push	Push current branch to origin
devx-push-with-pr	Push and create PR in one step
devx-pr-status	Check CI status for a PR (PR=, WAIT=, TIMEOUT=)
devx-pr-logs	Fetch logs for failed CI jobs (PR=, JOB=, TAIL=)
devx-pr-label	Add a label to a PR (PR=, LABEL=ready-to-merge)
devx-pr-review	Post a review on a PR (PR=, EVENT=, BODY=, CHECKLIST=)
devx-check-config	Validate devx configuration
devx-configure-gitea-pypi	Configure Gitea private PyPI registry
devx-env	Create .env from .env.example
devx-venv	Create Python venv with version check
devx-activate-scripts	Create shell/fish/zsh activate scripts
devx-install-hooks	Set git hooks path to hooks/
devx-install-tools	Install actionlint, git-cliff, act_runner, tea, hadolint
devx-install-checkmake	Install checkmake (Makefile linter)
devx-checkmake	Lint Makefiles with checkmake
devx-workflow-lint	Static lint of Gitea Actions YAML (actionlint)
devx-workflow-dryrun	Dry-run all workflows (act_runner)
devx-workflow-dryrun-safe	Best-effort dry-run (skips if act_runner missing)
devx-workflow-check	Static lint + dry-run
devx-notify-failure	Create Gitea issue on CI failure
devx-lint-ruff	Run ruff check
devx-lint-format	Run ruff format --check
devx-typecheck	Run pyright
devx-lint-bandit	Run bandit security scan
devx-lint-deps	Check dependencies for vulnerabilities (pip-audit)
devx-lint	Run all lint targets
devx-test-unit	Run unit tests without coverage
devx-pytest-cov	Run pytest with coverage enforcement
devx-check-mutable-globals	Scan for mutable path globals
devx-check-dep-docs	Validate pyproject.toml deps are documented
devx-check-test-coverage	Check changed files have corresponding tests
devx-check-docs	Validate docs for stale references
devx-check-test-speed	Verify test suite timing
devx-pre-push	Run lint + tests before push
devx-clean	Remove caches, build artifacts, coverage data
devx-setup-image	Link /opt/venv + install project (for pre-built image CI jobs)
devx-lint-dockerfiles	Lint Dockerfiles with hadolint (fail-fast, parameterized by DEVX_DOCKERFILE_PATHS)
devx-build-images	Build Docker images from manifest (no push)
devx-push-images	Build and push Docker images to Gitea registry
devx-build-images-dry-run	Show what would be built/pushed
devx-clean-images	Delete old image versions (keep last 2 + latest)

Variables (set BEFORE including devx.mak):

• DEVX_PYTHON — Python executable (default: python3)
• DEVX_VENV — venv directory (default: .venv)
• DEVX_BIN — venv bin directory (default: $(DEVX_VENV)/bin)
• DEVX_LINT_PATHS — paths for ruff/bandit (default: src/ tests/)
• DEVX_COV_PKG — coverage package (default: src/devx)
• DEVX_TEST_PATHS — pytest paths (default: tests/)
• DEVX_PR_BASE — PR base branch (default: master)
• DEVX_DOCKERFILE_PATHS — directory to search for Dockerfiles (default: docker)
• DEVX_GITEA_REGISTRY — registry URL (default: git.oblachno.oblachno.fyi)
• DEVX_IMAGE_MANIFEST — path to JSON manifest (default: docker/images.json)
• DEVX_IMAGE_OWNER — package owner for cleanup (default: oblachno-oss)

Pre-built Docker Runner Images

devx builds and publishes three tier images to the Gitea container registry to eliminate the 40-120s setup tax on every CI job:

Image	Contains	Used by jobs
ci-base-latest	Python 3.12 + devx[ci] + tea	detect-changes, detect-type, validate-commit-msg, pr-review, auto-merge, sync-wiki, vikunja, configure-repo
ci-quality-latest	ci-base + devx[lint] + actionlint + checkmake + hadolint	quality, badges
ci-full-latest	ci-quality + devx[release,molecule,deploy] + git-cliff + OpenTofu	release, publish, release-dry-run, molecule-tests, deploy jobs

Build process (in build-images.yml workflow):

1. ci-base builds FROM gitea/runner-images:ubuntu-latest
2. ci-quality builds FROM ci-base-latest
3. ci-full builds FROM ci-quality-latest

Each image is tagged latest and pushed to git.oblachno.oblachno.fyi/oblachno-oss/runner-images:<tier>-latest.

Using images in workflows:

jobs:
  quality:
    runs-on: docker
    container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-quality:latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up environment
        run: make setup-image  # links /opt/venv, installs project (no-deps)

Image build/push tools (tested Python modules):

• devx.tools.build_image — Build and push Docker images from Dockerfile or manifest
• devx.tools.clean_images — Delete old image versions via Gitea API (keep last N + latest)

Usage in project Makefile:

DEVX_PYTHON := $(BIN)/python
DEVX_MAK := $(shell $(BIN)/python -c \
    "from pathlib import Path; import devx; print(Path(devx.__file__).parent / 'make' / 'devx.mak')" \
    2>/dev/null)
-include $(DEVX_MAK)

# Aliases for project-specific names
lint-ruff: devx-lint-ruff
workflow-lint: devx-workflow-lint
create-task: devx-create-task

Key Conventions

• Python 3.12+ required (ruff/pyright target py312)
• 100% test coverage required (--cov-fail-under=100)
• Conventional commits on feature branches (no DEVX-N: prefix)
• Branch names must include DEVX-N task ID
• Line length: 120 chars
• Secrets are passed via environment variables, never on the command line
• All user-facing strings wrapped in _() for i18n