devx/AGENTS.md

# AGENTS.md — Project Conventions for devx

## Virtual Environment

All Python tools, tests, and scripts run inside a standard `.venv` directory.
Activate it before running any non-`make` command:

```bash
source activate.sh       # bash/zsh
source activate.fish     # fish
source activate.zsh      # zsh
```

If `.venv` doesn't exist, run `make setup` first. The `make` targets handle
venv activation automatically — always prefer `make <target>` over raw commands.

## Build & Test Commands

```bash
make setup           # Create venv, install deps, set up hooks, install CI tools
make install-tools   # Install actionlint, git-cliff, act_runner, tea, hadolint, vale to ~/.local/bin
make lint-all        # ruff + pyright + bandit + actionlint + lint-dockerfiles
make pytest-cov      # Unit tests with 100% coverage enforcement
make test-unit       # Unit tests without coverage
make workflow-lint   # Static lint of .gitea/workflows/*.yml (actionlint)
make workflow-dryrun # Dry-run all workflows in Docker (act_runner exec --dryrun)
make workflow-check  # workflow-lint + workflow-dryrun
make devx-check-doc-versions  # Verify docs version refs match __version__
make devx-vale       # Run Vale prose linter on docs and README
make clean           # Remove caches, build artifacts, coverage data
make check-workflow-artifact-deps  # Verify artifact download jobs depend on upload jobs
make check-workflow-tofu-init      # Verify tofu-state jobs have a tofu-init step
make check-docker-init             # Check Docker Compose services with healthchecks have init: true
make check-ansible-set-fact-to-json  # Check set_fact tasks don't misuse to_json
make check-alert-rules             # Validate Prometheus alert rules with promtool
```

`make setup` automatically installs all development tools:
- **Python deps** via `python -m devx.tools.setup` (pip install -e .[dev], pre-commit hooks)
- **actionlint, git-cliff, act_runner, tea, hadolint, vale** via `python -m devx.tools.install_tools` (CI/CD tools to ~/.local/bin)
- **tea CLI login** via `python -m devx.tools.setup` (configures `tea login` from `.env` `CI_GITEA_TOKEN`)

## Workflow Verification (Before Push)

Workflow YAML files (`.gitea/workflows/*.yml`) are verified with two tools:

1. **actionlint** — Static linter that catches syntax errors, invalid
   expressions, unknown keys, type mismatches, and shellcheck issues.
   Config: `.gitea/actionlint.yaml` (registers custom `docker` runner label).
   Installed automatically by `make setup` via `python -m devx.tools.install_tools`.

2. **act_runner exec --dryrun** — Gitea's own runner in dry-run mode.
   Validates job dependencies, step ordering, and Docker image selection
   without starting containers. Installed automatically by `make setup`.

Both run via `make workflow-check` and are part of `make lint-all`.
The pre-commit hook runs actionlint automatically when workflow files change.
The CI `validate` job runs `make setup-image` then `make lint-all`.
CI also runs a best-effort `make workflow-dryrun` step (skipped if act_runner is not installed in the CI Docker image).

## Architecture

devx is a reusable Python package providing development and CI/CD tools for oblachno-oss projects.

### Package Structure

```text
src/devx/
├── __init__.py          # Version (single source of truth, read by setuptools)
├── cli.py               # Click-based CLI entry point (devx command)
├── config.py            # Configuration system (DEVX_ env vars, .env loading)
├── api_clients.py       # GiteaClient, VikunjaClient — HTTP API wrappers
├── gitea_cli.py         # TeaCLI — wrapper around tea CLI with JSON parsing
├── i18n.py              # Translation system (gettext-based, translations.json)
├── exceptions.py        # Custom exception types
├── translations.json    # Translation strings (en, bg, de, pl, ru, zh)
├── ci/                  # CI/CD automation modules (run by workflows)
│   ├── release.py       # Automated versioning, tagging, changelog
│   ├── publish.py       # Build, publish to Gitea PyPI registry, create Gitea release (with retry)
│   ├── auto_merge.py    # Squash-merge PRs with task ID validation
│   ├── check_auto_merge_ready.py  # Pre-merge validation gate (branch, PR title, Vikunja, behind-master)
│   ├── _shared.py      # Shared utilities (get_latest_tag)
│   ├── classify_changes.py  # User-facing vs infrastructure change detection
│   ├── detect_release_commit.py  # Detect release commits on master
│   ├── validate_commit_msg.py    # Conventional commit validation
│   ├── pr_review.py     # Automated PR review + manual reviews (--event, --body, --checklist-confirmed)
│   ├── post_merge.py    # Vikunja task updates after merge
│   ├── sync_wiki.py     # Sync documentation to Gitea wiki
│   ├── push_badges.py   # Generate and push quality badges (--retries for retry on git push failures)
│   ├── notify_failure.py    # Create Gitea issues on CI failures (--auto-login)
│   ├── distribute_files.py  # Distribute files across parallel runners (LPT scheduling)
│   ├── distribute_items.py  # Distribute generic items (VMs, hosts) across parallel runners (LPT)
│   ├── integration_guard.py # Run pytest with cross-runner fail-fast
│   ├── check_translations.py  # Translation completeness check
│   ├── doc_coverage.py  # Documentation coverage check
│   ├── lint_docs.py  # Documentation linter (structure, links, headings, code blocks, orphans)
│   ├── validate_deploy_ref.py  # Validate git tag for deployments (--github-output)
│   ├── record_deployed_tag.py  # Record deployed tag to Gitea repo variable
│   ├── cancel_superseded_runs.py  # Cancel in-flight CI runs for the same PR branch
│   ├── check_workflow_artifact_deps.py  # Verify artifact download jobs depend on upload jobs
│   └── check_workflow_tofu_init.py  # Verify tofu-state jobs have a tofu-init step
├── tools/               # Developer tooling modules (run locally or by CI)
│   ├── setup.py         # Environment setup (venv, deps, hooks)
│   ├── install_tools.py # Install actionlint, git-cliff, act_runner, tea, hadolint, vale
│   ├── install_checkmake.py  # Install checkmake (Makefile linter)
│   ├── check_doc_versions.py  # Verify docs version refs match __version__
│   ├── build_image.py   # Build and push Docker images to Gitea registry
│   ├── clean_images.py  # Clean up old Docker image versions from Gitea registry
│   ├── check_test_speed.py  # Measure unit test execution time
│   ├── check_mutable_globals.py  # Detect module-level mutable globals (test isolation bugs)
│   ├── check_pyproject_deps.py   # Validate pyproject.toml deps have documentation comments
│   ├── check_test_coverage.py    # Ensure changed files have corresponding tests (configurable rules)
│   ├── check_agent_docs.py       # Validate docs for stale file references (configurable patterns)
│   ├── check_config.py       # Validate pyproject.toml [tool.devx] config
│   ├── configure_repo.py    # Branch protection and label setup
│   ├── generate_badges.py   # Badge SVG generation
│   ├── generate_cliff_config.py  # Generate git-cliff config (cliff.toml)
│   ├── create_task.py       # Create Vikunja tasks
│   ├── create_pr.py         # Create PRs with auto-derived title from Vikunja
│   ├── pr_status.py         # Check CI status for a PR/commit (--wait polls)
│   ├── pr_logs.py           # Fetch logs for failed CI jobs
│   ├── pr_label.py          # Add labels to PRs (idempotent)
│   ├── pre_push_check.py    # Validate Vikunja task existence before push
│   ├── check_docker_init.py  # Check Docker Compose services with healthchecks have init: true
│   ├── check_ansible_set_fact_to_json.py  # Check set_fact tasks don't misuse to_json
│   ├── check_alert_rules.py  # Validate Prometheus alert rules with promtool
│   └── _shared.py           # Shared tool utilities
├── opentofu.py          # OpenTofu output helpers (get_tofu_output, get_tofu_vm_ip, get_tofu_vm_field)
├── utils/               # Shared utilities (reusable across projects)
│   ├── api.py            # API response helpers (is_truthy, is_falsy) + APIClient base class
│   ├── ssh.py            # SSH exec + wait_for_ssh (pure-Python socket check)
│   ├── crypto.py         # Secret generation (shell-safe passwords)
│   ├── vault.py          # Ansible vault encrypt/decrypt helpers
│   ├── network.py        # HTTP connectivity check + wait_for_ssh
│   ├── confirm.py        # Typed confirmation validation for destructive ops
│   ├── json_registry.py  # File-locked JSON registry for local state
│   ├── step_tracker.py   # Multi-step operation tracking with reports
│   ├── logging.py        # XDG-compliant logging configuration
│   ├── ui.py             # say() — unified click.echo + logging output
│   └── jinja.py          # Jinja2 environment helpers + Ansible-compatible filters
└── molecule/            # Optional molecule testing helpers (for Ansible projects)
    ├── discover_runners.py    # Dynamic Gitea runner discovery
    ├── distribute_molecule.py # Distribute molecule scenarios across runners (LPT scheduling, --roles-root for multi-role)
    ├── molecule_all.py        # Run all molecule scenarios locally
    ├── start_docker.py        # Ensure Docker daemon is running for molecule tests
    └── platforms.py           # Supported molecule platforms
```

### Key Design Principles

- **Self-contained package** — `src/devx/` never imports from scripts outside the package
- **Module-based invocation** — All tools invoked via `python -m devx.ci.*` or `python -m devx.tools.*`
- **PYTHONPATH: src** — Workflows set `PYTHONPATH: src` (NOT `.:src` since there are no scripts at repo root)
- **Config via env vars** — `DEVX_*` environment variables with `.env` file fallback

## PR Workflow (Mandatory)

Every change to master goes through this workflow. No exceptions.

### Branch Protection (Required Gitea Settings)

Branch protection and labels are automatically configured by
`python -m devx.tools.configure_repo`, which runs as a step in the
`detect-and-configure` job in the post-merge workflow on every push to master.

The following rules are enforced for `master`:
- **Require pull request**: No direct pushes to master
- **Require approval review**: At least 1 `APPROVE` review before merge
- **Require status checks**: CI validate must pass
- **Block force pushes**: No history rewriting on master

### 1. Create Vikunja Task
Create a task in Vikunja to get a `DEVX-N` identifier.

**IMPORTANT:** The task title must NOT include the `DEVX-N:` prefix.
The `make create-pr` and `check_auto_merge_ready` commands automatically
prepend `DEVX-N: ` to the Vikunja task title when forming the PR title.
If the Vikunja task title already includes the prefix, the PR title will
have a double prefix and auto-merge validation will fail.

### 2. Create Branch
```bash
git checkout master && git pull
git checkout -b DEVX-N-short-description
```

### 3. Implement Changes
- Write code following conventions below
- Write/update tests (100% coverage required)
- Update documentation (CHANGELOG, README, AGENTS.md as needed)

### 4. Commit (Conventional Commits)
Branch commits use conventional commit format (no `DEVX-N:` prefix):
```text
feat: add new feature
fix: resolve bug
docs: update README
```

### 5. Push and Create PR
- **PR title format**: `DEVX-N: <vikunja task title>` (must match the Vikunja task title exactly)
- PR body: summary of changes, `Closes DEVX-N`
- Add `ready-to-merge` label **only after review is complete**

### 6. Review the PR

**Automated review (CI `validate` job):** Every PR triggers an automated
review via `python -m devx.ci.pr_review` as a step in the `validate` job.
This posts a review with
`COMMENT` (no issues) or `REQUEST_CHANGES` (issues found):

- Architecture compliance (no subprocess in CLI, no hardcoded URLs)
- Best practices (no `print()`, no bare `except`, no `TODO`/`FIXME`,
  no functions > 50 lines)
- Security (no hardcoded secrets, no `shell=True`, no `eval`/`exec`)
- i18n (no raw strings in `click.echo()` without `_()` wrapper)
- Resource management (no `open()` without `with`, no `Popen()` without cleanup)
- Documentation (source changes must include doc updates)
- Test coverage (source changes must include test updates)
- Commit conventions (conventional commit format on PR commits)

The agent **must** address all `REQUEST_CHANGES` issues before proceeding.

### 7. Address Review Comments
Fix each comment one by one, commit, and push. Re-review until satisfied.

### 8. Approve and Merge
Once all checklist items are verified and comments are addressed, approve
the PR. Then add the `ready-to-merge` label. The auto-merge workflow will:
1. **Validate** PR title format (`DEVX-N: <vikunja task title>`) and match against Vikunja task title
2. **Check** that at least one substantive APPROVE review exists
3. Wait for all CI checks to pass (including the `validate` job)
4. Squash-merge with title: `DEVX-N: <conventional commit message>`
5. The post-merge workflow marks the Vikunja task as done
6. The release workflow automatically versions, tags, and publishes

> **IMPORTANT**: Never manually merge PRs via the API. Always use the auto-merge
> workflow by adding the `ready-to-merge` label.

### Automated Release Pipeline

After a PR is merged to master, the **post-merge workflow**
(`.gitea/workflows/post-merge.yml`) runs automatically. Consolidated
into 2 jobs (from 7) to reduce runner overhead:

1. **detect-and-configure** — Configures repo (branch protection, labels),
   detects release commit, validates commit message. Outputs `is-release`
   and `is-automated` for the next job.

2. **release-and-maintain** — Runs all post-merge maintenance as
   conditional steps:
   - **release** (if not a release commit) — Runs `python -m devx.ci.release`
     which checks for user-facing changes via `classify_changes`, uses
     git-cliff for semver, updates `__version__`, updates `CHANGELOG.md`,
     runs lint+tests, commits with `release: vX.Y.Z [skip ci]`, creates
     annotated tag, pushes to master.
   - **publish** (if release created a tag) — Builds and publishes the
     package to the Gitea PyPI registry. Checks out the release tag
     within the same job.
   - **sync-wiki** (if not automated) — Syncs documentation to the Gitea wiki.
   - **vikunja** (if not automated) — Marks the corresponding Vikunja task as done.
   - **badges** (always) — Generates and pushes quality badge SVGs to the
     `badges` branch. Fetches latest master first to pick up release commits.

### Smart CI: User-Facing vs Workflow-Only Changes

Not all changes require a new release. The project classifies changes
using `python -m devx.ci.classify_changes`:

**Workflow-only paths** (infrastructure → no release needed):
- `.gitea/**` — Gitea Actions workflows
- `tests/**` — Test files
- `AGENTS.md`, `README.md`, `CHANGELOG.md` — Project docs
- `Makefile`, `cliff.toml`, `.pre-commit-config.yaml` — Config
- `.env.example`, `.gitignore` — Config
- `hooks/**` — Git hooks

**User-facing paths** (tool changes → release needed) — everything else:
- `src/devx/**` — Python package source (except `__init__.py`)
- `pyproject.toml` — Package metadata
- Any new file type not in the allowlist

## Script Separation and Import Rules

### Directory Layout

| Directory | Purpose | Release impact |
|-----------|---------|----------------|
| `src/devx/` | User-facing devx package | Changes trigger release |
| `src/devx/ci/` | CI/CD automation (run by workflows) | Part of package |
| `src/devx/tools/` | Developer tooling (run locally or by CI) | Part of package |
| `tests/` | Test files | Workflow-only (no release) |

### Import Rules

1. **`src/devx/` is self-contained** — the package never imports from outside `src/`
2. **CI modules** (`devx.ci.*`) may import from `devx.api_clients`, `devx.config`, `devx.gitea_cli`, `devx.i18n`
3. **Tool modules** (`devx.tools.*`) may import from `devx.api_clients`, `devx.config`, `devx.gitea_cli`
4. **Cross-module imports** within `devx.ci.*` or `devx.tools.*` are allowed but must be documented

### PYTHONPATH Configuration

All workflows use `PYTHONPATH: src` — devx has no scripts at the repo root,
so `.:src` is not needed. The `src` directory is the sole import root.

```yaml
- name: Run script
  env:
    PYTHONPATH: src
  run: python -m devx.ci.example
```

### tea CLI Integration

The `tea` Gitea CLI tool is used for Gitea API interactions. It is installed
by `python -m devx.tools.install_tools` and configured by
`python -m devx.tools.setup` (login profile from `.env` `CI_GITEA_TOKEN`).

**`devx.gitea_cli.TeaCLI`** — Python wrapper around `tea` CLI with JSON output parsing:
- `create_issue()` — Create issues with labels
- `list_labels()` / `create_label()` / `add_label()` — Label management
- `create_pr()` / `merge_pr()` / `review_pr()` — Pull request operations
- `create_release()` / `list_releases()` — Release management

**`devx.gitea_cli.configure_tea_login()`** — Configures tea login in
containerized CI environments where `make setup` was not called. Used by
`publish.py` (`--auto-login`) and `notify_failure.py` (`--auto-login`).
Raises `TeaCLIError` if login configuration fails — this prevents cryptic
"no available login" errors from subsequent tea commands.

**Error handling**: `TeaCLI._run()` includes both stdout and stderr in
`TeaCLIError` messages, because `tea` writes some errors (for example,
"no available login") to stdout, not stderr.

**Release creation retry**: `publish.py` retries Gitea release creation
up to 3 times with exponential backoff (2s, 4s) on transient failures.
"Already exists" errors are treated as success (idempotent).

### git-cliff Commit Preprocessing

Merge commits on master have the format `DEVX-N: <conventional commit>`. The
`cliff.toml` includes a `commit_preprocessors` entry that strips the `DEVX-N `
prefix before parsing. This ensures all merged work appears in the changelog.

### Version Bumping Rules (git-cliff)

| Commit type | Version bump |
|-------------|-------------|
| `feat:` | minor (0.X.0) |
| `fix:` | patch (0.0.X) |
| `feat!:` or `BREAKING CHANGE` | minor (pre-1.0) |
| `chore:`, `ci:`, `docs:` | no bump (excluded by cliff.toml) |

The version source is `__version__` in `src/devx/__init__.py`, read by
setuptools via `dynamic = ["version"]` in `pyproject.toml`.

### Title Format Summary

| What | Format | Example |
|------|--------|---------|
| Branch name | `DEVX-N-short-description` | `DEVX-12-add-release-script` |
| Branch commits | `<conventional commit>` | `feat: add release script` |
| PR title | `DEVX-N: <vikunja task title>` | `DEVX-12: Add release automation` |
| Merge commit | `DEVX-N: <conventional commit>` | `DEVX-12: feat: add release script` |

### Task ID Resolution

`auto_merge` resolves the task ID solely from the branch name (for example
`DEVX-12-fix-foo` → `DEVX-12`). Branch names must include the task ID
prefix — there is no `.taskid` file fallback. If a stale `.taskid` file
exists in the repo, a deprecation warning is printed advising its removal.

### Workflow `auto-merge` Job and `always()`

When `auto-merge` depends on a job that can be skipped (for example
`molecule-tests`), the `if:` condition MUST include `always() &&`
at the start. Without it, Gitea Actions skips `auto-merge` when any
dependency is skipped, even if the condition explicitly allows
`result == 'skipped'`.

```yaml
auto-merge:
  needs: [validate, molecule-tests]
  if: >-
    always() &&
    github.event_name == 'pull_request' &&
    needs.validate.result == 'success' &&
    (needs.molecule-tests.result == 'success' || needs.molecule-tests.result == 'skipped')
```

### LPT Test Distribution Algorithm

`distribute_molecule` and `distribute_files` use **LPT (Longest Processing
Time first)** scheduling instead of naive round-robin. This produces a more
balanced distribution when test items have varying costs:

1. **Weight estimation**: Each item is assigned a weight:
   - Molecule scenarios: heuristic by name (`nextcloud`=10, `gitea`=8,
     `binary`=2, default=3). See `_SCENARIO_WEIGHTS` in
     `distribute_molecule.py`.
   - Integration test files: weight by file size in bytes (as a proxy
     for test runtime).
2. **LPT assignment**: Items are sorted by weight (descending), then
   each is assigned to the runner with the least total weight.

This ensures heavy scenarios (for example `nextcloud`) are spread across
different runners rather than clustered on one, reducing the
longest-runner time from ~16 min to ~11 min with 6 runners.

## Config System

devx uses environment variables with `.env` file fallback for configuration.

### DEVX_ Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `DEVX_GITEA_API_URL` | `https://git.oblachno.oblachno.fyi/api/v1` | Gitea API base URL |
| `DEVX_VIKUNJA_API_URL` | `https://work.oblachno.oblachno.fyi/api/v1` | Vikunja API base URL |
| `DEVX_REPO_OWNER` | **(none — must be set)** | Repository owner for API calls |
| `DEVX_REPO_NAME` | **(none — must be set)** | Repository name (or `owner/repo`) |
| `DEVX_TASK_PREFIX` | `DEVX` | Task ID prefix (GRM, OBL-INFRA, etc.) |
| `DEVX_VIKUNJA_PROJECT_ID` | `6` | Vikunja project ID |
| `DEVX_LANG` | `en` | Language for i18n (en, bg, de, pl, ru, zh) |
| `CI_GITEA_TOKEN` | (from .env) | Gitea API token |
| `VIKUNJA_TOKEN` | (from .env) | Vikunja API token |

### Per-Project Overrides

Projects using devx can override the default API URLs and language by setting
`DEVX_*` environment variables or entries in their `.env` file. The config
system loads `.env` automatically via `python-dotenv`.

### pyproject.toml [tool.devx] Configuration

In addition to `DEVX_` env vars, many devx tools read configuration from
the `[tool.devx]` section in `pyproject.toml`. This allows per-project
customization without environment variables.

**Base config** (`[tool.devx]`):
- `task_prefix` — Task ID prefix (for example `"DEVX"`, `"GRM"`, `"OBL-INFRA"`)
- `vikunja_project_id` — Vikunja project ID
- `repo_owner` / `repo_name` — Gitea repository coordinates
- `gitea_api_url` / `vikunja_api_url` — API endpoints

**Tool-specific config**:
- `[tool.devx.check_mutable_globals]` — `scan_dirs`, `skip_dirs`, `known_safe`
- `[tool.devx.check_test_coverage]` — `rules` (source_pattern → test_paths mapping), `skip_patterns`
- `[tool.devx.check_agent_docs]` — `scan_dirs`, `deleted_files`, `deprecated_patterns`, `legitimate_indicators`

## devx.mak — Shared Makefile Fragment

`devx.mak` provides common Makefile targets that projects can include
via `-include $(DEVX_MAK)`. This eliminates Makefile duplication across
projects.

**Available targets** (all prefixed with `devx-`):

| Target | Purpose |
|--------|---------|
| `devx-create-task` | Create a Vikunja task |
| `devx-create-pr` | Create a PR with auto-derived title |
| `devx-push` | Push current branch to origin |
| `devx-push-with-pr` | Push and create PR in one step |
| `devx-pr-status` | Check CI status for a PR (`PR=`, `WAIT=`, `TIMEOUT=`) |
| `devx-pr-logs` | Fetch logs for failed CI jobs (`PR=`, `JOB=`, `TAIL=`) |
| `devx-pr-label` | Add a label to a PR (`PR=`, `LABEL=ready-to-merge`) |
| `devx-pr-review` | Post a review on a PR (`PR=`, `EVENT=`, `BODY=`, `CHECKLIST=`) |
| `devx-check-config` | Validate devx configuration |
| `devx-configure-gitea-pypi` | Configure Gitea private PyPI registry |
| `devx-env` | Create .env from .env.example |
| `devx-venv` | Create Python venv with version check |
| `devx-activate-scripts` | Create shell/fish/zsh activate scripts |
| `devx-install-hooks` | Set git hooks path to hooks/ |
| `devx-install-tools` | Install actionlint, git-cliff, act_runner, tea, hadolint |
| `devx-install-checkmake` | Install checkmake (Makefile linter) |
| `devx-checkmake` | Lint Makefiles with checkmake |
| `devx-workflow-lint` | Static lint of Gitea Actions YAML (actionlint) |
| `devx-workflow-dryrun` | Dry-run all workflows (act_runner) |
| `devx-workflow-dryrun-safe` | Best-effort dry-run (skips if act_runner missing) |
| `devx-workflow-check` | Static lint + dry-run |
| `devx-notify-failure` | Create Gitea issue on CI failure |
| `devx-lint-ruff` | Run ruff check |
| `devx-lint-format` | Run ruff format --check |
| `devx-typecheck` | Run pyright |
| `devx-lint-bandit` | Run bandit security scan |
| `devx-lint-deps` | Check dependencies for vulnerabilities (pip-audit) |
| `devx-lint` | Run all lint targets |
| `devx-test-unit` | Run unit tests without coverage |
| `devx-pytest-cov` | Run pytest with coverage enforcement |
| `devx-check-mutable-globals` | Scan for mutable path globals |
| `devx-check-dep-docs` | Validate pyproject.toml deps are documented |
| `devx-check-test-coverage` | Check changed files have corresponding tests |
| `devx-check-docs` | Validate docs for stale references |
| `devx-check-test-speed` | Verify test suite timing |
| `devx-pre-push` | Run lint + tests before push |
| `devx-clean` | Remove caches, build artifacts, coverage data |
| `devx-setup-image` | Link /opt/venv + install project (for pre-built image CI jobs) |
| `devx-lint-dockerfiles` | Lint Dockerfiles with hadolint (fail-fast, parameterized by `DEVX_DOCKERFILE_PATHS`) |
| `devx-build-images` | Build Docker images from manifest (no push) |
| `devx-push-images` | Build and push Docker images to Gitea registry |
| `devx-build-images-dry-run` | Show what would be built/pushed |
| `devx-clean-images` | Delete old image versions (keep last 2 + latest) |

**Variables** (set BEFORE including devx.mak):
- `DEVX_PYTHON` — Python executable (default: `python3`)
- `DEVX_VENV` — venv directory (default: `.venv`)
- `DEVX_BIN` — venv bin directory (default: `$(DEVX_VENV)/bin`)
- `DEVX_LINT_PATHS` — paths for ruff/bandit (default: `src/ tests/`)
- `DEVX_COV_PKG` — coverage package (default: `src/devx`)
- `DEVX_TEST_PATHS` — pytest paths (default: `tests/`)
- `DEVX_PR_BASE` — PR base branch (default: `master`)
- `DEVX_DOCKERFILE_PATHS` — directory to search for Dockerfiles (default: `docker`)
- `DEVX_GITEA_REGISTRY` — registry URL (default: `git.oblachno.oblachno.fyi`)
- `DEVX_IMAGE_MANIFEST` — path to JSON manifest (default: `docker/images.json`)
- `DEVX_IMAGE_OWNER` — package owner for cleanup (default: `oblachno-oss`)

## Pre-built Docker Runner Images

devx builds and publishes three tier images to the Gitea container registry
to eliminate the 40-120s setup tax on every CI job:

| Image | Contains | Used by jobs |
|-------|----------|-------------|
| `ci-base-latest` | Python 3.12 + devx[ci] + tea | auto-merge, detect-and-configure |
| `ci-quality-latest` | ci-base + devx[lint] + actionlint + checkmake + hadolint | (badges in release-and-maintain uses ci-full) |
| `ci-full-latest` | ci-quality + devx[release,molecule,deploy] + git-cliff + OpenTofu | validate, release-and-maintain, molecule-tests, build-and-push |

**Build process** (in `build-images.yml` workflow):
1. `ci-base` builds FROM `gitea/runner-images:ubuntu-latest`
2. `ci-quality` builds FROM `ci-base-latest`
3. `ci-full` builds FROM `ci-quality-latest`

Each image is tagged `latest` and pushed to
`git.oblachno.oblachno.fyi/oblachno-oss/runner-images:<tier>-latest`.

**Using images in workflows**:
```yaml
jobs:
  validate:
    runs-on: docker
    container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up environment
        run: make setup-image  # links /opt/venv, installs project (no-deps)
```

**Image build/push tools** (tested Python modules):
- `devx.tools.build_image` — Build and push Docker images from Dockerfile or manifest
- `devx.tools.clean_images` — Delete old image versions via Gitea API (keep last N + latest)

**Usage in project Makefile**:
```makefile
DEVX_PYTHON := $(BIN)/python
DEVX_MAK := $(shell $(BIN)/python -c \
    "from pathlib import Path; import devx; print(Path(devx.__file__).parent / 'make' / 'devx.mak')" \
    2>/dev/null)
-include $(DEVX_MAK)

# Aliases for project-specific names
lint-ruff: devx-lint-ruff
workflow-lint: devx-workflow-lint
create-task: devx-create-task
```

## Key Conventions

- Python 3.12+ required (ruff/pyright target `py312`)
- 100% test coverage required (`--cov-fail-under=100`)
- Conventional commits on feature branches (no `DEVX-N:` prefix)
- Branch names must include `DEVX-N` task ID
- Line length: 120 chars
- Secrets are passed via environment variables, never on the command line
- All user-facing strings wrapped in `_()` for i18n

### Container-Level Fix Verification (Mandatory)

**Rule:** Before pushing any fix that modifies container state (CA certs,
config files, installed packages, daemon restarts), reproduce the exact
sequence locally with the actual Docker image. Do not push to CI as the
first test.

This is a hard rule, not a suggestion. CI cycles take 20+ minutes and
ephemeral staging VMs are destroyed after each run, making interactive
debugging impossible. A local reproduction takes 30 seconds and catches
silent failures immediately.

**Procedure:**
1. `docker pull <actual_image>`
2. `docker run -d --name <test> ...` and wait for it to start
3. Run the exact commands from the Ansible task or script
4. Verify the state change took effect
5. Clean up: `docker rm -f <test>`

### Verified State Modification (Mandatory)

Ansible tasks that modify container state with `changed_when: false`
MUST include a post-task verification step that confirms the state
change took effect. `changed_when: false` suppresses both change
detection AND failure visibility — a task can silently do nothing and
report `ok`.

## Subagent Delegation Policy

Custom subagent profiles are defined in `.devin/agents/` (project-specific)
and `~/.config/devin/agents/` (global, shared across repos). The agent MUST
automatically delegate to the appropriate subagent based on the task —
the user should not need to specify which profile to use.

### Available Profiles

**Global** (shared across all projects):

| Profile | Location | Purpose |
|---------|----------|---------|
| `pr-reviewer` | `~/.config/devin/agents/` | 13-category PR checklist + quality gates |
| `release-check` | `~/.config/devin/agents/` | Pre-merge readiness validation |

**devx-specific** (in `.devin/agents/`):

| Profile | Purpose |
|---------|---------|
| `ci-investigator` | Investigate CI failures (validate, release-and-maintain, build-images) |
| `dep-upgrader` | Python dependency upgrades in pyproject.toml with dep-doc validation |
| `docker-image-builder` | Build/push/cleanup 3-tier runner images (ci-base, ci-quality, ci-full) |
| `doc-sync-specialist` | Doc coverage, doc linting, wiki sync integrity |
| `workflow-validator` | actionlint + act_runner dry-run validation |

### When to Delegate Automatically

| Trigger | Profile | Mode |
|---------|---------|------|
| CI run failure (validate, release-and-maintain, build-images) | `ci-investigator` | Background |
| PR ready for review | `pr-reviewer` | Foreground |
| Dependency upgrade requested | `dep-upgrader` | Background |
| Docker image build/push needed | `docker-image-builder` | Background |
| Doc coverage failure or wiki sync issue | `doc-sync-specialist` | Background |
| Workflow YAML modified or validation needed | `workflow-validator` | Background |
| Branch ready for merge | `release-check` | Foreground |

### Delegation Rules

1. **Auto-select the profile.** Do not ask the user which profile to use.
2. **Background by default, foreground when blocking.**
3. **Provide full context in the prompt** — subagents don't inherit conversation history.
4. **One subagent per concern.** Chain: investigate → fix in main session → review.
5. **Don't delegate minor work** (<30s, <50 lines of context).
6. **Compact after subagent returns.**
7. **Never skip delegation to save time** — it keeps main context small.


## Feedback Issue Handling

Subagents create Gitea issues in the current repo when they encounter
tool, workflow, or process issues that warrant follow-up. These issues
use the `feedback` label plus a category label (`tooling`,
`ci-improvement`, `doc-improvement`, `workflow-improvement`).

Standard labels are created automatically by `configure_repo` (runs in
post-merge on every master push). If a label does not exist yet, the
subagent's issue creation will still succeed — labels can be added
afterwards.

### When a Subagent Reports a Feedback Issue URL

1. **Acknowledge it** in your response to the user — mention the issue URL
2. **Do NOT close or modify** the issue — it is for follow-up work
3. **Do NOT create a PR** to address it unless the user explicitly asks
4. If the user asks to address feedback, spawn a subagent to investigate
   the issue and implement a fix

### Creating Feedback Issues Manually

As the parent agent, you can also create feedback issues directly using
the Gitea MCP (`issue_write` with `create_issue` method). Follow the
same format as subagents:

- Title: `[feedback] <category>: <short description>`
- Labels: `feedback` + category label
- Body: include context, tool/workflow, issue, reproduction, affected
  files, suggested investigation, and "Reported by: parent agent"

Always deduplicate first via `list_issues` with `labels: "feedback"`.