devx/docs/tech/ci-cd-workflow.md

# CI/CD Workflow

devx uses Gitea Actions for CI/CD automation. Two workflows implement a
complete pipeline: pull request validation and post-merge release
automation (including publishing).

## Workflow overview

```text
PR opened/synchronized ──► CI (ci.yml)
    │                         ├── validate (quality + detect-changes +
    │                         │              release-dry-run + pr-review +
    │                         │              pre-merge validation)
    │                         └── auto-merge ──► squash-merge to master
    │                                              │
    ▼                                              ▼
Push to master ──► Post-merge (post-merge.yml)
                     ├── detect-and-configure (detect-type +
                     │                          validate-commit-msg +
                     │                          configure-repo)
                     └── release-and-maintain
                          ├── release ──► tag vX.Y.Z
                          ├── publish ──► Gitea PyPI registry + Gitea release
                          ├── sync-wiki
                          ├── vikunja
                          └── badges (always runs)
```

## CI workflow (`ci.yml`)

Runs on pull requests (opened and synchronize) and manual dispatch.

### Jobs

#### `validate`

The single validation job. Consolidates the former `quality`,
`detect-changes`, `release-dry-run`, `pr-review`, and `pre-merge-check`
jobs into one job to save checkout+setup overhead. Runs on every PR.

**Quality steps**

The main quality gate:

1. **Lint all** — ruff check, ruff format check, pyright, bandit, actionlint
   (via `make lint-all`)
2. **Unit tests with 100% coverage** — `make pytest-cov`
3. **Check unit test speed** — `python -m devx.tools.check_test_speed
   --max-seconds 4 --max-single-seconds 0.5`
4. **Documentation coverage check** — `python -m devx.ci.doc_coverage
   --fail-on-missing`
5. **Translation completeness check** — `python -m devx.ci.check_translations`
6. **Dependency security scan** — `pip-audit --desc --skip-editable`
   (best-effort, non-blocking)
7. **Workflow dry-run validation** — `make workflow-dryrun` via act_runner
   (best-effort, skipped if act_runner is not installed)

**`detect-changes` step**

Classifies changes between `origin/master` and the PR head as user-facing or
workflow-only using `python -m devx.ci.classify_changes --github-output`.
Writes `user-facing-changed=true|false` to the job output for use by
downstream steps.

**`release-dry-run` step**

Only runs if the detect-changes step detected user-facing changes. Runs
`python -m devx.ci.release --dry-run` to validate that the release script
can calculate the next version and generate the changelog without making
changes. Non-blocking (uses `|| true`).

**`pr-review` step**

Runs on every pull request. Executes `python -m devx.ci.pr_review` with the
PR number and repository. Fetches the PR diff via the Gitea API and runs
automated checks, posting a structured review:

- `COMMENT` — no issues found
- `REQUEST_CHANGES` — issues found that must be addressed

Checks performed:
1. Architecture compliance — no subprocess in CLI, no hardcoded URLs
2. Best practices — no `print()`, no bare `except`, no `TODO`/`FIXME`,
   no functions > 50 lines
3. Security — no hardcoded secrets, no `shell=True`, no `eval`/`exec`
4. i18n — no raw strings in `click.echo()` without `_()` wrapper
5. Resource management — no `open()` without `with`, no `Popen()` without
   cleanup
6. Documentation — source changes must include doc updates
7. Test coverage — source changes must include test updates
8. Commit conventions — conventional commit format on PR commits

**Pre-merge validation step**

Runs on every pull request. Executes
`python -m devx.ci.check_auto_merge_ready` with the branch name, PR title,
repository, and PR number. Validates auto-merge preconditions before the
`auto-merge` job runs:

1. **Branch name** — must contain a valid task ID (for example,
   `DEVX-12-fix-foo` → `DEVX-12`)
2. **PR title format** — must be `{PREFIX}-N: <vikunja task title>`
3. **Vikunja task** — must exist and the title must match the PR title
4. **Branch state** — must not be behind master

#### `auto-merge`

Depends on `validate`. The final job in the CI workflow. Runs
`python -m devx.ci.auto_merge` with the branch name, PR title, repository,
and PR number:

1. **Read task ID** from branch name (for example, `DEVX-12-fix-foo` → `DEVX-12`)
2. **Validate PR title format** — must be `{PREFIX}-N: <vikunja task title>`
3. **Validate PR title matches Vikunja task** — fetches the Vikunja task and
   compares the title
4. **Extract conventional commit message** from PR commits (newest matching
   conventional format)
5. **Squash-merge** with title `{PREFIX}-N <conventional commit message>`
6. If the head branch is behind master (HTTP 405), automatically pulls master,
   rebases, force-pushes, and retries the merge

The merge commit push to master triggers the post-merge workflow.

### Smart CI: user-facing vs workflow-only changes

Not all changes require a new release. The `detect-changes` step in the
`validate` job classifies changes using
`python -m devx.ci.classify_changes`:

**Workflow-only paths** (infrastructure — no release needed):
- `.gitea/**` — Gitea Actions workflows
- `tests/**` — Test files
- `AGENTS.md`, `README.md`, `CHANGELOG.md` — Project docs
- `Makefile`, `cliff.toml`, `.pre-commit-config.yaml` — Config
- `.env.example`, `.gitignore` — Config
- `hooks/**` — Git hooks
- `src/devx/__init__.py` — Only contains `__version__` (release artifact)

**User-facing paths** (tool changes — release needed) — everything else:
- `src/devx/**` — Python package source (except `__init__.py`)
- `pyproject.toml` — Package metadata
- Any new file type not in the allowlist

Classification is configured in `pyproject.toml` under
`[tool.devx.classify]`. The framework provides `DEFAULT_INFRASTRUCTURE` — a
curated list of paths that are infrastructure for any Python project. Projects
inherit these automatically and only specify what is different.

Rule priority (first match wins):
1. `user_facing_overrides` — safety override (highest priority)
2. `infrastructure_overrides` — explicit per-file
3. `infrastructure` — DEFAULT_INFRASTRUCTURE + project-specific patterns
4. Default: user-facing (safe — any unknown file triggers release)

## Post-merge workflow (`post-merge.yml`)

Runs on every push to master. Consolidated into 2 jobs (from 7) to reduce
runner overhead: `detect-and-configure` (detect-type + validate-commit-msg +
configure-repo) and `release-and-maintain` (release + publish + sync-wiki +
badges + vikunja). Individual steps within `release-and-maintain` are
conditional on the `detect-and-configure` job's outputs.

### Job dependency graph

```text
detect-and-configure
    ├── configure-repo (independent, skip if release commit)
    ├── detect-type → is-release? is-automated?
    └── validate-commit-msg (skip if release commit)
            │
            ▼
release-and-maintain (needs detect-and-configure)
    ├── release (skip if release commit or workflow-only)
    │       └── publish (if release created a tag)
    ├── sync-wiki (skip if automated)
    ├── vikunja (skip if automated)
    └── badges (always runs)
```

`sync-wiki` and `vikunja` run only on non-automated commits (that is, real PR
merges) so that the wiki and task tracker are only updated when a human
change lands. They skip on release commits and automated commits.

The `badges` step always runs (even on release commits) so badges (tests,
coverage, version, etc.) are always current. It runs last so it picks up
any version bump the release step created.

When `release` creates a `release: vX.Y.Z` commit, the release commit's
post-merge run still updates badges (the version badge picks up the new
version). Other steps skip. The `publish` step builds and publishes the
package to the Gitea PyPI registry within the same `release-and-maintain`
job (it checks out the release tag).

### Post-merge jobs

#### `detect-and-configure`

The first post-merge job. Consolidates the former `detect-type`,
`validate-commit-msg`, and `configure-repo` jobs. Outputs `is-release`,
`is-automated`, and `user-facing-changed` for the `release-and-maintain`
job.

**`detect-type` step**

Checks if the latest commit is a release commit (`release: vX.Y.Z [skip ci]`)
using `python -m devx.ci.detect_release_commit`. Writes `is-release=true` or
`is-release=false` (and `is-automated`) to the job output. The
`release-and-maintain` job uses these to conditionally skip steps for
release commits.

**`validate-commit-msg` step**

Skips for release/automated commits. Validates the latest commit message
using `python -m devx.ci.validate_commit_msg --branch master`. On master,
commits must follow `{PREFIX}-N: <conventional commit>` format (added by
auto-merge).

**`configure-repo` step**

Ensures branch protection and labels are configured using
`python -m devx.tools.configure_repo --repo <name> --owner <owner>`:

- Sets up master branch protection (required status checks, block on rejected
  reviews, block on outdated branch)
- Creates standard labels
- Status check contexts read from `DEVX_STATUS_CHECKS` or default to
  `CI / validate (pull_request)`

On failure, the `notify_failure` step creates a Gitea issue.

#### `release-and-maintain`

Depends on `detect-and-configure`. The second post-merge job. Consolidates
the former `release`, `publish`, `sync-wiki`, `badges`, and `vikunja` jobs.
Individual steps are conditional on the `detect-and-configure` job's outputs.

**`release` step**

Skips for release commits and workflow-only changes. The core release
automation step. Runs `python -m devx.ci.release`:

1. **Classify changes** — calls `classify_changes.py` to check for user-facing
   changes. If only infrastructure files changed, exits without releasing.
2. **Calculate next version** — uses git-cliff to determine the next semver
   version from conventional commits since the last tag
3. **Update version file** — updates `__version__` in `src/devx/__init__.py`
4. **Update changelog** — prepends the new version section to `CHANGELOG.md`
   using git-cliff output
5. **Run tests** — executes `make lint-ruff` and `make pytest-cov` to verify
   the release is healthy. If either fails, the release is aborted — no
   commit, no tag. Use `--skip-tests` only for emergency releases.
6. **Commit** — stages the version file and changelog, commits with
   `release: vX.Y.Z [skip ci]` (uses `--no-verify` to bypass the commit-msg
   hook since release commits are a special case)
7. **Create tag** — creates an annotated tag `vX.Y.Z` with the changelog as
   the tag message
8. **Push** — pushes both the commit and tag to master

The script is idempotent: if there are no new conventional commits since the
last tag, it exits without doing anything. If the tag already exists (for example,
from a partial previous run), it skips tag creation and only pushes.

**Tag consistency**: Before releasing, the script fetches remote tags and
verifies all existing tags point to commits whose message matches the tag
version. This prevents duplicate release commits and ensures
tag/version/commit alignment.

**Version bumping rules** (git-cliff):

| Commit type | Version bump |
|-------------|-------------|
| `feat:` | minor (0.X.0) |
| `fix:` | patch (0.0.X) |
| `feat!:` or `BREAKING CHANGE` | minor (pre-1.0) |
| `chore:`, `ci:`, `docs:` | no bump (excluded by cliff.toml) |

On failure, the `notify_failure` step creates a Gitea issue via
`python -m devx.ci.notify_failure`.

**`sync-wiki` step**

Skips for automated commits. Syncs documentation from `docs/` to the Gitea
wiki using `python -m devx.ci.sync_wiki --repo <owner/repo> --strict`:

1. Reads `docs/mapping.json` to map file paths to wiki page titles
2. Lists existing wiki pages via the Gitea API
3. For each mapped file, reads content and creates or updates the wiki page
4. `--strict` runs a full integrity check: verifies page count, missing
   pages, stale pages, and content match. Fails if any page is empty or
   content doesn't match.

Pages that exist in the wiki but not in the mapping are left untouched (not
deleted).

On failure, the `notify_failure` step creates a Gitea issue.

**`badges` step**

Always runs (even on release commits). Generates and pushes quality badges
using `python -m devx.ci.push_badges`:

1. **Fetch latest master** — `git fetch origin master && git reset --hard
   origin/master` (ensures the version badge reflects the current state,
   even if the release step recently pushed a new version)
2. **Generate badges** — calls `devx.tools.generate_badges` which runs
   pytest-cov, doc-coverage, lint checks, and version extraction, then writes
   SVG files: `coverage.svg`, `tests.svg`, `docs.svg`, `quality.svg`,
   `version.svg`, `python.svg`
3. **Push to badges branch** — creates an orphan `badges` branch, copies SVG
   files, commits, and force-pushes
4. **Update README/docs** — switches back to master, replaces
   `raw/branch/badges/<name>.svg` URLs with `raw/commit/<sha>/<name>.svg`
   URLs (cache-busting — Gitea caches `raw/branch/` URLs for 6 hours),
   commits, and pushes

Supports `--retries` for retrying on git push failures (fetches latest master
and waits 10s between attempts).

On failure, the `notify_failure` step creates a Gitea issue.

**`vikunja` step**

Skips for automated commits. Updates the Vikunja task after a merge using
`python -m devx.ci.post_merge --git-sha <sha>`:

1. Extracts the task ID from the first line of the commit message
2. Marks the corresponding Vikunja task as done
3. Posts a comment with the merge SHA

On failure, the `notify_failure` step creates a Gitea issue.

**`publish` step**

Only runs if the `release` step created a tag. Builds and publishes the
package within the same `release-and-maintain` job (checks out the release
tag). Runs `python -m devx.ci.publish <tag> <owner/repo>`:

1. **Install dependencies** — build, twine, requests, python-dotenv, click,
   and the project itself
2. **Install CI tools** — git-cliff and tea via
   `python -m devx.tools.install_tools`
3. **Configure tea login** — `tea login add` using `CI_GITEA_TOKEN`
4. **Build and publish** — `python -m devx.ci.publish <tag> <owner/repo>`:
   - Build the package with `python -m build`
   - Publish to the Gitea PyPI registry (default) using `twine upload
     --repository-url <url> -u <token> -p <token>`
   - OR publish to standard PyPI if `PYPI_TOKEN` is set
   - OR skip publishing if `--skip-build` is passed (non-Python repos)
   - Create a Gitea release with git-cliff-generated release notes via
     `tea create release`

Publishing destination resolution (checked in order):
1. **Gitea PyPI registry** — if `--registry-url` is given, or
   `DEVX_PYPI_REGISTRY_URL` env var is set, or derived from `GITEA_API_URL`
2. **Standard PyPI** — if `PYPI_TOKEN` is set (takes precedence over Gitea
   registry)
3. **Skip** — if neither is configured, only the Gitea release is created

On failure, the `notify_failure` step creates a Gitea issue.

## CI scripts

### `auto_merge.py`

Auto-merge PR when all CI checks pass. Reads task ID from the branch name
(for example, `DEVX-12-fix-foo` → `DEVX-12`). Validates PR title format, checks the
Vikunja task exists and the title matches, extracts the conventional commit
message from PR commits, and squash-merges with
`{PREFIX}-N <conventional commit>` title.

```bash
python -m devx.ci.auto_merge <branch> <pr_title> <owner/repo> <pr_number>
```

### `release.py`

Automated release using git-cliff. Calculates next semver version from
conventional commits, updates `__version__` and `CHANGELOG.md`, runs lint and
tests, commits with `release: vX.Y.Z [skip ci]`, creates annotated tag, and
pushes. Idempotent — exits if no unreleased changes.

```bash
python -m devx.ci.release [--dry-run] [--skip-tests] [--verify]
```

- `--dry-run` — preview without making changes
- `--skip-tests` — skip lint and test verification (emergency only)
- `--verify` — check tag/version/changelog alignment and exit

### `publish.py`

Builds package, publishes to Gitea PyPI registry or standard PyPI, and
creates a Gitea release with git-cliff-generated notes.

```bash
python -m devx.ci.publish <tag> <owner/repo> [--registry-url <url>] [--skip-build]
```

### `pr_review.py`

Automated PR review. Fetches the PR diff via the Gitea API, runs automated
checks (architecture, best practices, security, i18n, resource management,
documentation, test coverage, commit conventions), and posts a structured
review with inline comments.

```bash
python -m devx.ci.pr_review <pr_number> <owner/repo>
```

### `notify_failure.py`

Creates a Gitea issue when a CI workflow fails. Uses the tea CLI for issue
creation with failure labels. Supports `--auto-login` to configure the tea
CLI login profile from `CI_GITEA_TOKEN`.

```bash
python -m devx.ci.notify_failure --repo <owner/repo> --run-id <id> \
  --workflow <name> --commit <sha> [--auto-login]
```

### `post_merge.py`

Updates Vikunja task after a merge to master. Extracts task ID from the
commit message, marks the task as done, and posts a comment with the merge SHA.

```bash
python -m devx.ci.post_merge <commit_msg> [--commit-sha <sha>] [--git-sha <sha>]
```

### `classify_changes.py`

Classifies git changes as user-facing or workflow-only. Uses a layered rule
system configured in `pyproject.toml`. Safe-by-default: any unknown file
defaults to user-facing.

```bash
python -m devx.ci.classify_changes [--base <ref>] [--head <ref>] \
  [--quiet] [--check <category>] [--github-output]
```

### `discover_runners.py`

Discovers available Gitea Actions runners at repository, organization, and
instance levels. Falls back to `MOLECULE_RUNNERS` repo variable or
`DEFAULT_MAX_RUNNERS` (3).

```bash
python -m devx.ci.discover_runners --owner <owner> --repo <repo> [--count] [--indices]
```

### `detect_release_commit.py`

Detects whether the latest git commit is a release commit. Writes
`is-release=true|false` to `$GITHUB_OUTPUT`.

```bash
python -m devx.ci.detect_release_commit
```

### `push_badges.py`

Generates SVG badge files, pushes them to the `badges` branch, and updates
README.md and docs/index.md with cache-busting `raw/commit/<sha>/` URLs.

```bash
python -m devx.ci.push_badges [--output-dir <dir>] [--branch <branch>] \
  [--no-readme-update] [--retries <n>]
```

### `distribute_molecule.py`

Distributes molecule (scenario, platform) pairs across N parallel runners.
Discovers scenarios under `ansible/roles/*/molecule/`.

```bash
python -m devx.molecule.distribute_molecule --runner-index <i> --max-runners <n>
python -m devx.molecule.distribute_molecule --list
python -m devx.molecule.distribute_molecule --list-platforms
```

### `molecule_ci_guard.py`

Runs molecule tests sequentially while polling the Gitea API for other runner
failures. Aborts early if another runner fails the same job.

```bash
python -m devx.molecule.molecule_ci_guard [--roles-root <dir>] pair1 pair2 ...
```

### `validate_commit_msg.py`

Validates commit messages. On feature branches: conventional commits only
(no `{PREFIX}-N` prefix). On master: must have `{PREFIX}-N` prefix from
auto-merge, followed by a conventional commit message.

```bash
python -m devx.ci.validate_commit_msg <commit_msg_file> [--branch <branch>]
```

### `sync_wiki.py`

Syncs documentation from `docs/` to the Gitea wiki via the API. Reads
`docs/mapping.json` for file-to-page mapping. Supports `--dry-run`,
`--verify`, and `--strict` (full integrity check).

```bash
python -m devx.ci.sync_wiki [--dry-run] [--repo <owner/repo>] [--verify] [--strict]
```

### `check_translations.py`

Validates translation files against the Python source code. Checks for
missing keys, dead keys, and missing languages.

```bash
python -m devx.ci.check_translations [--translations <file>]...
```

### `doc_coverage.py`

Checks documentation coverage for CLI commands and major modules. Parses
Click commands from `cli.py` and verifies documentation exists.

```bash
python -m devx.ci.doc_coverage [--docs-dir <dir>] [--fail-on-missing]
```

### `distribute_files.py`

Distributes files matching a glob pattern across N parallel runners
(round-robin). Writes the assigned file list to `$GITHUB_ENV`.

```bash
python -m devx.ci.distribute_files --pattern <glob> --runner-index <i> \
  --max-runners <n> [--github-env] [--skip-if-excess]
```

### `integration_guard.py`

Runs pytest with cross-runner failure detection. If any other
integration-tests matrix runner reports failure, the current pytest
subprocess is killed and this runner exits early.

```bash
python -m devx.ci.integration_guard -- <pytest args>
```

## Release process summary

The complete release process from PR to published package:

1. **PR merged** — `auto-merge` squash-merges the PR to master with
   `{PREFIX}-N <conventional commit>` title
2. **Post-merge triggers** — the merge push triggers `post-merge.yml`
3. **detect-and-configure** — detects release commit, validates commit
   message, and ensures branch protection/labels
4. **release** (step in `release-and-maintain`) — `release.py` calculates
   the next version, updates files, runs tests, commits
   `release: vX.Y.Z [skip ci]`, creates tag `vX.Y.Z`, and pushes to master
5. **publish** (step in `release-and-maintain`) — `publish.py` builds the
   package, publishes to the Gitea PyPI registry, and creates a Gitea
   release with git-cliff notes (checks out the release tag within the
   same job)
6. **sync-wiki** (step in `release-and-maintain`) — documentation is synced
   to the Gitea wiki
7. **vikunja** (step in `release-and-maintain`) — the corresponding Vikunja
   task is marked as done
8. **badges** (step in `release-and-maintain`) — quality badges are
   regenerated and pushed to the `badges` branch; README and docs/index.md
   are updated with cache-busting URLs

The release commit's post-merge run skips all steps except `badges` (which
picks up the new version number). This prevents infinite loops.

## Failure handling

Every job in the CI and post-merge workflows has a `notify_failure` step
that runs `if: failure()`. This creates a Gitea issue with the workflow name,
run ID, and commit SHA, ensuring failures that would otherwise go unnoticed
in the Actions tab are surfaced as issues. The issue is created via the tea
CLI with a `bug` label if available.