devx/docs/tech/ci-cd-workflow.md

# CI/CD Workflow

devx uses Gitea Actions for CI/CD automation. Three workflows implement a
complete pipeline: pull request validation, post-merge release automation, and
tag-triggered publishing.

## Workflow overview

```
PR opened/synchronized ──► CI (ci.yml)
    │                         ├── quality
    │                         ├── detect-changes
    │                         ├── release-dry-run (if user-facing)
    │                         ├── pr-review
    │                         └── auto-merge ──► squash-merge to master
    │                                              │
    ▼                                              ▼
Push to master ──► Post-merge (post-merge.yml)
                     ├── detect-type
                     ├── validate-commit-msg
                     ├── release ──► tag vX.Y.Z
                     ├── sync-wiki        │
                     ├── badges           │
                     ├── vikunja          │
                     └── configure-repo   │
                                          │
                                          ▼
Tag push (v*) ──► Publish (publish.yml)
                     └── publish ──► Gitea PyPI registry + Gitea release
```

## CI workflow (`ci.yml`)

Runs on pull requests (opened and synchronize) and manual dispatch.

### Jobs

#### `quality`

The main quality gate. Runs on every PR:

1. **Lint all** — ruff check, ruff format check, pyright, bandit, actionlint
   (via `make lint-all`)
2. **Unit tests with 100% coverage** — `make pytest-cov`
3. **Check unit test speed** — `python -m devx.tools.check_test_speed
   --max-seconds 4 --max-single-seconds 0.5`
4. **Documentation coverage check** — `python -m devx.ci.doc_coverage
   --fail-on-missing`
5. **Translation completeness check** — `python -m devx.ci.check_translations`
6. **Dependency security scan** — `pip-audit --desc --skip-editable`
   (best-effort, non-blocking)
7. **Workflow dry-run validation** — `make workflow-dryrun` via act_runner
   (best-effort, skipped if act_runner is not installed)

#### `detect-changes`

Classifies changes between `origin/master` and the PR head as user-facing or
workflow-only using `python -m devx.ci.classify_changes --github-output`.
Writes `user-facing-changed=true|false` to the job output for use by
downstream jobs.

#### `release-dry-run`

Depends on `quality` and `detect-changes`. Only runs if user-facing changes
are detected. Runs `python -m devx.ci.release --dry-run` to validate that
the release script can calculate the next version and generate the changelog
without making changes. Non-blocking (uses `|| true`).

#### `pr-review`

Runs on every pull request. Executes `python -m devx.ci.pr_review` with the
PR number and repository. Fetches the PR diff via the Gitea API and runs
automated checks, posting a structured review:

- `COMMENT` — no issues found
- `REQUEST_CHANGES` — issues found that must be addressed

Checks performed:
1. Architecture compliance — no subprocess in CLI, no hardcoded URLs
2. Best practices — no `print()`, no bare `except`, no `TODO`/`FIXME`,
   no functions > 50 lines
3. Security — no hardcoded secrets, no `shell=True`, no `eval`/`exec`
4. i18n — no raw strings in `click.echo()` without `_()` wrapper
5. Resource management — no `open()` without `with`, no `Popen()` without
   cleanup
6. Documentation — source changes must include doc updates
7. Test coverage — source changes must include test updates
8. Commit conventions — conventional commit format on PR commits

#### `auto-merge`

Depends on `quality`, `detect-changes`, and `pr-review`. The final job in the
CI workflow. Runs `python -m devx.ci.auto_merge` with the branch name, PR
title, repository, and PR number:

1. **Read task ID** from branch name (e.g., `DEVX-12-fix-foo` → `DEVX-12`)
2. **Validate PR title format** — must be `{PREFIX}-N: <vikunja task title>`
3. **Validate PR title matches Vikunja task** — fetches the Vikunja task and
   compares the title
4. **Extract conventional commit message** from PR commits (newest matching
   conventional format)
5. **Squash-merge** with title `{PREFIX}-N <conventional commit message>`
6. If the head branch is behind master (HTTP 405), automatically pulls master,
   rebases, force-pushes, and retries the merge

The merge commit push to master triggers the post-merge workflow.

### Smart CI: user-facing vs workflow-only changes

Not all changes require a new release. The `detect-changes` job classifies
changes using `python -m devx.ci.classify_changes`:

**Workflow-only paths** (infrastructure — no release needed):
- `.gitea/**` — Gitea Actions workflows
- `tests/**` — Test files
- `AGENTS.md`, `README.md`, `CHANGELOG.md` — Project docs
- `Makefile`, `cliff.toml`, `.pre-commit-config.yaml` — Config
- `.env.example`, `.gitignore` — Config
- `hooks/**` — Git hooks
- `src/devx/__init__.py` — Only contains `__version__` (release artifact)

**User-facing paths** (tool changes — release needed) — everything else:
- `src/devx/**` — Python package source (except `__init__.py`)
- `pyproject.toml` — Package metadata
- Any new file type not in the allowlist

Classification is configured in `pyproject.toml` under
`[tool.devx.classify]`. The framework provides `DEFAULT_INFRASTRUCTURE` — a
curated list of paths that are infrastructure for any Python project. Projects
inherit these automatically and only specify what is different.

Rule priority (first match wins):
1. `user_facing_overrides` — safety override (highest priority)
2. `infrastructure_overrides` — explicit per-file
3. `infrastructure` — DEFAULT_INFRASTRUCTURE + project-specific patterns
4. Default: user-facing (safe — any unknown file triggers release)

## Post-merge workflow (`post-merge.yml`)

Runs on every push to master. A single workflow with conditional jobs
replaces separate workflows for release, wiki sync, badges, and Vikunja task
updates.

### Job dependency graph

```
detect-type ──┬── validate-commit-msg (skip if release commit)
              ├── release (skip if release commit)
              │       │
              │       ├── sync-wiki (needs release)
              │       ├── badges (needs release, ALWAYS runs)
              │       └── vikunja (needs release)
              └── configure-repo (independent, skip if release commit)
```

`sync-wiki` and `vikunja` depend on `release` succeeding so that the wiki and
task tracker are only updated when the code is actually released. If release
fails, they are skipped to avoid leaving the wiki or Vikunja in an
inconsistent state.

The `badges` job uses `if: always()` with no is-release condition so it runs
on every push to master, including release commits. This ensures badges
(tests, coverage, version, etc.) are always current.

When `release` creates a `release: vX.Y.Z` commit, the release commit's
post-merge run still updates badges (the version badge picks up the new
version). Other jobs skip. The tag push triggers `publish.yml`.

### Jobs

#### `detect-type`

Checks if the latest commit is a release commit (`release: vX.Y.Z [skip ci]`)
using `python -m devx.ci.detect_release_commit`. Writes `is-release=true` or
`is-release=false` to the job output. All subsequent jobs use this to
conditionally skip for release commits.

#### `validate-commit-msg`

Depends on `detect-type`. Skips for release commits. Validates the latest
commit message using `python -m devx.ci.validate_commit_msg --branch master`.
On master, commits must follow `{PREFIX}-N: <conventional commit>` format
(added by auto-merge).

#### `release`

Depends on `detect-type`. Skips for release commits. The core release
automation job. Runs `python -m devx.ci.release`:

1. **Classify changes** — calls `classify_changes.py` to check for user-facing
   changes. If only infrastructure files changed, exits without releasing.
2. **Calculate next version** — uses git-cliff to determine the next semver
   version from conventional commits since the last tag
3. **Update version file** — updates `__version__` in `src/devx/__init__.py`
4. **Update changelog** — prepends the new version section to `CHANGELOG.md`
   using git-cliff output
5. **Run tests** — executes `make lint-ruff` and `make pytest-cov` to verify
   the release is healthy. If either fails, the release is aborted — no
   commit, no tag. Use `--skip-tests` only for emergency releases.
6. **Commit** — stages the version file and changelog, commits with
   `release: vX.Y.Z [skip ci]` (uses `--no-verify` to bypass the commit-msg
   hook since release commits are a special case)
7. **Create tag** — creates an annotated tag `vX.Y.Z` with the changelog as
   the tag message
8. **Push** — pushes both the commit and tag to master

The script is idempotent: if there are no new conventional commits since the
last tag, it exits without doing anything. If the tag already exists (e.g.,
from a partial previous run), it skips tag creation and only pushes.

**Tag consistency**: Before releasing, the script fetches remote tags and
verifies all existing tags point to commits whose message matches the tag
version. This prevents duplicate release commits and ensures
tag/version/commit alignment.

**Version bumping rules** (git-cliff):

| Commit type | Version bump |
|-------------|-------------|
| `feat:` | minor (0.X.0) |
| `fix:` | patch (0.0.X) |
| `feat!:` or `BREAKING CHANGE` | minor (pre-1.0) |
| `chore:`, `ci:`, `docs:` | no bump (excluded by cliff.toml) |

On failure, the `notify_failure` step creates a Gitea issue via
`python -m devx.ci.notify_failure`.

#### `sync-wiki`

Depends on `detect-type` and `release`. Skips for release commits. Syncs
documentation from `docs/` to the Gitea wiki using
`python -m devx.ci.sync_wiki --repo <owner/repo> --strict`:

1. Reads `docs/mapping.json` to map file paths to wiki page titles
2. Lists existing wiki pages via the Gitea API
3. For each mapped file, reads content and creates or updates the wiki page
4. `--strict` runs a full integrity check: verifies page count, missing
   pages, stale pages, and content match. Fails if any page is empty or
   content doesn't match.

Pages that exist in the wiki but not in the mapping are left untouched (not
deleted).

On failure, the `notify_failure` step creates a Gitea issue.

#### `badges`

Depends on `detect-type` and `release`. Uses `if: always()` so it runs on
every push to master, including release commits. Generates and pushes quality
badges using `python -m devx.ci.push_badges`:

1. **Fetch latest master** — `git fetch origin master && git reset --hard
   origin/master` (ensures the version badge reflects the current state,
   even if the release job just pushed a new version)
2. **Generate badges** — calls `devx.tools.generate_badges` which runs
   pytest-cov, doc-coverage, lint checks, and version extraction, then writes
   SVG files: `coverage.svg`, `tests.svg`, `docs.svg`, `quality.svg`,
   `version.svg`, `python.svg`
3. **Push to badges branch** — creates an orphan `badges` branch, copies SVG
   files, commits, and force-pushes
4. **Update README/docs** — switches back to master, replaces
   `raw/branch/badges/<name>.svg` URLs with `raw/commit/<sha>/<name>.svg`
   URLs (cache-busting — Gitea caches `raw/branch/` URLs for 6 hours),
   commits, and pushes

Supports `--retries` for retrying on git push failures (fetches latest master
and waits 10s between attempts).

On failure, the `notify_failure` step creates a Gitea issue.

#### `vikunja`

Depends on `detect-type` and `release`. Skips for release commits. Updates
the Vikunja task after a merge using `python -m devx.ci.post_merge --git-sha
<sha>`:

1. Extracts the task ID from the first line of the commit message
2. Marks the corresponding Vikunja task as done
3. Posts a comment with the merge SHA

On failure, the `notify_failure` step creates a Gitea issue.

#### `configure-repo`

Depends on `detect-type`. Skips for release commits. Ensures branch
protection and labels are configured using
`python -m devx.tools.configure_repo --repo <name> --owner <owner>`:

- Sets up master branch protection (required status checks, block on rejected
  reviews, block on outdated branch)
- Creates standard labels
- Status check contexts read from `DEVX_STATUS_CHECKS` or default to
  `CI / quality (pull_request)`

On failure, the `notify_failure` step creates a Gitea issue.

## Publish workflow (`publish.yml`)

Runs on tag pushes matching `v*`. Triggered by the `release` job in the
post-merge workflow when it creates and pushes a new version tag.

### Job: `publish`

1. **Install dependencies** — build, twine, requests, python-dotenv, click,
   and the project itself
2. **Install CI tools** — git-cliff and tea via
   `python -m devx.tools.install_tools`
3. **Configure tea login** — `tea login add` using `REPO_TOKEN`
4. **Build and publish** — `python -m devx.ci.publish <tag> <owner/repo>`:
   - Build the package with `python -m build`
   - Publish to the Gitea PyPI registry (default) using `twine upload
     --repository-url <url> -u <token> -p <token>`
   - OR publish to standard PyPI if `PYPI_TOKEN` is set
   - OR skip publishing if `--skip-build` is passed (non-Python repos)
   - Create a Gitea release with git-cliff-generated release notes via
     `tea create release`

Publishing destination resolution (checked in order):
1. **Gitea PyPI registry** — if `--registry-url` is given, or
   `DEVX_PYPI_REGISTRY_URL` env var is set, or derived from `GITEA_API_URL`
2. **Standard PyPI** — if `PYPI_TOKEN` is set (takes precedence over Gitea
   registry)
3. **Skip** — if neither is configured, only the Gitea release is created

On failure, the `notify_failure` step creates a Gitea issue.

## CI scripts

### `auto_merge.py`

Auto-merge PR when all CI checks pass. Reads task ID from the branch name
(e.g., `DEVX-12-fix-foo` → `DEVX-12`). Validates PR title format, checks the
Vikunja task exists and the title matches, extracts the conventional commit
message from PR commits, and squash-merges with
`{PREFIX}-N <conventional commit>` title.

```bash
python -m devx.ci.auto_merge <branch> <pr_title> <owner/repo> <pr_number>
```

### `release.py`

Automated release using git-cliff. Calculates next semver version from
conventional commits, updates `__version__` and `CHANGELOG.md`, runs lint and
tests, commits with `release: vX.Y.Z [skip ci]`, creates annotated tag, and
pushes. Idempotent — exits if no unreleased changes.

```bash
python -m devx.ci.release [--dry-run] [--skip-tests] [--verify]
```

- `--dry-run` — preview without making changes
- `--skip-tests` — skip lint and test verification (emergency only)
- `--verify` — check tag/version/changelog alignment and exit

### `publish.py`

Builds package, publishes to Gitea PyPI registry or standard PyPI, and
creates a Gitea release with git-cliff-generated notes.

```bash
python -m devx.ci.publish <tag> <owner/repo> [--registry-url <url>] [--skip-build]
```

### `pr_review.py`

Automated PR review. Fetches the PR diff via the Gitea API, runs automated
checks (architecture, best practices, security, i18n, resource management,
documentation, test coverage, commit conventions), and posts a structured
review with inline comments.

```bash
python -m devx.ci.pr_review <pr_number> <owner/repo>
```

### `notify_failure.py`

Creates a Gitea issue when a CI workflow fails. Uses the tea CLI for issue
creation with failure labels. Supports `--auto-login` to configure the tea
CLI login profile from `REPO_TOKEN`.

```bash
python -m devx.ci.notify_failure --repo <owner/repo> --run-id <id> \
  --workflow <name> --commit <sha> [--auto-login]
```

### `post_merge.py`

Updates Vikunja task after a merge to master. Extracts task ID from the
commit message, marks the task as done, and posts a comment with the merge SHA.

```bash
python -m devx.ci.post_merge <commit_msg> [--commit-sha <sha>] [--git-sha <sha>]
```

### `classify_changes.py`

Classifies git changes as user-facing or workflow-only. Uses a layered rule
system configured in `pyproject.toml`. Safe-by-default: any unknown file
defaults to user-facing.

```bash
python -m devx.ci.classify_changes [--base <ref>] [--head <ref>] \
  [--quiet] [--check <category>] [--github-output]
```

### `discover_runners.py`

Discovers available Gitea Actions runners at repository, organization, and
instance levels. Falls back to `MOLECULE_RUNNERS` repo variable or
`DEFAULT_MAX_RUNNERS` (3).

```bash
python -m devx.ci.discover_runners --owner <owner> --repo <repo> [--count] [--indices]
```

### `detect_release_commit.py`

Detects whether the latest git commit is a release commit. Writes
`is-release=true|false` to `$GITHUB_OUTPUT`.

```bash
python -m devx.ci.detect_release_commit
```

### `push_badges.py`

Generates SVG badge files, pushes them to the `badges` branch, and updates
README.md and docs/index.md with cache-busting `raw/commit/<sha>/` URLs.

```bash
python -m devx.ci.push_badges [--output-dir <dir>] [--branch <branch>] \
  [--no-readme-update] [--retries <n>]
```

### `distribute_molecule.py`

Distributes molecule (scenario, platform) pairs across N parallel runners.
Discovers scenarios under `ansible/roles/*/molecule/`.

```bash
python -m devx.molecule.distribute_molecule --runner-index <i> --max-runners <n>
python -m devx.molecule.distribute_molecule --list
python -m devx.molecule.distribute_molecule --list-platforms
```

### `molecule_ci_guard.py`

Runs molecule tests sequentially while polling the Gitea API for other runner
failures. Aborts early if another runner fails the same job.

```bash
python -m devx.molecule.molecule_ci_guard [--roles-root <dir>] pair1 pair2 ...
```

### `validate_commit_msg.py`

Validates commit messages. On feature branches: conventional commits only
(no `{PREFIX}-N` prefix). On master: must have `{PREFIX}-N` prefix from
auto-merge, followed by a conventional commit message.

```bash
python -m devx.ci.validate_commit_msg <commit_msg_file> [--branch <branch>]
```

### `sync_wiki.py`

Syncs documentation from `docs/` to the Gitea wiki via the API. Reads
`docs/mapping.json` for file-to-page mapping. Supports `--dry-run`,
`--verify`, and `--strict` (full integrity check).

```bash
python -m devx.ci.sync_wiki [--dry-run] [--repo <owner/repo>] [--verify] [--strict]
```

### `check_translations.py`

Validates translation files against the Python source code. Checks for
missing keys, dead keys, and missing languages.

```bash
python -m devx.ci.check_translations [--translations <file>]...
```

### `doc_coverage.py`

Checks documentation coverage for CLI commands and major modules. Parses
Click commands from `cli.py` and verifies documentation exists.

```bash
python -m devx.ci.doc_coverage [--docs-dir <dir>] [--fail-on-missing]
```

### `distribute_files.py`

Distributes files matching a glob pattern across N parallel runners
(round-robin). Writes the assigned file list to `$GITHUB_ENV`.

```bash
python -m devx.ci.distribute_files --pattern <glob> --runner-index <i> \
  --max-runners <n> [--github-env] [--skip-if-excess]
```

### `integration_guard.py`

Runs pytest with cross-runner failure detection. If any other
integration-tests matrix runner reports failure, the current pytest
subprocess is killed and this runner exits early.

```bash
python -m devx.ci.integration_guard -- <pytest args>
```

## Release process summary

The complete release process from PR to published package:

1. **PR merged** — `auto-merge` squash-merges the PR to master with
   `{PREFIX}-N <conventional commit>` title
2. **Post-merge triggers** — the merge push triggers `post-merge.yml`
3. **detect-type** — confirms the commit is not a release commit
4. **release** — `release.py` calculates the next version, updates files,
   runs tests, commits `release: vX.Y.Z [skip ci]`, creates tag `vX.Y.Z`,
   and pushes to master
5. **Tag push triggers publish** — the tag push triggers `publish.yml`
6. **publish** — `publish.py` builds the package, publishes to the Gitea PyPI
   registry, and creates a Gitea release with git-cliff notes
7. **sync-wiki** — documentation is synced to the Gitea wiki
8. **badges** — quality badges are regenerated and pushed to the `badges`
   branch; README and docs/index.md are updated with cache-busting URLs
9. **vikunja** — the corresponding Vikunja task is marked as done
10. **configure-repo** — branch protection and labels are ensured

The release commit's post-merge run skips all jobs except `badges` (which
picks up the new version number). This prevents infinite loops.

## Failure handling

Every job in the post-merge and publish workflows has a `notify_failure` step
that runs `if: failure()`. This creates a Gitea issue with the workflow name,
run ID, and commit SHA, ensuring failures that would otherwise go unnoticed
in the Actions tab are surfaced as issues. The issue is created via the tea
CLI with a `bug` label if available.