GRM-64: refactor: migrate from scripts/ to devx package
This commit is contained in:
@@ -17,10 +17,11 @@ make workflow-check # workflow-lint + workflow-dryrun
|
||||
```
|
||||
|
||||
`make setup` automatically installs all development tools:
|
||||
- **Python deps** via `scripts/setup.py` (pip install -e .[dev], ansible-galaxy, pre-commit hooks)
|
||||
- **checkmake** via `scripts/install_checkmake.py` (Makefile linter)
|
||||
- **actionlint, git-cliff, act_runner, tea** via `scripts/install_tools.py` (CI/CD tools to ~/.local/bin)
|
||||
- **tea CLI login** via `scripts/setup.py` (configures `tea login` from `.env` `REPO_TOKEN`)
|
||||
- **Python deps** via `devx.tools.setup` (pip install -e .[dev], ansible-galaxy, pre-commit hooks)
|
||||
- **devx package** via `make install-devx` (installs the devx package from git, providing all CI/CD tools)
|
||||
- **checkmake** via `devx.tools.install_checkmake` (Makefile linter)
|
||||
- **actionlint, git-cliff, act_runner, tea** via `devx.tools.install_tools` (CI/CD tools to ~/.local/bin)
|
||||
- **tea CLI login** via `devx.tools.setup` (configures `tea login` from `.env` `REPO_TOKEN`)
|
||||
|
||||
## Workflow Verification (Before Push)
|
||||
|
||||
@@ -29,7 +30,7 @@ Workflow YAML files (`.gitea/workflows/*.yml`) are verified with two tools:
|
||||
1. **actionlint** — Static linter that catches syntax errors, invalid
|
||||
expressions, unknown keys, type mismatches, and shellcheck issues.
|
||||
Config: `.gitea/actionlint.yaml` (registers custom `docker` runner label).
|
||||
Installed automatically by `make setup` via `scripts/install_tools.py`.
|
||||
Installed automatically by `make setup` via `devx.tools.install_tools`.
|
||||
|
||||
2. **act_runner exec --dryrun** — Gitea's own runner in dry-run mode.
|
||||
Validates job dependencies, step ordering, and Docker image selection
|
||||
@@ -44,7 +45,7 @@ CI also runs a best-effort `make workflow-dryrun` step (skipped if act_runner is
|
||||
|
||||
- **Python CLI** (`src/gitea_runner_manager/`) — Click-based CLI that delegates to Ansible
|
||||
- **Ansible Role** (`ansible/roles/gitea-runner/`) — Idempotent role for rootless Docker runner setup
|
||||
- **CI Scripts** (`scripts/`) — Automation for auto-merge, post-merge, release, publishing, molecule distribution, PR reviews, failure notifications
|
||||
- **devx package** (installed from git) — Reusable CI/CD tools: auto-merge, post-merge, release, publishing, molecule distribution, PR reviews, failure notifications
|
||||
- **Versioning** (`cliff.toml`) — git-cliff configuration for automated semver versioning from conventional commits
|
||||
|
||||
## PR Workflow (Mandatory)
|
||||
@@ -54,7 +55,8 @@ Every change to master goes through this workflow. No exceptions.
|
||||
### Branch Protection (Required Gitea Settings)
|
||||
|
||||
Branch protection and labels are automatically configured by
|
||||
`scripts/configure_repo.py`, which runs as a `configure-repo` job in
|
||||
`devx.tools.configure_repo` (run as `python -m devx.tools.configure_repo`),
|
||||
which runs as a `configure-repo` job in
|
||||
the post-merge workflow on every push to master.
|
||||
|
||||
The following rules are enforced for `master`:
|
||||
@@ -102,7 +104,7 @@ UX, documentation, workflow compliance, maintainability, resource
|
||||
management, backwards compatibility, and logging.
|
||||
|
||||
**Automated review (CI `pr-review` job):** Every PR triggers an automated
|
||||
review via `scripts/ci/pr_review.py`. This job posts a review with
|
||||
review via `devx.ci.pr_review` (run as `python -m devx.ci.pr_review`). This job posts a review with
|
||||
`COMMENT` (no issues) or `REQUEST_CHANGES` (issues found) based on
|
||||
the **[auto]** items in the checklist:
|
||||
|
||||
@@ -125,9 +127,9 @@ must go through **every category** in `REVIEW_CHECKLIST.md` and verify
|
||||
the **[manual]** items by reviewing the full diff
|
||||
(`git diff master...HEAD`).
|
||||
|
||||
Post review comments using `scripts/ci/review_pr.py`:
|
||||
Post review comments using `devx.ci.review_pr` (run as `python -m devx.ci.review_pr`):
|
||||
```bash
|
||||
REPO_TOKEN=<token> python3 scripts/ci/review_pr.py <pr_number> <owner/repo> \
|
||||
REPO_TOKEN=<token> python -m devx.ci.review_pr <pr_number> <owner/repo> \
|
||||
--event REQUEST_CHANGES \
|
||||
--body "Review summary" \
|
||||
--comments-json comments.json
|
||||
@@ -140,7 +142,7 @@ Fix each comment one by one, commit, and push. Re-review until satisfied.
|
||||
Once all checklist items are verified and comments are addressed, post
|
||||
an approval review with `--checklist-confirmed` and `--checklist-categories`:
|
||||
```bash
|
||||
REPO_TOKEN=<token> python3 scripts/ci/review_pr.py <pr_number> <owner/repo> \
|
||||
REPO_TOKEN=<token> python -m devx.ci.review_pr <pr_number> <owner/repo> \
|
||||
--event APPROVE --checklist-confirmed \
|
||||
--checklist-categories 1,2,3,4,5,6,7,8,9,10,11,12,13 \
|
||||
--body "All 13 REVIEW_CHECKLIST.md categories verified. Architecture: <summary>. Security: <summary>. Tests: <summary>. Docs: <summary>."
|
||||
@@ -179,7 +181,7 @@ test infrastructure flakiness.
|
||||
### Dynamic Runner Discovery
|
||||
|
||||
Molecule tests are distributed across available Gitea Actions runners
|
||||
dynamically via `scripts/ci/discover_runners.py`. The `discover-runners`
|
||||
dynamically via `devx.molecule.discover_runners`. The `discover-runners`
|
||||
job queries the Gitea API for runners at all levels (repo, org, instance)
|
||||
and generates a dynamic matrix. If the API can't see instance-level runners
|
||||
(no admin scope), it falls back to the `MOLECULE_RUNNERS` repo variable,
|
||||
@@ -201,9 +203,9 @@ Vikunja task updates:
|
||||
release commit (`release: vX.Y.Z`). All subsequent jobs skip for
|
||||
release commits (the `[skip ci]` tag also prevents re-triggering).
|
||||
|
||||
2. **release** — Runs `scripts/ci/release.py` which:
|
||||
- **Checks for user-facing changes** via `scripts/ci/classify_changes.py` — if only
|
||||
workflow/infrastructure files changed (`.gitea/`, `scripts/`, `docs/`, `tests/`,
|
||||
2. **release** — Runs `devx.ci.release` which:
|
||||
- **Checks for user-facing changes** via `devx.ci.classify_changes` — if only
|
||||
workflow/infrastructure files changed (`.gitea/`, `docs/`, `tests/`,
|
||||
`AGENTS.md`, `Makefile`, etc.), the release is **skipped entirely** — no version
|
||||
bump, no tag, no publish. This prevents unnecessary releases for CI/docs-only changes.
|
||||
- Uses **git-cliff** to calculate the next semver version from conventional commits
|
||||
@@ -233,16 +235,15 @@ which builds and publishes the package to PyPI.
|
||||
### Smart CI: User-Facing vs Workflow-Only Changes
|
||||
|
||||
Not all changes require the full CI pipeline or a new release. The project
|
||||
classifies changes into two categories using `scripts/ci/classify_changes.py`:
|
||||
classifies changes into two categories using `devx.ci.classify_changes`:
|
||||
|
||||
**Classification strategy (safe-by-default):** Any file NOT in the explicit
|
||||
workflow-only allowlist is treated as user-facing. This prevents new file
|
||||
types from accidentally skipping releases.
|
||||
types from accidentally skipping releases. Classification is config-driven
|
||||
via `[tool.devx.classify]` in `pyproject.toml`.
|
||||
|
||||
**Workflow-only paths** (infrastructure → no release needed):
|
||||
- `.gitea/**` — Gitea Actions workflows
|
||||
- `scripts/ci/**` — CI/CD automation scripts
|
||||
- `scripts/setup.py`, `scripts/molecule_all.py`, `scripts/install_tools.py`, `scripts/__init__.py` — Dev tooling and package init
|
||||
- `docs/**` — Documentation
|
||||
- `tests/**` — Test files
|
||||
- `AGENTS.md`, `README.md`, `CHANGELOG.md`, `TROUBLESHOOTING.md` — Project docs
|
||||
@@ -256,9 +257,15 @@ types from accidentally skipping releases.
|
||||
- `pyproject.toml` — Package metadata
|
||||
- Any new file type not in the allowlist
|
||||
|
||||
**Script directory structure:**
|
||||
- `scripts/` — Dev tools (run locally by developers): `check_test_speed.py`, `configure_repo.py`, `install_checkmake.py`, `install_tools.py`, `setup.py`, `molecule_all.py`, `generate_badges.py`, `gitea_cli.py`
|
||||
- `scripts/ci/` — CI/CD automation (run by workflows): `release.py`, `publish.py`, `auto_merge.py`, `classify_changes.py`, `detect_release_commit.py`, `push_badges.py`, `doc_coverage.py`, `sync_wiki.py`, `distribute_molecule.py`, `molecule_ci_guard.py`, `discover_runners.py`, `notify_failure.py`, `post_merge.py`, `pr_review.py`, `review_pr.py`, `validate_commit_msg.py`, `platforms.py`
|
||||
**devx module structure** (installed from git, not in this repo):
|
||||
- `devx.ci.*` — CI/CD automation (run by workflows): release, publish, auto_merge, classify_changes, detect_release_commit, push_badges, doc_coverage, sync_wiki, distribute_molecule, molecule_ci_guard, discover_runners, notify_failure, post_merge, pr_review, review_pr, validate_commit_msg
|
||||
- `devx.tools.*` — Dev tools (run locally): check_test_speed, configure_repo, install_checkmake, install_tools, setup, generate_badges
|
||||
- `devx.molecule.*` — Molecule helpers: molecule_all, platforms, discover_runners, distribute_molecule, molecule_ci_guard
|
||||
- `devx.gitea_cli` — Tea CLI wrapper
|
||||
- `devx.i18n` — i18n translation system
|
||||
- `devx.config` — Shared configuration (DEVX_* env vars)
|
||||
- `devx.api_clients` — GiteaClient, VikunjaClient
|
||||
- `devx.exceptions` — APIError and other exceptions
|
||||
|
||||
**CI behavior based on classification:**
|
||||
- **Molecule tests**: Only run when `ansible/` or `.ansible-lint` files change
|
||||
@@ -269,80 +276,76 @@ types from accidentally skipping releases.
|
||||
**AI agents must follow these rules:**
|
||||
- When working on workflow/CI/docs-only changes, use `ci:` or `docs:` commit prefixes
|
||||
- Do NOT bump the version or create tags for workflow-only changes
|
||||
- The `classify_changes.py` script enforces this automatically — no manual intervention needed
|
||||
- When adding a new CI script, place it in `scripts/ci/`. Dev tools go in `scripts/`.
|
||||
- The `classify_changes` module enforces this automatically — no manual intervention needed
|
||||
|
||||
## Script Separation and Import Rules
|
||||
## Source Code Separation and devx Integration
|
||||
|
||||
The codebase enforces strict separation between the GRM tool and CI/dev scripts:
|
||||
The codebase enforces strict separation between the GRM tool and the devx package:
|
||||
|
||||
### Directory Layout
|
||||
|
||||
| Directory | Purpose | Release impact |
|
||||
|-----------|---------|----------------|
|
||||
| `src/gitea_runner_manager/` | User-facing GRM CLI tool | Changes trigger release |
|
||||
| `scripts/` | Dev tools (run locally) | Workflow-only (no release) |
|
||||
| `scripts/ci/` | CI/CD automation (run by workflows) | Workflow-only (no release) |
|
||||
| `devx` package (installed from git) | Reusable CI/CD and dev tools | Not in this repo (no release impact) |
|
||||
| `ansible/` | Ansible role for runner setup | Changes trigger release |
|
||||
|
||||
### Import Rules
|
||||
|
||||
1. **`src/gitea_runner_manager/` NEVER imports from `scripts/`** — the tool is self-contained
|
||||
2. **Scripts MAY import from `gitea_runner_manager`** — one-way dependency (scripts use the tool's API clients, config, i18n)
|
||||
3. **Cross-script imports** (scripts importing from other scripts) are allowed within `scripts/ci/` but must be documented
|
||||
4. **`scripts/gitea_cli.py`** is a shared wrapper around the `tea` CLI — CI scripts import from it for Gitea API operations (issues, labels, PRs, releases, reviews)
|
||||
1. **`src/gitea_runner_manager/` NEVER imports from devx** — the GRM tool is self-contained
|
||||
2. **devx MAY import from `gitea_runner_manager`** — one-way dependency (devx uses the tool's API clients, config, i18n)
|
||||
3. **Cross-module imports within devx** are allowed (devx modules importing from other devx modules) and must be documented
|
||||
4. **`devx.gitea_cli`** is a shared wrapper around the `tea` CLI — devx modules import from it for Gitea API operations (issues, labels, PRs, releases, reviews)
|
||||
|
||||
### tea CLI Integration
|
||||
|
||||
The `tea` Gitea CLI tool is used for Gitea API interactions in CI scripts. It is installed by `scripts/install_tools.py` and configured by `scripts/setup.py` (login profile from `.env` `REPO_TOKEN`).
|
||||
The `tea` Gitea CLI tool is used for Gitea API interactions in devx. It is installed by `devx.tools.install_tools` and configured by `devx.tools.setup` (login profile from `.env` `REPO_TOKEN`).
|
||||
|
||||
**`scripts/gitea_cli.py`** — Python wrapper around `tea` CLI with JSON output parsing:
|
||||
**`devx.gitea_cli`** — Python wrapper around `tea` CLI with JSON output parsing:
|
||||
- `TeaCLI.create_issue()` — Create issues with labels
|
||||
- `TeaCLI.list_labels()` / `TeaCLI.create_label()` / `TeaCLI.add_label()` — Label management
|
||||
- `TeaCLI.create_pr()` / `TeaCLI.merge_pr()` / `TeaCLI.review_pr()` — Pull request operations
|
||||
- `TeaCLI.create_release()` / `TeaCLI.list_releases()` — Release management
|
||||
- `TeaCLI.list_branches()` — Branch listing
|
||||
|
||||
**Scripts using tea (via `gitea_cli.py`):**
|
||||
- `scripts/ci/publish.py` — Creates Gitea releases via `tea releases create`
|
||||
- `scripts/ci/notify_failure.py` — Creates issues via `tea issues create` (falls back to `GiteaClient` if tea not installed)
|
||||
- `scripts/configure_repo.py` — Creates labels via `tea labels create` (falls back to `GiteaClient` if tea fails; branch protection still uses `GiteaClient` since tea only supports basic protect/unprotect)
|
||||
**Modules using tea (via `devx.gitea_cli`):**
|
||||
- `devx.ci.publish` — Creates Gitea releases via `tea releases create`
|
||||
- `devx.ci.notify_failure` — Creates issues via `tea issues create` (falls back to `GiteaClient` if tea not installed)
|
||||
- `devx.tools.configure_repo` — Creates labels via `tea labels create` (falls back to `GiteaClient` if tea fails; branch protection still uses `GiteaClient` since tea only supports basic protect/unprotect)
|
||||
|
||||
**Operations still using `GiteaClient` (not supported by tea):**
|
||||
- PR reviews (`review_pr.py`) — tea v0.14.1 only supports interactive reviews
|
||||
- Wiki page management (`sync_wiki.py`)
|
||||
- Commit status checks (`auto_merge.py`)
|
||||
- Runner discovery (`discover_runners.py`)
|
||||
- Branch protection with detailed config (`configure_repo.py`)
|
||||
- PR file/commit listing (`pr_review.py`)
|
||||
- PR reviews (`devx.ci.review_pr`) — tea v0.14.1 only supports interactive reviews
|
||||
- Wiki page management (`devx.ci.sync_wiki`)
|
||||
- Commit status checks (`devx.ci.auto_merge`)
|
||||
- Runner discovery (`devx.molecule.discover_runners`)
|
||||
- Branch protection with detailed config (`devx.tools.configure_repo`)
|
||||
- PR file/commit listing (`devx.ci.pr_review`)
|
||||
|
||||
### PYTHONPATH Configuration
|
||||
|
||||
Scripts have different import requirements. Workflows must set `PYTHONPATH` accordingly:
|
||||
Since devx is installed as a package (via `pip install` from git), it is importable directly. Workflows only need `PYTHONPATH=src` when a devx module imports from `gitea_runner_manager`:
|
||||
|
||||
| PYTHONPATH | When to use | Example scripts |
|
||||
| PYTHONPATH | When to use | Example modules |
|
||||
|------------|-------------|-----------------|
|
||||
| `src` | Script imports from `gitea_runner_manager` | `auto_merge.py`, `pr_review.py`, `review_pr.py`, `sync_wiki.py`, `post_merge.py`, `classify_changes.py`, `discover_runners.py`, `doc_coverage.py` |
|
||||
| `.:src` | Script imports from both `gitea_runner_manager` and `scripts.gitea_cli` | `publish.py`, `notify_failure.py`, `configure_repo.py` |
|
||||
| `.` | Script imports from other `scripts.ci.*` modules | `release.py` (imports `classify_changes.has_user_facing_changes`) |
|
||||
| (none) | Script has no GRM or cross-script imports | `detect_release_commit.py`, `distribute_molecule.py`, `molecule_ci_guard.py`, `push_badges.py`, `validate_commit_msg.py` |
|
||||
| `src` | Module imports from `gitea_runner_manager` | `devx.ci.auto_merge`, `devx.ci.pr_review`, `devx.ci.review_pr`, `devx.ci.sync_wiki`, `devx.ci.post_merge`, `devx.ci.classify_changes`, `devx.molecule.discover_runners`, `devx.ci.doc_coverage` |
|
||||
| (none) | Module has no GRM imports | `devx.ci.detect_release_commit`, `devx.molecule.distribute_molecule`, `devx.molecule.molecule_ci_guard`, `devx.ci.push_badges`, `devx.ci.validate_commit_msg` |
|
||||
|
||||
**In workflows**, always use `env:` blocks (not inline `PYTHONPATH=value`):
|
||||
```yaml
|
||||
- name: Run script
|
||||
- name: Run module
|
||||
env:
|
||||
PYTHONPATH: src
|
||||
run: python3 scripts/ci/example.py
|
||||
run: python -m devx.ci.example
|
||||
```
|
||||
|
||||
**Locally**, the current directory is in `sys.path` by default, so `PYTHONPATH` is usually not needed.
|
||||
**Locally**, devx is installed as a package, so only `PYTHONPATH=src` is needed if importing from `gitea_runner_manager`.
|
||||
|
||||
### Shared Constants
|
||||
|
||||
`scripts/ci/platforms.py` is the single source of truth for the molecule
|
||||
platform matrix. Both `scripts/ci/distribute_molecule.py` (CI) and
|
||||
`scripts/molecule_all.py` (dev tool) import `PLATFORMS` from it — this
|
||||
avoids dev tools importing directly from CI scripts.
|
||||
`devx.molecule.platforms` is the single source of truth for the molecule
|
||||
platform matrix. Both `devx.molecule.distribute_molecule` (CI) and
|
||||
`devx.molecule.molecule_all` (dev tool) import `PLATFORMS` from it — this
|
||||
avoids dev tools importing directly from CI modules.
|
||||
|
||||
2. **Publish workflow** (`.gitea/workflows/publish.yml`):
|
||||
- Triggers on tag push (`v*`)
|
||||
@@ -350,7 +353,7 @@ avoids dev tools importing directly from CI scripts.
|
||||
- Builds the Python package
|
||||
- Optionally publishes to PyPI (if `PYPI_TOKEN` is set)
|
||||
- Creates a Gitea release with git-cliff-generated release notes
|
||||
- On failure, creates a Gitea issue via `scripts/ci/notify_failure.py`
|
||||
- On failure, creates a Gitea issue via `devx.ci.notify_failure`
|
||||
|
||||
### git-cliff Commit Preprocessing
|
||||
|
||||
@@ -379,6 +382,15 @@ The version source is `__version__` in `src/gitea_runner_manager/__init__.py`, r
|
||||
| PR title | `GRM-N: <vikunja task title>` | `GRM-33: Add mandatory PR review step` |
|
||||
| Merge commit | `GRM-N <conventional commit>` | `GRM-33 feat: add review script` |
|
||||
|
||||
### Configuration
|
||||
|
||||
The devx package is configured via `DEVX_*` environment variables:
|
||||
- `DEVX_TASK_PREFIX=GRM` — Prefix for Vikunja task identifiers
|
||||
- `DEVX_VIKUNJA_PROJECT_ID=6` — Vikunja project ID for task tracking
|
||||
- `DEVX_VERSION_FILE=src/gitea_runner_manager/__init__.py` — Path to the version source file
|
||||
|
||||
Change classification is config-driven via `[tool.devx.classify]` in `pyproject.toml`, which defines the workflow-only and user-facing path patterns.
|
||||
|
||||
## Key Conventions
|
||||
|
||||
- Python 3.12+ required (ruff/pyright target `py312`)
|
||||
@@ -404,7 +416,7 @@ main.yml → systemd_check → user_setup → rootless_docker → install_runner
|
||||
|
||||
6 scenarios: `default`, `multi-instance`, `lifecycle`, `template-content`, `deregister`, `update`
|
||||
4 platforms: `ubuntu-2204`, `ubuntu-2404`, `debian-12`, `archlinux`
|
||||
Platform list is defined in `scripts/ci/platforms.py` (single source of truth)
|
||||
Platform list is defined in `devx.molecule.platforms` (single source of truth)
|
||||
|
||||
## Known Issues
|
||||
|
||||
@@ -438,14 +450,14 @@ docs/
|
||||
|
||||
### Wiki Sync
|
||||
|
||||
- **On merge to master**: `sync-wiki.yml` workflow runs `scripts/ci/sync_wiki.py` which pushes all `/docs/` content to the Gitea wiki via API
|
||||
- **On merge to master**: `sync-wiki.yml` workflow runs `devx.ci.sync_wiki` which pushes all `/docs/` content to the Gitea wiki via API
|
||||
- **On release tag**: Same sync runs, plus the wiki is tagged with the release version
|
||||
- `mapping.json` maps each file path to a wiki page title (e.g., `user/getting-started.md` → `Getting-Started`)
|
||||
- README.md is a lean entry point with links to the wiki — no detailed content
|
||||
|
||||
### Documentation Coverage
|
||||
|
||||
- `scripts/ci/doc_coverage.py` checks that all CLI commands, Python modules, and CI scripts are documented
|
||||
- `devx.ci.doc_coverage` checks that all CLI commands, Python modules, and CI scripts are documented
|
||||
- Runs as a CI step in the quality job with `--fail-on-missing` (blocks CI if docs are missing)
|
||||
- Enforced: 100% coverage for public CLI commands and major architectural components
|
||||
|
||||
|
||||
Reference in New Issue
Block a user