GRM-64: refactor: migrate from scripts/ to devx package
This commit is contained in:
@@ -78,7 +78,7 @@ flowchart TD
|
||||
|
||||
From `AGENTS.md`, the project also includes:
|
||||
|
||||
- **CI Scripts** (`scripts/`) — Automation for auto-merge, post-merge, release, publishing, molecule distribution, PR reviews, failure notifications
|
||||
- **devx package** (installed from git) — Reusable CI/CD tools: auto-merge, post-merge, release, publishing, molecule distribution, PR reviews, failure notifications
|
||||
- **Versioning** (`cliff.toml`) — git-cliff configuration for automated semver versioning from conventional commits
|
||||
|
||||
## Python Modules
|
||||
|
||||
+24
-24
@@ -60,10 +60,10 @@ Review the full diff (`git diff master...HEAD`) focusing on:
|
||||
- **User experience**: Clear error messages, intuitive CLI flags, helpful output
|
||||
- **Documentation**: Completeness and relevance of docs, CHANGELOG entries, AGENTS.md updates
|
||||
|
||||
Post review comments using `scripts/ci/review_pr.py`:
|
||||
Post review comments using `devx.ci.review_pr`:
|
||||
|
||||
```bash
|
||||
REPO_TOKEN=<token> python3 scripts/ci/review_pr.py <pr_number> <owner/repo> \
|
||||
REPO_TOKEN=<token> python -m devx.ci.review_pr <pr_number> <owner/repo> \
|
||||
--event REQUEST_CHANGES \
|
||||
--body "Review summary" \
|
||||
--comments-json comments.json
|
||||
@@ -78,7 +78,7 @@ Fix each comment one by one, commit, and push. Re-review until satisfied.
|
||||
Once all comments are addressed:
|
||||
|
||||
```bash
|
||||
REPO_TOKEN=<token> python3 scripts/ci/review_pr.py <pr_number> <owner/repo> \
|
||||
REPO_TOKEN=<token> python -m devx.ci.review_pr <pr_number> <owner/repo> \
|
||||
--event APPROVE \
|
||||
--body "All comments addressed. LGTM."
|
||||
```
|
||||
@@ -96,7 +96,7 @@ Then add the `ready-to-merge` label. The auto-merge workflow will:
|
||||
|
||||
After the squash-merge:
|
||||
|
||||
- The **post-merge workflow** (`.gitea/workflows/post-merge.yml`) triggers on push to `master` and runs `scripts/ci/post_merge.py` to mark the Vikunja task as done, extracting the task ID from the merge commit message.
|
||||
- The **post-merge workflow** (`.gitea/workflows/post-merge.yml`) triggers on push to `master` and runs `devx.ci.post_merge` to mark the Vikunja task as done, extracting the task ID from the merge commit message.
|
||||
- The **release workflow** (`.gitea/workflows/release.yml`) triggers on push to `master` and automatically versions, tags, and publishes (see below).
|
||||
|
||||
## Branch Protection (Required Gitea Settings)
|
||||
@@ -131,8 +131,8 @@ The `quality` job in `.gitea/workflows/ci.yml` runs:
|
||||
1. `make setup` — full environment setup
|
||||
2. `make lint-all` — ruff + pyright + bandit + ansible-lint + checkmake
|
||||
3. `make pytest-cov` — unit tests with 100% coverage enforcement
|
||||
4. `python3 scripts/check_test_speed.py --max-seconds 10` — verify unit tests run fast
|
||||
5. `PYTHONPATH=src python3 scripts/ci/release.py --dry-run` — release dry-run validation
|
||||
4. `python -m devx.tools.check_test_speed --max-seconds 10` — verify unit tests run fast
|
||||
5. `PYTHONPATH=src python -m devx.ci.release --dry-run` — release dry-run validation
|
||||
|
||||
## Automated Release Pipeline
|
||||
|
||||
@@ -144,7 +144,7 @@ After a PR is merged to master, the release pipeline runs automatically.
|
||||
- Sets up full dev environment (`make setup`) so lint and tests can run
|
||||
- Installs git-cliff (version 2.13.0)
|
||||
- Configures git as `grm-ci-bot`
|
||||
- Runs `scripts/ci/release.py` which uses **git-cliff** to:
|
||||
- Runs `devx.ci.release` which uses **git-cliff** to:
|
||||
- Calculate the next semver version from conventional commits since the last tag
|
||||
- Update `__version__` in `src/gitea_runner_manager/__init__.py` (single source of truth)
|
||||
- Update `CHANGELOG.md` with the new version section
|
||||
@@ -155,7 +155,7 @@ After a PR is merged to master, the release pipeline runs automatically.
|
||||
- Push both the commit and tag to master
|
||||
- `--skip-tests` flag bypasses test verification (emergency use only, not recommended)
|
||||
- Loops are prevented by `has_unreleased_changes` — after a release commit is tagged, the next run finds no unreleased changes and exits
|
||||
- On failure, creates a Gitea issue via `scripts/ci/notify_failure.py`
|
||||
- On failure, creates a Gitea issue via `devx.ci.notify_failure`
|
||||
|
||||
### Publish Workflow (`.gitea/workflows/publish.yml`)
|
||||
|
||||
@@ -166,25 +166,25 @@ After a PR is merged to master, the release pipeline runs automatically.
|
||||
- Builds the Python package
|
||||
- Optionally publishes to PyPI (if `PYPI_TOKEN` is set)
|
||||
- Creates a Gitea release with git-cliff-generated release notes
|
||||
- Uses `scripts/ci/publish.py` for build and publish orchestration
|
||||
- On failure, creates a Gitea issue via `scripts/ci/notify_failure.py`
|
||||
- Uses `devx.ci.publish` for build and publish orchestration
|
||||
- On failure, creates a Gitea issue via `devx.ci.notify_failure`
|
||||
|
||||
### Auto-Merge Workflow (`.gitea/workflows/auto-merge.yml`)
|
||||
|
||||
- Triggers on `pull_request` labeled events
|
||||
- Runs `scripts/ci/auto_merge.py` with the branch name, PR title, repository, PR number, and label name
|
||||
- Runs `devx.ci.auto_merge` with the branch name, PR title, repository, PR number, and label name
|
||||
- Validates PR title format, checks for APPROVE review, waits for CI, and squash-merges
|
||||
|
||||
### Post-Merge Workflow (`.gitea/workflows/post-merge.yml`)
|
||||
|
||||
- Triggers on push to `master`
|
||||
- Runs `scripts/ci/post_merge.py` with the latest commit message and commit SHA
|
||||
- Runs `devx.ci.post_merge` with the latest commit message and commit SHA
|
||||
- Marks the corresponding Vikunja task as done
|
||||
|
||||
### Smart CI: User-Facing vs Workflow-Only Changes
|
||||
|
||||
Not all changes require the full CI pipeline or a new release. The project uses
|
||||
`scripts/ci/classify_changes.py` to classify changed files into two categories:
|
||||
`devx.ci.classify_changes` to classify changed files into two categories:
|
||||
|
||||
**User-facing paths** (tool changes → release needed):
|
||||
- `src/gitea_runner_manager/**` — Python CLI source
|
||||
@@ -192,28 +192,28 @@ Not all changes require the full CI pipeline or a new release. The project uses
|
||||
- `pyproject.toml` — Package metadata
|
||||
|
||||
**Workflow-only paths** (infrastructure → no release needed):
|
||||
- `.gitea/workflows/**`, `scripts/**`, `docs/**`, `tests/**`
|
||||
- `.gitea/workflows/**`, `docs/**`, `tests/**`
|
||||
- `AGENTS.md`, `README.md`, `CHANGELOG.md`, `Makefile`, `cliff.toml`, etc.
|
||||
|
||||
**CI behavior based on classification:**
|
||||
- **Molecule tests**: Only run when `ansible/` or `.ansible-lint` files change
|
||||
- **Release dry-run**: Only runs when user-facing files change (separate `release-dry-run` job)
|
||||
- **Quality job** (lint, unit tests, coverage, doc-coverage): Always runs
|
||||
- **Release workflow**: `release.py` calls `classify_changes.py` to check if any
|
||||
- **Release workflow**: `release.py` calls `classify_changes` to check if any
|
||||
user-facing files changed since the last tag. If not, the release is skipped
|
||||
entirely — no version bump, no tag, no publish.
|
||||
|
||||
### Dynamic Runner Discovery
|
||||
|
||||
Molecule tests are distributed across available Gitea Actions runners
|
||||
dynamically. The `discover-runners` job runs `scripts/ci/discover_runners.py` which queries the Gitea API for
|
||||
dynamically. The `discover-runners` job runs `devx.molecule.discover_runners` which queries the Gitea API for
|
||||
registered runners at three levels (repo, org, instance) and generates
|
||||
a matrix of runner indices. If the API query fails (e.g., no admin
|
||||
access for instance-level runners), it falls back to the
|
||||
`MOLECULE_RUNNERS` repo variable, then to a default of 3.
|
||||
|
||||
The `molecule-tests` job uses `fromJSON()` to consume the dynamic
|
||||
matrix, and passes the runner count to `distribute_molecule.py
|
||||
matrix, and passes the runner count to `python -m devx.molecule.distribute_molecule
|
||||
--max-runners` so test pairs are evenly distributed.
|
||||
|
||||
When adding or removing Gitea runners:
|
||||
@@ -223,19 +223,19 @@ When adding or removing Gitea runners:
|
||||
|
||||
### Molecule Test Distribution
|
||||
|
||||
`scripts/ci/distribute_molecule.py` discovers all molecule scenarios
|
||||
`devx.molecule.distribute_molecule` discovers all molecule scenarios
|
||||
under `ansible/roles/*/molecule/` and crosses them with the supported
|
||||
OS platform matrix (defined in `scripts/ci/platforms.py`), then splits
|
||||
OS platform matrix (defined in `devx.molecule.platforms`), then splits
|
||||
the resulting test pairs evenly across the requested number of runners.
|
||||
Each pair is encoded as `scenario|platform_name|platform_image|platform_command`.
|
||||
|
||||
`scripts/ci/molecule_ci_guard.py` runs the actual molecule test for a
|
||||
`devx.molecule.molecule_ci_guard` runs the actual molecule test for a
|
||||
given test pair, with CI context (Gitea URL, token, run ID) for
|
||||
reporting results back to the commit status API.
|
||||
|
||||
### Commit Message Validation
|
||||
|
||||
`scripts/ci/validate_commit_msg.py` validates that commit messages
|
||||
`devx.ci.validate_commit_msg` validates that commit messages
|
||||
follow the conventional commit format (`feat:`, `fix:`, `docs:`, etc.).
|
||||
It is used by the pre-commit hook to enforce conventional commits on
|
||||
feature branches.
|
||||
@@ -243,7 +243,7 @@ feature branches.
|
||||
### Release Commit Detection
|
||||
|
||||
The `detect-type` job in the post-merge workflow runs
|
||||
`scripts/ci/detect_release_commit.py` to check whether the latest commit
|
||||
`devx.ci.detect_release_commit` to check whether the latest commit
|
||||
is a release commit (format: `release: vX.Y.Z`). When a release commit
|
||||
is detected, all post-merge jobs (release, sync-wiki, badges, vikunja)
|
||||
are skipped — the tag push triggers the publish workflow instead.
|
||||
@@ -251,9 +251,9 @@ are skipped — the tag push triggers the publish workflow instead.
|
||||
### Badge Generation and Push
|
||||
|
||||
The `badges` job in the post-merge workflow runs
|
||||
`scripts/ci/push_badges.py` which:
|
||||
`devx.ci.push_badges` which:
|
||||
1. Fetches the latest master and hard-resets to it (picks up release commits)
|
||||
2. Generates quality badge SVG files via `scripts/generate_badges.py`
|
||||
2. Generates quality badge SVG files via `devx.tools.generate_badges`
|
||||
3. Creates an orphan `badges` branch
|
||||
4. Copies SVG files to the branch root
|
||||
5. Force-pushes the branch to the remote
|
||||
|
||||
@@ -55,9 +55,9 @@ Every change to master goes through this workflow. No exceptions.
|
||||
3. **Implement** — write code, tests (100% coverage), update docs
|
||||
4. **Commit** — conventional commits (no `GRM-N:` prefix on branch)
|
||||
5. **Push & create PR** — title: `GRM-N: <vikunja task title>`, body: summary + `Closes GRM-N`
|
||||
6. **Review** — review the full diff focusing on: functional completeness, edge cases, technical excellence (architecture, SRP, deduplication, code smells, best practices, code quality, reusability, clean code, readability, maintainability, extensibility), performance, security, UX, documentation completeness/relevance. Post review comments via `scripts/review_pr.py`.
|
||||
6. **Review** — review the full diff focusing on: functional completeness, edge cases, technical excellence (architecture, SRP, deduplication, code smells, best practices, code quality, reusability, clean code, readability, maintainability, extensibility), performance, security, UX, documentation completeness/relevance. Post review comments via `devx.ci.review_pr`.
|
||||
7. **Address comments** — fix each comment, commit, push, re-review
|
||||
8. **Approve** — post an `APPROVE` review via `scripts/review_pr.py`
|
||||
8. **Approve** — post an `APPROVE` review via `devx.ci.review_pr`
|
||||
9. **Add `ready-to-merge` label** — auto-merge workflow squash-merges with title `GRM-N <conventional commit message>`, post-merge workflow marks the Vikunja task as done, release workflow automatically versions and tags
|
||||
|
||||
### Branch Protection (Required Gitea Settings)
|
||||
|
||||
@@ -10,7 +10,7 @@ Key technical decisions for the GRM project, extracted from `CHANGELOG.md` and `
|
||||
|
||||
**Decision:** Use `dynamic = ["version"]` in `pyproject.toml` with setuptools `attr` to source the version from `__version__` in `src/gitea_runner_manager/__init__.py`.
|
||||
|
||||
**Rationale:** `__init__.py` is the single source of truth for the version. The release script (`scripts/release.py`) only updates `__init__.py` — there is no need to touch `pyproject.toml`. `grm --version` reports this version directly. This eliminates version duplication across files and ensures the runtime version always matches the tagged release.
|
||||
**Rationale:** `__init__.py` is the single source of truth for the version. The release script (`devx.ci.release`) only updates `__init__.py` — there is no need to touch `pyproject.toml`. `grm --version` reports this version directly. This eliminates version duplication across files and ensures the runtime version always matches the tagged release.
|
||||
|
||||
**Source:** `CHANGELOG.md` (Unreleased — Added), `AGENTS.md` (Version Bumping Rules)
|
||||
|
||||
@@ -34,7 +34,7 @@ Key technical decisions for the GRM project, extracted from `CHANGELOG.md` and `
|
||||
|
||||
**Decision:** Use conventional commits on feature branches and git-cliff (`cliff.toml`) to calculate the next semver version from commit history, generate the changelog, and automate releases.
|
||||
|
||||
**Rationale:** `scripts/release.py` uses git-cliff to calculate the next version from conventional commits since the last tag. Merge commits on master have the format `GRM-N <conventional commit>`, so `cliff.toml` includes a `commit_preprocessors` entry that strips the `GRM-N ` prefix before parsing. Version bumping rules: `feat:` → minor, `fix:` → patch, `feat!:`/`BREAKING CHANGE` → minor (pre-1.0), `chore:`/`ci:`/`docs:` → no bump. This fully automates versioning and changelog generation.
|
||||
**Rationale:** `devx.ci.release` uses git-cliff to calculate the next version from conventional commits since the last tag. Merge commits on master have the format `GRM-N <conventional commit>`, so `cliff.toml` includes a `commit_preprocessors` entry that strips the `GRM-N ` prefix before parsing. Version bumping rules: `feat:` → minor, `fix:` → patch, `feat!:`/`BREAKING CHANGE` → minor (pre-1.0), `chore:`/`ci:`/`docs:` → no bump. This fully automates versioning and changelog generation.
|
||||
|
||||
**Source:** `CHANGELOG.md` (Unreleased — Added), `AGENTS.md` (Automated Release Pipeline, git-cliff Commit Preprocessing, Version Bumping Rules), `cliff.toml`
|
||||
|
||||
@@ -58,7 +58,7 @@ Key technical decisions for the GRM project, extracted from `CHANGELOG.md` and `
|
||||
|
||||
**Decision:** Require branch protection on `master` (require pull request, require approval review, require status checks, block force pushes) and use an auto-merge workflow that programmatically enforces the APPROVE review check.
|
||||
|
||||
**Rationale:** Branch protection is the primary gate — no direct pushes to master, at least 1 APPROVE review before merge, CI quality + molecule tests must pass, and no history rewriting. The auto-merge workflow (`scripts/auto_merge.py`) enforces the APPROVE review check programmatically as a defense-in-depth measure. When the `ready-to-merge` label is added, the workflow validates PR title format, checks for APPROVE review, waits for CI, and squash-merges with title `GRM-N <conventional commit message>`. The post-merge workflow then marks the Vikunja task as done.
|
||||
**Rationale:** Branch protection is the primary gate — no direct pushes to master, at least 1 APPROVE review before merge, CI quality + molecule tests must pass, and no history rewriting. The auto-merge workflow (`devx.ci.auto_merge`) enforces the APPROVE review check programmatically as a defense-in-depth measure. When the `ready-to-merge` label is added, the workflow validates PR title format, checks for APPROVE review, waits for CI, and squash-merges with title `GRM-N <conventional commit message>`. The post-merge workflow then marks the Vikunja task as done.
|
||||
|
||||
**Source:** `CHANGELOG.md` (Unreleased — Added: mandatory PR review step, auto_merge.py), `AGENTS.md` (Branch Protection, PR Workflow step 8)
|
||||
|
||||
|
||||
@@ -46,8 +46,8 @@ The `make setup` target (from the `Makefile`):
|
||||
- Installs/updates `pip`, `setuptools`, and `wheel`
|
||||
- Creates `.env` from `.env.example` if not present
|
||||
- Generates shell activation scripts (`activate.sh`, `activate.fish`, `activate.zsh`)
|
||||
- Installs `checkmake` via `scripts/install_checkmake.py`
|
||||
- Runs `scripts/setup.sh` to install dependencies and hooks
|
||||
- Installs `checkmake` via `devx.tools.install_checkmake`
|
||||
- Runs `python -m devx.tools.setup` to install dependencies and hooks
|
||||
|
||||
### Developer Quick Start
|
||||
|
||||
@@ -101,7 +101,7 @@ Individual lint targets from the `Makefile`:
|
||||
| `lint-ruff` | `ruff check src/ tests/` |
|
||||
| `lint-format` | `ruff format --check src/ tests/` |
|
||||
| `typecheck` | `pyright` |
|
||||
| `lint-bandit` | `bandit -r src/ scripts/` |
|
||||
| `lint-bandit` | `bandit -r src/` |
|
||||
| `ansible-lint` | `ansible-lint ansible/` |
|
||||
| `makefile-lint` | `checkmake Makefile` |
|
||||
| `lint` | ruff + format check + pyright + bandit |
|
||||
|
||||
@@ -37,13 +37,13 @@ All scenarios test idempotence (second run produces zero changes).
|
||||
|
||||
4 platforms are tested: `ubuntu-2204`, `ubuntu-2404`, `debian-12`, `archlinux`.
|
||||
|
||||
The platform list is defined in `scripts/distribute_molecule.py` (single source of truth).
|
||||
The platform list is defined in `devx.molecule.distribute_molecule` (single source of truth).
|
||||
|
||||
### CI Test Distribution
|
||||
|
||||
CI runs all 6 scenarios × 4 platforms (24 test pairs) distributed across 3 parallel runners.
|
||||
|
||||
From `.gitea/workflows/ci.yml`, the `molecule-tests` job uses a matrix of `runner-index: [0, 1, 2]` and calls `scripts/distribute_molecule.py --runner-index <index> --max-runners 3` to discover assigned test pairs, then runs `scripts/molecule_ci_guard.py` with those pairs.
|
||||
From `.gitea/workflows/ci.yml`, the `molecule-tests` job uses a matrix of `runner-index: [0, 1, 2]` and calls `python -m devx.molecule.distribute_molecule --runner-index <index> --max-runners 3` to discover assigned test pairs, then runs `python -m devx.molecule.molecule_ci_guard` with those pairs.
|
||||
|
||||
## Integration Tests
|
||||
|
||||
|
||||
Reference in New Issue
Block a user