diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index bec05a4..fafc407 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -30,6 +30,10 @@ jobs: run: | . .venv/bin/activate PYTHONPATH=src python3 scripts/release.py --dry-run || true + - name: Documentation coverage check + run: | + . .venv/bin/activate + PYTHONPATH=src python3 scripts/doc_coverage.py detect-changes: runs-on: docker diff --git a/.gitea/workflows/sync-wiki.yml b/.gitea/workflows/sync-wiki.yml new file mode 100644 index 0000000..55c4a5d --- /dev/null +++ b/.gitea/workflows/sync-wiki.yml @@ -0,0 +1,31 @@ +name: Sync Wiki + +on: + push: + branches: [master] + push: + tags: + - 'v*' + +jobs: + sync-wiki: + runs-on: docker + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: Set up environment + run: make setup + - name: Sync documentation to wiki + env: + REPO_TOKEN: ${{ secrets.REPO_TOKEN }} + PYTHONPATH: src + run: | + . .venv/bin/activate + python3 scripts/sync_wiki.py --repo "${{ github.repository }}" + - name: Tag wiki on release + if: startsWith(github.ref, 'refs/tags/v') + env: + REPO_TOKEN: ${{ secrets.REPO_TOKEN }} + run: | + echo "Release tag ${{ github.ref_name }} — wiki synced with release" diff --git a/AGENTS.md b/AGENTS.md index d5a29a6..eb8b420 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -204,3 +204,48 @@ Platform list is defined in `scripts/distribute_molecule.py` (single source of t - `ansible-lint` may warn about `command-instead-of-module` for `systemctl --user` calls — this is expected (systemd module doesn't support user services) and skipped in `.ansible-lint` - Molecule Docker driver may print "Event loop is closed" warnings on interrupt — harmless + +## Documentation-as-Code + +All documentation lives in `/docs/` and is synced to the Gitea wiki automatically. + +### Structure + +``` +docs/ +├── index.md # Wiki homepage +├── mapping.json # File-to-wiki-page title mapping +├── user/ # User documentation +│ ├── getting-started.md +│ ├── installation.md +│ ├── cli-commands.md +│ ├── troubleshooting.md +│ └── faq.md +└── tech/ # Technical documentation + ├── architecture.md + ├── development-setup.md + ├── ci-cd-workflow.md + ├── testing-strategy.md + ├── decision-log.md + └── contributing.md +``` + +### Wiki Sync + +- **On merge to master**: `sync-wiki.yml` workflow runs `scripts/sync_wiki.py` which pushes all `/docs/` content to the Gitea wiki via API +- **On release tag**: Same sync runs, plus the wiki is tagged with the release version +- `mapping.json` maps each file path to a wiki page title (e.g., `user/getting-started.md` → `Getting-Started`) +- README.md is a lean entry point with links to the wiki — no detailed content + +### Documentation Coverage + +- `scripts/doc_coverage.py` checks that all CLI commands, Python modules, and CI scripts are documented +- Runs as a CI step in the quality job +- Goal: 100% coverage for public CLI commands and major architectural components + +### Updating Documentation + +1. Edit files in `/docs/` +2. If adding a new page, add it to `docs/mapping.json` +3. Commit and create a PR (standard PR workflow) +4. On merge, wiki is automatically synced diff --git a/README.md b/README.md index 991bbbb..cc50a3f 100644 --- a/README.md +++ b/README.md @@ -2,437 +2,44 @@ A lean command-line tool to automate the installation, configuration, and lifecycle management of Gitea Actions runners on Arch Linux, Ubuntu, and Debian hosts. -Each runner runs in an isolated **rootless Docker** environment under a dedicated system user, enabling multiple runners to operate in parallel on the same host without conflicts. The runner binary (`gitea_runner`) is installed directly and managed as a systemd user service. +Each runner runs in an isolated **rootless Docker** environment under a dedicated system user, enabling multiple runners to operate in parallel on the same host without conflicts. -> **Pronunciation note:** GRM is short for *Gitea Runner Manager*, but say it like **ГРЪМ** (roughly "GRUM" in Latin letters) — the Bulgarian word for **thunder**. Wherever there are clouds, there may be thunders. This is an open-source project from **Oblachno** (облачно means *cloudy* in Bulgarian). +> **Pronunciation:** GRM is short for *Gitea Runner Manager*, but say it like **ГРЪМ** (roughly "GRUM") — the Bulgarian word for **thunder**. An open-source project from **Oblachno** (облачно means *cloudy* in Bulgarian). [![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions) - -## Commit Convention & Branch Naming - -This project uses **conventional commits** and **GRM-N branch prefixes**. See [AGENTS.md](AGENTS.md) for the full workflow. - -| What | Format | Example | -|------|--------|---------| -| Branch name | `GRM-N-short-description` | `GRM-33-add-pr-review-step` | -| Branch commits | `` | `feat: add review script` | -| PR title | `GRM-N: ` | `GRM-33: Add mandatory PR review step` | -| Merge commit | `GRM-N ` | `GRM-33 feat: add review script` | - -### PR Workflow - -Every change to master goes through a mandatory review workflow: - -1. **Create Vikunja task** — get a `GRM-N` identifier -2. **Create branch** — `GRM-N-short-description` -3. **Implement** — write code, tests (100% coverage), update docs -4. **Commit** — conventional commits (no `GRM-N:` prefix on branch) -5. **Push & create PR** — title: `GRM-N: ` -6. **Review** — review the full diff focusing on: functional completeness, edge cases, technical excellence (architecture, SRP, deduplication, code smells, best practices, code quality, reusability, clean code, readability, maintainability, extensibility), performance, security, UX, documentation completeness/relevance. Post review comments via `scripts/review_pr.py`. -7. **Address comments** — fix each comment, commit, push, re-review -8. **Approve** — post an `APPROVE` review via `scripts/review_pr.py` -9. **Add `ready-to-merge` label** — auto-merge workflow squash-merges with title `GRM-N `, post-merge workflow marks the Vikunja task as done, release workflow automatically versions and tags - -### Automated Versioning & Releases - -Versioning is fully automated using [git-cliff](https://git-cliff.org): - -1. **After merge to master** — the release workflow runs `scripts/release.py` -2. **git-cliff calculates the next version** from conventional commits since the last tag -3. **Version file is updated** (`__init__.py`) and a `chore(release): prepare for vX.Y.Z` commit is created -4. **An annotated tag `vX.Y.Z`** is pushed with the changelog as the tag message -5. **The publish workflow triggers** on the tag — builds the package, optionally publishes to PyPI, and creates a Gitea release with generated release notes - -| Commit type | Version bump | -|-------------|-------------| -| `feat:` | minor | -| `fix:` | patch | -| `feat!:` / `BREAKING CHANGE` | minor (pre-1.0) | -| `chore:`, `ci:`, `docs:` | no bump | - -`grm --version` reports the current version from `__init__.py`. - -## Features - -- **Simple and focused** — no unnecessary features. -- **Secure** — no hardcoded secrets, uses scoped tokens. -- **Idempotent** — can be run multiple times safely. -- **Flexible** — accepts a plain IP address or hostname, and allows specifying the SSH user and private key. -- **Runner registry** — stores runner connection metadata locally after installation. Subsequent commands need only the runner name. -- **Lifecycle management** — start, stop, enable, disable, status, and remove runners via CLI. -- **Multi-instance** — run multiple isolated runners on the same host, each with its own system user, rootless Docker daemon, data directory, and systemd user service. -- **Rootless Docker** — each runner gets its own rootless Docker daemon, avoiding conflicts with the host's Docker installation and enabling true parallel execution. -- **Systemd-managed** — runners run as systemd user services (`gitea-runner.service`) under dedicated per-runner system users. - -## Supported Operating Systems - -- Arch Linux -- Ubuntu 22.04 / 24.04 -- Debian 12 - -All supported OSes are tested in CI via molecule scenarios on every PR. - -## Prerequisites - -- **SSH key authentication** — The remote host must be reachable via SSH using the user specified with `--user` and the private key specified with `--key`. GRM uses Ansible under the hood, which connects to the target host over SSH to execute all installation and configuration tasks. Without valid SSH credentials, Ansible cannot establish a connection and the deployment will fail. -- **Sudo access** — GRM requires root privileges on the remote host to create system users, install packages, and configure rootless Docker. By default, you will be prompted interactively for the sudo password. For automation or uninterrupted workflows, configure passwordless sudo on the remote host and pass `--no-ask-become-pass`. +[![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/LICENSE) ## Quick Start -### Developer Setup - ```bash -git clone https://git.oblachno.oblachno.com/oblachno/gitea-runner-manager.git -cd gitea-runner-manager -pyenv install 3.12 -pyenv local 3.12 +git clone https://git.oblachno.oblachno.fyi/oblachno-oss/grm.git +cd grm make setup -``` - -### Configure Gitea Credentials - -```bash -cp .env.example .env -# Edit .env: -# GITEA_URL=https://git.example.com -# GITEA_REGISTRATION_TOKEN=your-registration-token -``` - -`GITEA_REGISTRATION_TOKEN` is the runner registration token obtained from your Gitea instance (Admin → Actions → Runners → Create Registration Token). - -#### Admin API Token (optional) - -Set `GITEA_ADMIN_TOKEN` to enable informational API checks during integration test. This is **optional** — the test primarily verifies the runner by checking: - -1. **`.runner` registration file** exists and contains valid JSON (proves successful registration) -2. **Systemd user service** is active (proves daemon is polling for jobs) - -API checks, if enabled, are purely informational and do not affect pass/fail. - -### Install a Runner - -Using the CLI (you will be prompted for the sudo password by default): - -```bash +cp .env.example .env # Edit with your Gitea URL and registration token grm install 192.168.1.10 --user ubuntu --key ~/.ssh/id_ed25519 --name prod-runner ``` -> **Automation tip:** Configure passwordless sudo on the remote host and pass `--no-ask-become-pass` to skip the password prompt. This is recommended for CI/CD pipelines. +## Documentation -Using Make: +Full documentation lives on the [**GRM Wiki**](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki). -```bash -make install HOST=192.168.1.10 USER=ubuntu KEY=~/.ssh/id_ed25519 NAME=prod-runner -``` +### User Documentation -### Runner Registry +- [Getting Started](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/Getting-Started) — Installation, quick start, first run +- [Installation](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/Installation) — Prerequisites, setup, multiple instances +- [CLI Commands](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/CLI-Commands) — All commands with arguments and options +- [Troubleshooting](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/Troubleshooting) — Common issues and solutions +- [FAQ](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/FAQ) — Frequently asked questions -After installation, GRM stores each runner's connection details (host, user, SSH key, Gitea URL) in a local JSON registry at `~/.local/share/grm/runners.json`. This means you rarely need to repeat connection arguments: +### Technical Documentation -```bash -# List all registered runners with live systemd status -grm list -``` - -### Manage Runner Lifecycle - -Once a runner is installed, lifecycle commands work by runner name only: - -```bash -# Start a runner -grm start prod-runner - -# Stop a runner -grm stop prod-runner - -# Enable a runner to start on boot -grm enable prod-runner - -# Disable a runner (stops, deregisters, and disables systemd) -grm disable prod-runner --token - -# Check runner status -grm status prod-runner - -# Remove a runner completely -grm remove prod-runner --token -``` - -You can override any stored value by passing the corresponding flag: - -```bash -grm start prod-runner --host 192.168.1.11 --user root -``` - -> **Automation tip:** If the remote host has passwordless sudo configured, pass `--no-ask-become-pass`. - -### Multiple Instances on the Same Host - -Each runner instance is fully isolated with its own system user, rootless Docker daemon, data directory, and systemd user service: - -```bash -# Install two runners on the same host -grm install 192.168.1.10 --user ubuntu --name workflow-runner -grm install 192.168.1.10 --user ubuntu --name build-runner - -# Manage them independently by name -grm stop workflow-runner -grm status build-runner -``` - -### Verify Runner - -The installer performs an automated integration test that verifies: - -1. **`.runner` file exists** with valid JSON containing `id`, `uuid`, `token`, `address` — this proves successful registration with Gitea -2. **Systemd user service is active** — this proves the daemon is polling for jobs - -You can also check the Gitea UI under **Actions → Runners** to confirm the runner appears as **Online**. - -Optional: If `GITEA_ADMIN_TOKEN` is set, the installer will also query the Gitea API and report whether the runner appears in the admin or repo runners list. This is purely informational. - -### View Logs - -**GRM application logs** (Python CLI output): - -```bash -# Application log file (all messages including DEBUG) -cat ~/.local/state/grm/logs/grm.log - -# Enable debug logging in the current session -GRM_LOG_LEVEL=DEBUG grm install 192.168.1.10 --user ubuntu --name prod-runner -``` - -**Runner logs** (on the remote host): - -```bash -# Runner logs (via systemd user service) -sudo -u grm- journalctl --user -u gitea-runner -f -``` - -The GRM application writes to two destinations: - -| Destination | Level | Content | -|-------------|-------|---------| -| Console (stdout) | `GRM_LOG_LEVEL` (default: INFO) | Colorised user-facing messages and operation reports | -| `~/.local/state/grm/logs/grm.log` | DEBUG | All messages with timestamps and severity | - -Set `GRM_LOG_LEVEL` to one of `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL` to control console verbosity. The log file always captures everything at DEBUG level regardless of the console setting. - -Console output is automatically colorised via ``click.echo``: operation headers in bright cyan, completed steps in green, failures in red, and status updates in yellow. - -## Architecture - -GRM consists of two layers: - -1. **Python CLI** (`src/gitea_runner_manager/`) — built with Click, handles argument parsing, environment loading, i18n translations, and delegates to Ansible via the `ansible-playbook` subprocess. -2. **Ansible Role** (`ansible/roles/gitea-runner/`) — idempotent role that creates a dedicated system user, sets up rootless Docker, installs the runner binary, creates a systemd user service, and registers the runner with Gitea. - -``` -grm install - └── RunnerManager.install() - └── ansible-playbook ansible/install-runner.yml - └── role: gitea-runner - ├── user_setup.yml (create per-runner system user + lingering) - ├── rootless_docker.yml (rootless Docker setup under runner user) - ├── install_runner.yml (download binary, config, register, service) - ├── prune.yml (Docker prune timer) - └── integration_test.yml (validate service is active) -``` - -Each runner runs as a systemd user service under a dedicated system user (`grm-`). Each instance has fully isolated resources: - -- **User**: `grm-` (dedicated system user with lingering enabled) -- **Home**: `/home/grm-/` -- **Data**: `/var/lib/gitea-runner//` -- **Config**: `/etc/gitea-runner//` -- **Service**: `gitea-runner.service` (systemd user service) -- **Docker socket**: `/run/user//docker.sock` (rootless, per-runner) - -## Configuration - -All tunable values are exposed as Ansible variables in `ansible/roles/gitea-runner/defaults/main.yml`: - -| Variable | Default | Description | -|----------|---------|-------------| -| `gitea_runner_version` | `1.0.8` | Runner binary version | -| `runner_labels` | `docker,ubuntu-latest:docker://runner-images:ubuntu-22.04` | Runner labels | -| `skip_runner_registration` | `false` | Skip API registration (useful for tests) | -| `gitea_runner_user_prefix` | `grm-` | Prefix for per-runner system users | -| `gitea_runner_base_home` | `/home` | Base directory for runner user homes | -| `gitea_runner_service_user` | `{{ prefix }}{{ runner_name }}` | Per-runner system user | -| `gitea_runner_home` | `{{ base_home }}/{{ service_user }}` | Runner user home directory | -| `gitea_runner_base_data_dir` | `/var/lib/gitea-runner` | Base data directory (instance-scoped) | -| `gitea_runner_base_config_dir` | `/etc/gitea-runner` | Base config directory (instance-scoped) | -| `gitea_runner_data_dir` | `{{ base }}/{{ runner_name }}` | Runtime data directory per instance | -| `gitea_runner_config_dir` | `{{ base }}/{{ runner_name }}` | Config directory per instance | -| `gitea_runner_binary_path` | `/usr/local/bin/gitea_runner` | Binary install path | -| `gitea_runner_prune_until` | `24h` | Prune resources older than this | -| `gitea_runner_prune_schedule` | `daily` | systemd timer schedule | -| `gitea_runner_prune_label` | `gitea-runner=true` | Docker label for pruning | -| `gitea_runner_service_restart_sec` | `5` | systemd RestartSec value | -| `gitea_runner_log_level` | `info` | Runner log level | -| `gitea_runner_container_label` | `gitea-runner=true` | Container label | -| `docker_gpg_key_path` | `/etc/apt/keyrings/docker.asc` | Docker GPG key path | -| `GRM_LANG` | `en` | CLI language: `en`, `bg`, `de`, `ru`, `zh` | -| `GRM_LOG_LEVEL` | `INFO` | Console verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` | -| `GRM_GITEA_API_URL` | `https://git.oblachno.oblachno.fyi/api/v1` | Gitea API URL for CI scripts | -| `GRM_VIKUNJA_API_URL` | `https://work.oblachno.oblachno.fyi/api/v1` | Vikunja API URL for post-merge scripts | -| `GRM_REPO_OWNER` | `oblachno-oss` | Repository owner for CI scripts | -| `GRM_REPO_NAME` | `grm` | Repository name for CI scripts | -| `GRM_VIKUNJA_PROJECT_ID` | `6` | Vikunja project ID for task tracking | - -Override any variable by passing it to the CLI with `--extra-vars` or by setting it in your Ansible inventory. - -## Development - -### Project Structure - -``` -. -├── src/gitea_runner_manager/ # Python CLI source -│ ├── cli.py # Click commands -│ ├── runner_manager.py # Ansible orchestration + registry integration -│ ├── executor.py # Ansible subprocess execution -│ ├── registry.py # Local JSON runner registry -│ ├── i18n.py # Translations (en, bg, de, ru, zh) -│ └── exceptions.py # Custom exceptions -├── ansible/ -│ ├── roles/gitea-runner/ # Main Ansible role -│ │ ├── defaults/main.yml # Default variables -│ │ ├── tasks/ # Task files -│ │ ├── templates/ # Jinja2 templates -│ │ └── molecule/ # Test scenarios -│ ├── install-runner.yml # Install playbook -│ ├── update-runner.yml # Update playbook -│ ├── start-runner.yml # Start playbook -│ ├── stop-runner.yml # Stop playbook -│ ├── enable-runner.yml # Enable playbook -│ ├── disable-runner.yml # Disable playbook -│ ├── status-runner.yml # Status playbook -│ └── remove-runner.yml # Remove playbook -├── tests/ -│ ├── unit/ # Unit tests -│ └── integration/ # Integration tests -├── Makefile # Build & test automation -└── pyproject.toml # Python project metadata -``` - -### Setup Development Environment - -```bash -make setup # Creates venv, installs deps, sets up hooks -source .venv/bin/activate -``` - -### Running Linters - -```bash -make lint # Python (ruff + pyright + bandit) -make lint-bandit # Security scan only -make ansible-lint # Ansible -make makefile-lint # Makefile -``` - -## Testing - -### Unit Tests - -```bash -make test-unit -``` - -Runs pytest with 100% coverage requirement. - -### Molecule Tests - -```bash -make molecule # Quick: all 6 scenarios on Ubuntu 22.04 -make molecule-all # Full: all 6 scenarios on all 4 supported OSes -``` - -Runs six scenarios: - -- **default** — Rootless Docker runner installation -- **multi-instance** — Two isolated runner instances on the same host -- **lifecycle** — Stop, disable, re-enable, and start sequence -- **template-content** — Verify rendered systemd user service and prune templates -- **deregister** — Runner deregistration -- **update** — Runner binary update - -All scenarios test idempotence (second run produces zero changes). - -CI runs all 6 scenarios × 4 platforms (24 test pairs) distributed across 3 parallel runners. - -### Integration Tests - -```bash -make test-integration -``` - -Tests the full CLI lifecycle commands end-to-end ( mocked executor boundary). - -### Full Test Suite - -```bash -make test-all # Runs unit tests + linters + molecule -``` - -## Troubleshooting - -### "Event loop is closed" warning - -This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure. - -### Runner appears offline after installation - -- Check that the `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` environment variables are correct. -- Verify the runner service is running: `sudo -u grm- systemctl --user status gitea-runner`. -- Check logs for registration errors. - -### Integration test fails - -The test checks two things: - -1. **`.runner` file missing or invalid** — Registration failed. Check: - - `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` are correct - - Runner logs for registration errors - - The `.runner` file should exist at `/var/lib/gitea-runner//.runner` - -2. **Service not running** — Daemon failed to start. Check: - - `sudo -u grm- systemctl --user status gitea-runner` - - Logs for connection errors - -### Rootless Docker: service fails to start - -- Check the service status: `sudo -u grm- systemctl --user status gitea-runner`. -- Verify the rootless Docker daemon is running: `sudo -u grm- systemctl --user status docker`. -- Verify the Docker socket exists: `ls /run/user/$(id -u grm-)/docker.sock`. -- Check logs: `sudo -u grm- journalctl --user -u gitea-runner -f`. -- Ensure lingering is enabled for the runner user: `loginctl show-user grm- | grep Linger`. - -## Makefile Targets - -| Target | Description | -|--------|-------------| -| `setup` | Full environment setup | -| `install` | Installs a runner on a host | -| `update` | Updates a runner on a host | -| `start` | Starts a runner instance | -| `stop` | Stops a runner instance | -| `enable` | Enables a runner to start on boot | -| `disable` | Disables and deregisters a runner | -| `status` | Checks runner status | -| `remove` | Removes a runner completely | -| `list` | Lists registered runners with live status | -| `lint` | Runs Python linters (ruff, pyright, bandit) | -| `lint-bandit` | Runs `bandit` security scanner | -| `ansible-lint` | Runs `ansible-lint` | -| `test-unit` | Runs unit tests with coverage | -| `test-integration` | Runs integration tests | -| `molecule` | Runs Ansible Molecule tests | -| `test-all` | Runs all tests | +- [Architecture](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/Architecture) — High-level design, component interactions +- [Development Setup](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/Development-Setup) — Environment setup, dependencies, local testing +- [CI/CD Workflow](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/CI-CD-Workflow) — How CI works, release process, branch protection +- [Testing Strategy](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/Testing-Strategy) — Unit, integration, and Molecule tests +- [Decision Log](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/Decision-Log) — Key technical decisions and rationale +- [Contributing Guide](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/wiki/Contributing-Guide) — Coding standards, PR workflow, commit rules ## License -GPL-3.0 \ No newline at end of file +GPL-3.0 diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..3060261 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,28 @@ +# GRM — Gitea Runner Manager + +A lean command-line tool to automate the installation, configuration, and lifecycle management of Gitea Actions runners on Arch Linux, Ubuntu, and Debian hosts. + +> **Pronunciation:** GRM is short for *Gitea Runner Manager*, but say it like **ГРЪМ** (roughly "GRUM") — the Bulgarian word for **thunder**. An open-source project from **Oblachno** (облачно means *cloudy* in Bulgarian). + +## User Documentation + +- [Getting Started](Getting-Started) — Installation, quick start, first run +- [Installation](Installation) — Prerequisites, setup, multiple instances +- [CLI Commands](CLI-Commands) — All commands with arguments and options +- [Troubleshooting](Troubleshooting) — Common issues and solutions +- [FAQ](FAQ) — Frequently asked questions + +## Technical Documentation + +- [Architecture](Architecture) — High-level design, component interactions, data flow +- [Development Setup](Development-Setup) — Environment setup, dependencies, local testing +- [CI/CD Workflow](CI-CD-Workflow) — How CI works, release process, branch protection +- [Testing Strategy](Testing-Strategy) — Unit, integration, and Molecule tests +- [Decision Log](Decision-Log) — Key technical decisions and rationale +- [Contributing Guide](Contributing-Guide) — Coding standards, PR workflow, commit rules + +## Quick Links + +- [Repository](https://git.oblachno.oblachno.fyi/oblachno-oss/grm) +- [CI/CD Pipeline](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions) +- [Changelog](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/src/branch/master/CHANGELOG.md) diff --git a/docs/mapping.json b/docs/mapping.json new file mode 100644 index 0000000..9f74da2 --- /dev/null +++ b/docs/mapping.json @@ -0,0 +1,14 @@ +{ + "index.md": "Home", + "user/getting-started.md": "Getting-Started", + "user/installation.md": "Installation", + "user/cli-commands.md": "CLI-Commands", + "user/troubleshooting.md": "Troubleshooting", + "user/faq.md": "FAQ", + "tech/architecture.md": "Architecture", + "tech/development-setup.md": "Development-Setup", + "tech/ci-cd-workflow.md": "CI-CD-Workflow", + "tech/testing-strategy.md": "Testing-Strategy", + "tech/decision-log.md": "Decision-Log", + "tech/contributing.md": "Contributing-Guide" +} diff --git a/docs/tech/architecture.md b/docs/tech/architecture.md new file mode 100644 index 0000000..23ad4aa --- /dev/null +++ b/docs/tech/architecture.md @@ -0,0 +1,97 @@ +# Architecture + +GRM consists of two layers: + +1. **Python CLI** (`src/gitea_runner_manager/`) — built with Click, handles argument parsing, environment loading, i18n translations, and delegates to Ansible via the `ansible-playbook` subprocess. +2. **Ansible Role** (`ansible/roles/gitea-runner/`) — idempotent role that creates a dedicated system user, sets up rootless Docker, installs the runner binary, creates a systemd user service, and registers the runner with Gitea. + +## Component Tree + +``` +grm install + └── RunnerManager.install() + └── ansible-playbook ansible/install-runner.yml + └── role: gitea-runner + ├── user_setup.yml (create per-runner system user + lingering) + ├── rootless_docker.yml (rootless Docker setup under runner user) + ├── install_runner.yml (download binary, config, register, service) + ├── prune.yml (Docker prune timer) + └── integration_test.yml (validate service is active) +``` + +The Ansible role task execution order (from `AGENTS.md`): + +``` +main.yml → systemd_check → user_setup → rootless_docker → install_runner → prune → integration_test +``` + +- `install_runner.yml` handles: download, config, validate, register, service +- `main.yml` handles: prune, integration_test (NOT install_runner — avoids duplicates) +- `systemctl --user` tasks must be guarded by `docker_rootless_setup` +- Template creation tasks are NOT guarded by `docker_rootless_setup` (they just create files) + +## Per-Runner Isolation + +Each runner runs as a systemd user service under a dedicated system user (`grm-`). Each instance has fully isolated resources: + +- **User**: `grm-` (dedicated system user with lingering enabled) +- **Home**: `/home/grm-/` +- **Data**: `/var/lib/gitea-runner//` +- **Config**: `/etc/gitea-runner//` +- **Service**: `gitea-runner.service` (systemd user service) +- **Docker socket**: `/run/user//docker.sock` (rootless, per-runner) + +## Component Interactions + +```mermaid +flowchart TD + CLI["Python CLI
src/gitea_runner_manager/
(Click)"] + RM["RunnerManager
runner_manager.py"] + EXEC["Executor
executor.py"] + REG["Registry
registry.py
~/.local/share/grm/runners.json"] + ANS["ansible-playbook subprocess"] + ROLE["Ansible Role
ansible/roles/gitea-runner/"] + USER["user_setup.yml
create system user + lingering"] + DOCKER["rootless_docker.yml
rootless Docker setup"] + INSTALL["install_runner.yml
download, config, register, service"] + PRUNE["prune.yml
Docker prune timer"] + TEST["integration_test.yml
validate service active"] + GITEA["Gitea instance
registration + API"] + SYSTEMD["systemd user service
gitea-runner.service"] + + CLI --> RM + RM --> REG + RM --> EXEC + EXEC -->|subprocess| ANS + ANS --> ROLE + ROLE --> USER + ROLE --> DOCKER + ROLE --> INSTALL + ROLE --> PRUNE + ROLE --> TEST + INSTALL -->|register| GITEA + INSTALL --> SYSTEMD + DOCKER --> SYSTEMD +``` + +## Additional Components + +From `AGENTS.md`, the project also includes: + +- **CI Scripts** (`scripts/`) — Automation for auto-merge, post-merge, release, publishing, molecule distribution, PR reviews, failure notifications +- **Versioning** (`cliff.toml`) — git-cliff configuration for automated semver versioning from conventional commits + +## Python Modules + +The Python CLI layer (`src/gitea_runner_manager/`) consists of the following modules: + +| Module | Description | +|--------|-------------| +| `cli.py` | Click-based CLI entry point — defines all commands (install, update, start, stop, enable, disable, status, remove, list) | +| `runner_manager.py` | Ansible orchestration + registry integration — delegates to executor and manages runner lifecycle | +| `executor.py` | Ansible subprocess execution — runs `ansible-playbook` with extra-vars via temp JSON files | +| `registry.py` | Local JSON runner registry at `~/.local/share/grm/runners.json` — stores connection metadata | +| `i18n.py` | Internationalization translations (en, bg, de, ru, zh) | +| `exceptions.py` | Custom exceptions (`GRMError`, `APIError`) | +| `api_clients.py` | Gitea and Vikunja API client classes for CI automation scripts | +| `config.py` | Configuration constants (API URLs, repo owner/name, project IDs) — overridable via environment variables | diff --git a/docs/tech/ci-cd-workflow.md b/docs/tech/ci-cd-workflow.md new file mode 100644 index 0000000..f187096 --- /dev/null +++ b/docs/tech/ci-cd-workflow.md @@ -0,0 +1,231 @@ +# CI/CD Workflow + +Every change to master goes through a mandatory PR workflow. No exceptions. + +## PR Workflow + +### 1. Create Vikunja Task + +Create a task in Vikunja project 6 to get a `GRM-N` identifier. + +### 2. Create Branch + +```bash +git checkout master && git pull +git checkout -b GRM-N-short-description +``` + +### 3. Implement Changes + +- Write code following conventions +- Write/update tests (100% coverage required) +- Update documentation (CHANGELOG, README, AGENTS.md as needed) + +### 4. Commit (Conventional Commits) + +Branch commits use conventional commit format (no `GRM-N:` prefix): + +``` +feat: add new feature +fix: resolve bug +docs: update README +``` + +### 5. Push and Create PR + +- **PR title format**: `GRM-N: ` (must match the Vikunja task title exactly) +- PR body: summary of changes, `Closes GRM-N` +- Add `ready-to-merge` label **only after review is complete** + +### 6. Review the PR (Mandatory — Before Adding ready-to-merge Label) + +Review the full diff (`git diff master...HEAD`) focusing on: + +- **Functional completeness**: Does the code do what it claims? Are all requirements met? +- **Edge cases**: Are boundary conditions, empty inputs, error paths handled? +- **Technical excellence**: + - Architecture compliance and evolution + - Single Responsibility Principle (SRP) + - Deduplication (no copy-paste, single source of truth) + - Code smells detection and removal + - Best industry practices + - Industry-grade code quality + - Reusability + - Clean code + - Readability + - Maintainability + - Extensibility +- **Performance**: No unnecessary allocations, O(n) vs O(n²), efficient data structures +- **Security**: No secrets in logs/process list, input validation, no injection vectors +- **User experience**: Clear error messages, intuitive CLI flags, helpful output +- **Documentation**: Completeness and relevance of docs, CHANGELOG entries, AGENTS.md updates + +Post review comments using `scripts/review_pr.py`: + +```bash +REPO_TOKEN= python3 scripts/review_pr.py \ + --event REQUEST_CHANGES \ + --body "Review summary" \ + --comments-json comments.json +``` + +### 7. Address Review Comments + +Fix each comment one by one, commit, and push. Re-review until satisfied. + +### 8. Approve and Merge + +Once all comments are addressed: + +```bash +REPO_TOKEN= python3 scripts/review_pr.py \ + --event APPROVE \ + --body "All comments addressed. LGTM." +``` + +Then add the `ready-to-merge` label. The auto-merge workflow will: + +1. **Validate** PR title format and match against Vikunja task title +2. **Check** that at least one APPROVE review exists +3. Wait for all CI checks to pass +4. Squash-merge with title: `GRM-N ` (space-separated) +5. The post-merge workflow marks the Vikunja task as done +6. The release workflow automatically versions, tags, and publishes + +### 9. Post-Merge Automation + +After the squash-merge: + +- The **post-merge workflow** (`.gitea/workflows/post-merge.yml`) triggers on push to `master` and runs `scripts/post_merge.py` to mark the Vikunja task as done, extracting the task ID from the merge commit message. +- The **release workflow** (`.gitea/workflows/release.yml`) triggers on push to `master` and automatically versions, tags, and publishes (see below). + +## Branch Protection (Required Gitea Settings) + +Configure the following branch protection rules for `master` in Gitea repo settings: + +- **Require pull request**: No direct pushes to master +- **Require approval review**: At least 1 `APPROVE` review before merge +- **Require status checks**: CI quality + molecule tests must pass +- **Block force pushes**: No history rewriting on master + +The auto-merge workflow enforces the APPROVE review check programmatically as a defense-in-depth measure, but branch protection is the primary gate. + +## CI Path Filtering + +The CI workflow (`.gitea/workflows/ci.yml`) includes a `detect-changes` job that checks whether any files under `ansible/` or `.ansible-lint` have changed. If no Ansible files are changed, molecule tests are skipped — this prevents non-Ansible changes (e.g., Python scripts, workflow YAML, docs) from being blocked by molecule test infrastructure flakiness. + +The `detect-changes` job: + +- For pull requests: compares `origin/master` against the PR head SHA +- For pushes to master: compares `HEAD~1` against `HEAD` +- Outputs `ansible-changed` as `true` or `false` + +The `molecule-tests` job depends on both `quality` and `detect-changes`, and only runs if `ansible-changed == 'true'`. + +CI triggers only on `opened` and `synchronize` PR events (not `labeled`). + +## CI Quality Job + +The `quality` job in `.gitea/workflows/ci.yml` runs: + +1. `make setup` — full environment setup +2. `make lint-all` — ruff + pyright + bandit + ansible-lint + checkmake +3. `make pytest-cov` — unit tests with 100% coverage enforcement +4. `python3 scripts/check_test_speed.py --max-seconds 10` — verify unit tests run fast +5. `PYTHONPATH=src python3 scripts/release.py --dry-run` — release dry-run validation + +## Automated Release Pipeline + +After a PR is merged to master, the release pipeline runs automatically. + +### Release Workflow (`.gitea/workflows/release.yml`) + +- Triggers on push to `master` +- Sets up full dev environment (`make setup`) so lint and tests can run +- Installs git-cliff (version 2.13.0) +- Configures git as `grm-ci-bot` +- Runs `scripts/release.py` which uses **git-cliff** to: + - Calculate the next semver version from conventional commits since the last tag + - Update `__version__` in `src/gitea_runner_manager/__init__.py` (single source of truth) + - Update `CHANGELOG.md` with the new version section + - **Run `make lint-ruff` and `make pytest-cov`** to verify the release is healthy + - If lint or tests fail, **abort immediately** — no commit, no tag + - Commit with `release: vX.Y.Z` prefix (cleaner than `chore(release):`) + - Create an annotated tag `vX.Y.Z` on the release commit + - Push both the commit and tag to master +- `--skip-tests` flag bypasses test verification (emergency use only, not recommended) +- Loops are prevented by `has_unreleased_changes` — after a release commit is tagged, the next run finds no unreleased changes and exits +- On failure, creates a Gitea issue via `scripts/notify_failure.py` + +### Publish Workflow (`.gitea/workflows/publish.yml`) + +- Triggers on tag push (`v*`) +- Installs git-cliff (version 2.13.0) +- Installs build tools (`build`, `twine`, `requests`, `python-dotenv`, `click`) +- Validates `PYPI_TOKEN` is set (warns if missing) +- Builds the Python package +- Optionally publishes to PyPI (if `PYPI_TOKEN` is set) +- Creates a Gitea release with git-cliff-generated release notes +- Uses `scripts/publish.py` for build and publish orchestration +- On failure, creates a Gitea issue via `scripts/notify_failure.py` + +### Auto-Merge Workflow (`.gitea/workflows/auto-merge.yml`) + +- Triggers on `pull_request` labeled events +- Runs `scripts/auto_merge.py` with the branch name, PR title, repository, PR number, and label name +- Validates PR title format, checks for APPROVE review, waits for CI, and squash-merges + +### Post-Merge Workflow (`.gitea/workflows/post-merge.yml`) + +- Triggers on push to `master` +- Runs `scripts/post_merge.py` with the latest commit message and commit SHA +- Marks the corresponding Vikunja task as done + +## git-cliff Commit Preprocessing + +Merge commits on master have the format `GRM-N `. The `GRM-N ` prefix is not a valid conventional commit prefix, so `cliff.toml` includes a `commit_preprocessors` entry that strips it before parsing: + +```toml +commit_preprocessors = [ + # Strip GRM-N task ID prefix from merge commits so git-cliff sees conventional commits + { pattern = "^GRM-\\d+\\s+", replace = "" }, +] +``` + +This ensures all merged work appears in the changelog. + +### git-cliff Configuration Highlights (`cliff.toml`) + +- `conventional_commits = true` — parse conventional commit format +- `filter_unconventional = true` — skip non-conventional commits +- `render_always = true` — always render the changelog +- `trim = true` — trim whitespace +- Commit parsers group commits into: Features, Bug Fixes, Documentation, Performance, Refactor, Styling, Testing, Miscellaneous Tasks, Security, Revert, Other +- `chore(release): prepare for`, `chore(deps.*)`, `chore(pr)`, `chore(pull)` commits are skipped +- `sort_commits = "oldest"` — oldest commits first + +## Version Bumping Rules (git-cliff) + +| Commit type | Version bump | +|-------------|-------------| +| `feat:` | minor (0.X.0) | +| `fix:` | patch (0.0.X) | +| `feat!:` or `BREAKING CHANGE` | minor (pre-1.0: major would be 1.0.0) | +| `chore:`, `ci:`, `docs:` | no bump (excluded by cliff.toml) | + +From `cliff.toml` `[bump]` section: + +- `features_always_bump_minor = true` +- `breaking_always_bump_major = false` +- `initial_tag = "0.1.0"` + +The version source is `__version__` in `src/gitea_runner_manager/__init__.py`, read by setuptools via `dynamic = ["version"]` in `pyproject.toml`. The release script only updates `__init__.py` — no need to touch `pyproject.toml`. `grm --version` reports this version. + +## Title Format Summary + +| What | Format | Example | +|------|--------|---------| +| Branch name | `GRM-N-short-description` | `GRM-33-add-pr-review-step` | +| Branch commits | `` | `feat: add review script` | +| PR title | `GRM-N: ` | `GRM-33: Add mandatory PR review step` | +| Merge commit | `GRM-N ` | `GRM-33 feat: add review script` | diff --git a/docs/tech/contributing.md b/docs/tech/contributing.md new file mode 100644 index 0000000..b93e797 --- /dev/null +++ b/docs/tech/contributing.md @@ -0,0 +1,100 @@ +# Contributing Guide + +## Key Conventions + +- Python 3.12+ required (ruff/pyright target `py312`) +- 100% test coverage required (`--cov-fail-under=100`) +- Conventional commits on feature branches (no `GRM-N:` prefix) +- Branch names must include `GRM-N` task ID +- Line length: 120 chars +- Secrets are passed via temp JSON files, never on the command line (CWE-214) +- CI triggers only on `opened` and `synchronize` PR events (not `labeled`) + +## Code Style Rules + +- **Python version**: 3.12+ (ruff and pyright target `py312`) +- **Line length**: 120 characters +- **Test coverage**: 100% required (`--cov-fail-under=100`) +- **Secrets handling**: Secrets are passed via temp JSON files with `0600` permissions, never on the command line (CWE-214). Extra-vars are written to a temporary JSON file and passed via `--extra-vars @tempfile`, which is deleted after execution. This prevents secrets from being visible in the process list (`ps aux`). +- **Linting**: `make lint-all` runs ruff + pyright + bandit + ansible-lint + checkmake + +## Commit Rules + +Branch commits use conventional commit format (no `GRM-N:` prefix): + +``` +feat: add new feature +fix: resolve bug +docs: update README +``` + +### Version Bumping Rules + +| Commit type | Version bump | +|-------------|-------------| +| `feat:` | minor (0.X.0) | +| `fix:` | patch (0.0.X) | +| `feat!:` or `BREAKING CHANGE` | minor (pre-1.0: major would be 1.0.0) | +| `chore:`, `ci:`, `docs:` | no bump (excluded by cliff.toml) | + +## Branch Naming + +| What | Format | Example | +|------|--------|---------| +| Branch name | `GRM-N-short-description` | `GRM-33-add-pr-review-step` | +| Branch commits | `` | `feat: add review script` | +| PR title | `GRM-N: ` | `GRM-33: Add mandatory PR review step` | +| Merge commit | `GRM-N ` | `GRM-33 feat: add review script` | + +## PR Workflow Summary + +Every change to master goes through this workflow. No exceptions. + +1. **Create Vikunja task** — get a `GRM-N` identifier (Vikunja project 6) +2. **Create branch** — `GRM-N-short-description` +3. **Implement** — write code, tests (100% coverage), update docs +4. **Commit** — conventional commits (no `GRM-N:` prefix on branch) +5. **Push & create PR** — title: `GRM-N: `, body: summary + `Closes GRM-N` +6. **Review** — review the full diff focusing on: functional completeness, edge cases, technical excellence (architecture, SRP, deduplication, code smells, best practices, code quality, reusability, clean code, readability, maintainability, extensibility), performance, security, UX, documentation completeness/relevance. Post review comments via `scripts/review_pr.py`. +7. **Address comments** — fix each comment, commit, push, re-review +8. **Approve** — post an `APPROVE` review via `scripts/review_pr.py` +9. **Add `ready-to-merge` label** — auto-merge workflow squash-merges with title `GRM-N `, post-merge workflow marks the Vikunja task as done, release workflow automatically versions and tags + +### Branch Protection (Required Gitea Settings) + +Configure the following branch protection rules for `master` in Gitea repo settings: + +- **Require pull request**: No direct pushes to master +- **Require approval review**: At least 1 `APPROVE` review before merge +- **Require status checks**: CI quality + molecule tests must pass +- **Block force pushes**: No history rewriting on master + +The auto-merge workflow enforces the APPROVE review check programmatically as a defense-in-depth measure, but branch protection is the primary gate. + +## Build & Test Commands + +```bash +make setup # Create venv, install deps, set up hooks +make lint-all # ruff + pyright + bandit + ansible-lint + checkmake +make pytest-cov # Unit tests with 100% coverage enforcement +make test-unit # Unit tests without coverage +make molecule # All 6 scenarios on Ubuntu 22.04 +make molecule-all # All 6 scenarios on all 4 supported OSes +make test-all # pytest-cov + molecule +``` + +## Ansible Role Conventions + +``` +main.yml → systemd_check → user_setup → rootless_docker → install_runner → prune → integration_test +``` + +- `install_runner.yml` handles: download, config, validate, register, service +- `main.yml` handles: prune, integration_test (NOT install_runner — avoids duplicates) +- `systemctl --user` tasks must be guarded by `docker_rootless_setup` +- Template creation tasks are NOT guarded by `docker_rootless_setup` (they just create files) + +## Known Issues + +- `ansible-lint` may warn about `command-instead-of-module` for `systemctl --user` calls — this is expected (systemd module doesn't support user services) and skipped in `.ansible-lint` +- Molecule Docker driver may print "Event loop is closed" warnings on interrupt — harmless diff --git a/docs/tech/decision-log.md b/docs/tech/decision-log.md new file mode 100644 index 0000000..d8433d6 --- /dev/null +++ b/docs/tech/decision-log.md @@ -0,0 +1,75 @@ +# Decision Log + +Key technical decisions for the GRM project, extracted from `CHANGELOG.md` and `AGENTS.md`. + +--- + +## ADR-001: Dynamic Versioning via `__init__.py` + +**Date:** 2026-06-21 (v0.2.0 unreleased) + +**Decision:** Use `dynamic = ["version"]` in `pyproject.toml` with setuptools `attr` to source the version from `__version__` in `src/gitea_runner_manager/__init__.py`. + +**Rationale:** `__init__.py` is the single source of truth for the version. The release script (`scripts/release.py`) only updates `__init__.py` — there is no need to touch `pyproject.toml`. `grm --version` reports this version directly. This eliminates version duplication across files and ensures the runtime version always matches the tagged release. + +**Source:** `CHANGELOG.md` (Unreleased — Added), `AGENTS.md` (Version Bumping Rules) + +--- + +## ADR-002: Rootless Docker per Runner + +**Date:** Project inception (documented in README Architecture) + +**Decision:** Each runner instance runs in an isolated rootless Docker environment under a dedicated system user (`grm-`), with its own Docker socket at `/run/user//docker.sock`. + +**Rationale:** Rootless Docker per-runner avoids conflicts with the host's Docker installation and enables true parallel execution of multiple runners on the same host. Each instance has fully isolated resources: user, home, data directory, config directory, systemd user service, and Docker socket. This is a core feature of GRM — enabling multiple isolated runners on the same host. + +**Source:** `README.md` (Architecture, Features), `AGENTS.md` (Architecture) + +--- + +## ADR-003: Conventional Commits + git-cliff for Automated Versioning + +**Date:** 2026-06-21 (v0.2.0 unreleased) + +**Decision:** Use conventional commits on feature branches and git-cliff (`cliff.toml`) to calculate the next semver version from commit history, generate the changelog, and automate releases. + +**Rationale:** `scripts/release.py` uses git-cliff to calculate the next version from conventional commits since the last tag. Merge commits on master have the format `GRM-N `, so `cliff.toml` includes a `commit_preprocessors` entry that strips the `GRM-N ` prefix before parsing. Version bumping rules: `feat:` → minor, `fix:` → patch, `feat!:`/`BREAKING CHANGE` → minor (pre-1.0), `chore:`/`ci:`/`docs:` → no bump. This fully automates versioning and changelog generation. + +**Source:** `CHANGELOG.md` (Unreleased — Added), `AGENTS.md` (Automated Release Pipeline, git-cliff Commit Preprocessing, Version Bumping Rules), `cliff.toml` + +--- + +## ADR-004: Enforce Tests Pass Before Tagging a Release + +**Date:** 2026-06-21 (v0.2.2) + +**Decision:** The release workflow runs `make lint-ruff` and `make pytest-cov` before creating a release commit or tag. If lint or tests fail, the release aborts immediately — no commit, no tag. + +**Rationale:** This ensures every tagged release is healthy. A `--skip-tests` flag exists for emergency use only but is not recommended. This decision was made as a bug fix after identifying that releases could be tagged without verifying test health. Loops are prevented by `has_unreleased_changes` — after a release commit is tagged, the next run finds no unreleased changes and exits. + +**Source:** `CHANGELOG.md` (0.2.2 — Bug Fixes: "Enforce tests pass before tagging a release"), `AGENTS.md` (Automated Release Pipeline) + +--- + +## ADR-005: Branch Protection + Auto-Merge Workflow + +**Date:** 2026-06-21 (v0.2.0 unreleased) + +**Decision:** Require branch protection on `master` (require pull request, require approval review, require status checks, block force pushes) and use an auto-merge workflow that programmatically enforces the APPROVE review check. + +**Rationale:** Branch protection is the primary gate — no direct pushes to master, at least 1 APPROVE review before merge, CI quality + molecule tests must pass, and no history rewriting. The auto-merge workflow (`scripts/auto_merge.py`) enforces the APPROVE review check programmatically as a defense-in-depth measure. When the `ready-to-merge` label is added, the workflow validates PR title format, checks for APPROVE review, waits for CI, and squash-merges with title `GRM-N `. The post-merge workflow then marks the Vikunja task as done. + +**Source:** `CHANGELOG.md` (Unreleased — Added: mandatory PR review step, auto_merge.py), `AGENTS.md` (Branch Protection, PR Workflow step 8) + +--- + +## ADR-006: Path-Based CI Filtering for Molecule Tests + +**Date:** 2026-06-21 (v0.2.0 unreleased) + +**Decision:** The CI workflow includes a `detect-changes` job that checks whether any files under `ansible/` or `.ansible-lint` have changed. If no Ansible files are changed, molecule tests are skipped. + +**Rationale:** This prevents non-Ansible changes (e.g., Python scripts, workflow YAML, docs) from being blocked by molecule test infrastructure flakiness. Molecule tests are only relevant when Ansible files change. The `molecule-tests` job depends on both `quality` and `detect-changes`, and only runs if `ansible-changed == 'true'`. CI triggers only on `opened` and `synchronize` PR events (not `labeled`) to avoid redundant runs. + +**Source:** `AGENTS.md` (CI Path Filtering), `.gitea/workflows/ci.yml` (detect-changes job) diff --git a/docs/tech/development-setup.md b/docs/tech/development-setup.md new file mode 100644 index 0000000..0a0f2fd --- /dev/null +++ b/docs/tech/development-setup.md @@ -0,0 +1,108 @@ +# Development Setup + +## Project Structure + +``` +. +├── src/gitea_runner_manager/ # Python CLI source +│ ├── cli.py # Click commands +│ ├── runner_manager.py # Ansible orchestration + registry integration +│ ├── executor.py # Ansible subprocess execution +│ ├── registry.py # Local JSON runner registry +│ ├── i18n.py # Translations (en, bg, de, ru, zh) +│ └── exceptions.py # Custom exceptions +├── ansible/ +│ ├── roles/gitea-runner/ # Main Ansible role +│ │ ├── defaults/main.yml # Default variables +│ │ ├── tasks/ # Task files +│ │ ├── templates/ # Jinja2 templates +│ │ └── molecule/ # Test scenarios +│ ├── install-runner.yml # Install playbook +│ ├── update-runner.yml # Update playbook +│ ├── start-runner.yml # Start playbook +│ ├── stop-runner.yml # Stop playbook +│ ├── enable-runner.yml # Enable playbook +│ ├── disable-runner.yml # Disable playbook +│ ├── status-runner.yml # Status playbook +│ └── remove-runner.yml # Remove playbook +├── tests/ +│ ├── unit/ # Unit tests +│ └── integration/ # Integration tests +├── Makefile # Build & test automation +└── pyproject.toml # Python project metadata +``` + +## Setup Development Environment + +```bash +make setup # Creates venv, installs deps, sets up hooks +source .venv/bin/activate +``` + +The `make setup` target (from the `Makefile`): + +- Verifies Python 3.12+ is installed +- Creates a virtualenv in `.venv` +- Installs/updates `pip`, `setuptools`, and `wheel` +- Creates `.env` from `.env.example` if not present +- Generates shell activation scripts (`activate.sh`, `activate.fish`, `activate.zsh`) +- Installs `checkmake` via `scripts/install_checkmake.py` +- Runs `scripts/setup.sh` to install dependencies and hooks + +### Developer Quick Start + +```bash +git clone https://git.oblachno.oblachno.com/oblachno/gitea-runner-manager.git +cd gitea-runner-manager +pyenv install 3.12 +pyenv local 3.12 +make setup +``` + +### Configure Gitea Credentials + +```bash +cp .env.example .env +# Edit .env: +# GITEA_URL=https://git.example.com +# GITEA_REGISTRATION_TOKEN=your-registration-token +``` + +`GITEA_REGISTRATION_TOKEN` is the runner registration token obtained from your Gitea instance (Admin → Actions → Runners → Create Registration Token). + +#### Admin API Token (optional) + +Set `GITEA_ADMIN_TOKEN` to enable informational API checks during integration test. This is **optional** — the test primarily verifies the runner by checking: + +1. **`.runner` registration file** exists and contains valid JSON (proves successful registration) +2. **Systemd user service** is active (proves daemon is polling for jobs) + +API checks, if enabled, are purely informational and do not affect pass/fail. + +## Running Linters + +```bash +make lint # Python (ruff + pyright + bandit) +make lint-bandit # Security scan only +make ansible-lint # Ansible +make makefile-lint # Makefile +``` + +The full lint target (`make lint-all`) runs all of the above: + +```bash +make lint-all # ruff + pyright + bandit + ansible-lint + checkmake +``` + +Individual lint targets from the `Makefile`: + +| Target | Description | +|--------|-------------| +| `lint-ruff` | `ruff check src/ tests/` | +| `lint-format` | `ruff format --check src/ tests/` | +| `typecheck` | `pyright` | +| `lint-bandit` | `bandit -r src/ scripts/` | +| `ansible-lint` | `ansible-lint ansible/` | +| `makefile-lint` | `checkmake Makefile` | +| `lint` | ruff + format check + pyright + bandit | +| `lint-all` | lint + ansible-lint + makefile-lint | diff --git a/docs/tech/testing-strategy.md b/docs/tech/testing-strategy.md new file mode 100644 index 0000000..4747e80 --- /dev/null +++ b/docs/tech/testing-strategy.md @@ -0,0 +1,87 @@ +# Testing Strategy + +## Unit Tests + +```bash +make test-unit +``` + +Runs pytest with 100% coverage requirement. + +From the `Makefile`: + +- `test-unit` — `pytest tests/unit/ -v --no-cov` (unit tests without coverage) +- `pytest-cov` — `pytest tests/unit/ -v --cov=src/gitea_runner_manager --cov=scripts --cov-report=term-missing --cov-fail-under=100` (unit tests with 100% coverage enforcement) + +The coverage requirement is `--cov-fail-under=100` — 100% test coverage is required. + +## Molecule Tests + +```bash +make molecule # Quick: all 6 scenarios on Ubuntu 22.04 +make molecule-all # Full: all 6 scenarios on all 4 supported OSes +``` + +Runs six scenarios: + +- **default** — Rootless Docker runner installation +- **multi-instance** — Two isolated runner instances on the same host +- **lifecycle** — Stop, disable, re-enable, and start sequence +- **template-content** — Verify rendered systemd user service and prune templates +- **deregister** — Runner deregistration +- **update** — Runner binary update + +All scenarios test idempotence (second run produces zero changes). + +### Platforms + +4 platforms are tested: `ubuntu-2204`, `ubuntu-2404`, `debian-12`, `archlinux`. + +The platform list is defined in `scripts/distribute_molecule.py` (single source of truth). + +### CI Test Distribution + +CI runs all 6 scenarios × 4 platforms (24 test pairs) distributed across 3 parallel runners. + +From `.gitea/workflows/ci.yml`, the `molecule-tests` job uses a matrix of `runner-index: [0, 1, 2]` and calls `scripts/distribute_molecule.py --runner-index --max-runners 3` to discover assigned test pairs, then runs `scripts/molecule_ci_guard.py` with those pairs. + +## Integration Tests + +```bash +make test-integration +``` + +Tests the full CLI lifecycle commands end-to-end (mocked executor boundary). + +From the `Makefile`: + +- `test-integration` — `pytest tests/integration/ -v --no-cov` + +## Full Test Suite + +```bash +make test-all # Runs unit tests + linters + molecule +``` + +From the `Makefile`: + +- `test-all` — `pytest-cov + molecule` (unit tests with coverage + all 6 molecule scenarios on Ubuntu 22.04) + +## Build & Test Commands Summary + +From `AGENTS.md`: + +```bash +make setup # Create venv, install deps, set up hooks +make lint-all # ruff + pyright + bandit + ansible-lint + checkmake +make pytest-cov # Unit tests with 100% coverage enforcement +make test-unit # Unit tests without coverage +make molecule # All 6 scenarios on Ubuntu 22.04 +make molecule-all # All 6 scenarios on all 4 supported OSes +make test-all # pytest-cov + molecule +``` + +## Known Issues + +- `ansible-lint` may warn about `command-instead-of-module` for `systemctl --user` calls — this is expected (systemd module doesn't support user services) and skipped in `.ansible-lint` +- Molecule Docker driver may print "Event loop is closed" warnings on interrupt — harmless diff --git a/docs/user/cli-commands.md b/docs/user/cli-commands.md new file mode 100644 index 0000000..54171e3 --- /dev/null +++ b/docs/user/cli-commands.md @@ -0,0 +1,233 @@ +# CLI Commands + +GRM provides the following CLI commands for managing Gitea Actions runners. The base command is `grm`. + +## install + +Install and configure a Gitea Runner on a remote host. + +```bash +grm install [options] +``` + +**Arguments:** + +| Argument | Description | +|----------|-------------| +| `host` | Remote host (IP address or hostname) | + +**Options:** + +| Option | Short | Default | Description | +|--------|-------|---------|-------------| +| `--user` | `-u` | `GITEA_RUNNER_USER` env or current login | SSH user | +| `--key` | `-k` | `GITEA_RUNNER_KEY` env | Path to SSH private key | +| `--name` | `-n` | hostname | Gitea Runner name | +| `--token` | `-t` | `GITEA_REGISTRATION_TOKEN` env | Registration token | +| `--url` | — | `GITEA_URL` env | Gitea URL | +| `--admin-token` | `-a` | `REPO_TOKEN` env | Gitea admin API token for integration test | +| `--integration-retries` | `-r` | `3` (`GITEA_INTEGRATION_RETRIES` env) | Integration test API retries | +| `--labels` | `-l` | `GITEA_RUNNER_LABELS` env | Runner labels for Gitea Actions. Example: `docker:docker://alpine:latest` | +| `--ask-become-pass/--no-ask-become-pass` | — | `--ask-become-pass` | Prompt for sudo password (default) or skip it | + +**Example:** + +```bash +grm install 192.168.1.10 --user ubuntu --key ~/.ssh/id_ed25519 --name prod-runner +``` + +## update + +Update the Gitea Runner binary on a remote host. + +```bash +grm update [options] +``` + +**Arguments:** + +| Argument | Description | +|----------|-------------| +| `host` | Remote host (IP address or hostname) | + +**Options:** + +| Option | Short | Default | Description | +|--------|-------|---------|-------------| +| `--user` | `-u` | `GITEA_RUNNER_USER` env or current login | SSH user | +| `--key` | `-k` | `GITEA_RUNNER_KEY` env | Path to SSH private key | +| `--version` | `-v` | — | Specific Gitea Runner version | +| `--ask-become-pass/--no-ask-become-pass` | — | `--ask-become-pass` | Prompt for sudo password (default) or skip it | + +## start + +Start a registered Gitea Runner. + +```bash +grm start [options] +``` + +**Arguments:** + +| Argument | Description | +|----------|-------------| +| `runner_name` | Name of the registered runner | + +**Options (common lifecycle options):** + +| Option | Short | Description | +|--------|-------|-------------| +| `--host` | — | Override host from registry | +| `--user` | `-u` | Override user from registry | +| `--key` | `-k` | Override SSH key from registry | +| `--ask-become-pass/--no-ask-become-pass` | — | Prompt for sudo password (default) or skip it | + +**Example:** + +```bash +grm start prod-runner +# Override stored values: +grm start prod-runner --host 192.168.1.11 --user root +``` + +## stop + +Stop a registered Gitea Runner. + +```bash +grm stop [options] +``` + +**Arguments:** + +| Argument | Description | +|----------|-------------| +| `runner_name` | Name of the registered runner | + +**Options (common lifecycle options):** + +| Option | Short | Description | +|--------|-------|-------------| +| `--host` | — | Override host from registry | +| `--user` | `-u` | Override user from registry | +| `--key` | `-k` | Override SSH key from registry | +| `--ask-become-pass/--no-ask-become-pass` | — | Prompt for sudo password (default) or skip it | + +## enable + +Enable a registered Gitea Runner to start on boot. + +```bash +grm enable [options] +``` + +**Arguments:** + +| Argument | Description | +|----------|-------------| +| `runner_name` | Name of the registered runner | + +**Options (common lifecycle options):** + +| Option | Short | Description | +|--------|-------|-------------| +| `--host` | — | Override host from registry | +| `--user` | `-u` | Override user from registry | +| `--key` | `-k` | Override SSH key from registry | +| `--ask-become-pass/--no-ask-become-pass` | — | Prompt for sudo password (default) or skip it | + +## disable + +Disable a registered Gitea Runner and deregister it. + +```bash +grm disable [options] +``` + +**Arguments:** + +| Argument | Description | +|----------|-------------| +| `runner_name` | Name of the registered runner | + +**Options:** + +| Option | Short | Default | Description | +|--------|-------|---------|-------------| +| `--host` | — | from registry | Override host from registry | +| `--user` | `-u` | from registry | Override user from registry | +| `--key` | `-k` | from registry | Override SSH key from registry | +| `--token` | `-t` | `GITEA_REGISTRATION_TOKEN` env | Registration token | +| `--url` | — | `GITEA_URL` env | Gitea URL | +| `--ask-become-pass/--no-ask-become-pass` | — | `--ask-become-pass` | Prompt for sudo password (default) or skip it | + +**Example:** + +```bash +grm disable prod-runner --token +``` + +## status + +Check the status of a registered Gitea Runner. + +```bash +grm status [options] +``` + +**Arguments:** + +| Argument | Description | +|----------|-------------| +| `runner_name` | Name of the registered runner | + +**Options (common lifecycle options):** + +| Option | Short | Description | +|--------|-------|-------------| +| `--host` | — | Override host from registry | +| `--user` | `-u` | Override user from registry | +| `--key` | `-k` | Override SSH key from registry | +| `--ask-become-pass/--no-ask-become-pass` | — | Prompt for sudo password (default) or skip it | + +## remove + +Remove a registered Gitea Runner completely. + +```bash +grm remove [options] +``` + +**Arguments:** + +| Argument | Description | +|----------|-------------| +| `runner_name` | Name of the registered runner | + +**Options:** + +| Option | Short | Default | Description | +|--------|-------|---------|-------------| +| `--host` | — | from registry | Override host from registry | +| `--user` | `-u` | from registry | Override user from registry | +| `--key` | `-k` | from registry | Override SSH key from registry | +| `--token` | `-t` | `GITEA_REGISTRATION_TOKEN` env | Registration token | +| `--url` | — | `GITEA_URL` env | Gitea URL | +| `--force` | `-f` | — | Skip remote cleanup and only remove the local registry entry | +| `--ask-become-pass/--no-ask-become-pass` | — | `--ask-become-pass` | Prompt for sudo password (default) or skip it | + +**Example:** + +```bash +grm remove prod-runner --token +``` + +## list + +List all registered runners with live status. + +```bash +grm list +``` + +This command takes no arguments or options. It displays a table with columns: NAME, HOST, USER, LABELS, STATUS for all runners stored in the local registry at `~/.local/share/grm/runners.json`. diff --git a/docs/user/faq.md b/docs/user/faq.md new file mode 100644 index 0000000..5572742 --- /dev/null +++ b/docs/user/faq.md @@ -0,0 +1,25 @@ +# FAQ + +### How do I obtain the Gitea registration token? + +The runner registration token is obtained from your Gitea instance: **Admin → Actions → Runners → Create Registration Token**. Set it as `GITEA_REGISTRATION_TOKEN` in your `.env` file or pass it via `--token` on the command line. + +### How do I skip the sudo password prompt for automation? + +Configure passwordless sudo on the remote host and pass `--no-ask-become-pass` to the CLI command. This is recommended for CI/CD pipelines. + +### Can I run multiple runners on the same host? + +Yes. Each runner instance is fully isolated with its own system user (`grm-`), rootless Docker daemon, data directory, and systemd user service. Install additional runners with different `--name` values and manage them independently by name. + +### Why does my runner appear offline after installation? + +Check that `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` are correct, verify the runner service is running with `sudo -u grm- systemctl --user status gitea-runner`, and check the logs for registration errors. You can also confirm the runner appears as **Online** in the Gitea UI under **Actions → Runners**. + +### What does the "Event loop is closed" warning mean? + +This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure. + +### Where are runner connection details stored? + +GRM stores each runner's connection details (host, user, SSH key, Gitea URL) in a local JSON registry at `~/.local/share/grm/runners.json`. After installation, lifecycle commands work by runner name only — you can override any stored value by passing the corresponding flag. diff --git a/docs/user/getting-started.md b/docs/user/getting-started.md new file mode 100644 index 0000000..2ada2dc --- /dev/null +++ b/docs/user/getting-started.md @@ -0,0 +1,88 @@ +# Getting Started + +## Developer Setup + +```bash +git clone https://git.oblachno.oblachno.com/oblachno/gitea-runner-manager.git +cd gitea-runner-manager +pyenv install 3.12 +pyenv local 3.12 +make setup +``` + +## Configure Gitea Credentials + +```bash +cp .env.example .env +# Edit .env: +# GITEA_URL=https://git.example.com +# GITEA_REGISTRATION_TOKEN=your-registration-token +``` + +`GITEA_REGISTRATION_TOKEN` is the runner registration token obtained from your Gitea instance (Admin → Actions → Runners → Create Registration Token). + +### Admin API Token (optional) + +Set `GITEA_ADMIN_TOKEN` to enable informational API checks during integration test. This is **optional** — the test primarily verifies the runner by checking: + +1. **`.runner` registration file** exists and contains valid JSON (proves successful registration) +2. **Systemd user service** is active (proves daemon is polling for jobs) + +API checks, if enabled, are purely informational and do not affect pass/fail. + +## Install a Runner + +Using the CLI (you will be prompted for the sudo password by default): + +```bash +grm install 192.168.1.10 --user ubuntu --key ~/.ssh/id_ed25519 --name prod-runner +``` + +> **Automation tip:** Configure passwordless sudo on the remote host and pass `--no-ask-become-pass` to skip the password prompt. This is recommended for CI/CD pipelines. + +Using Make: + +```bash +make install HOST=192.168.1.10 USER=ubuntu KEY=~/.ssh/id_ed25519 NAME=prod-runner +``` + +## Verify Runner + +The installer performs an automated integration test that verifies: + +1. **`.runner` file exists** with valid JSON containing `id`, `uuid`, `token`, `address` — this proves successful registration with Gitea +2. **Systemd user service is active** — this proves the daemon is polling for jobs + +You can also check the Gitea UI under **Actions → Runners** to confirm the runner appears as **Online**. + +Optional: If `GITEA_ADMIN_TOKEN` is set, the installer will also query the Gitea API and report whether the runner appears in the admin or repo runners list. This is purely informational. + +## View Logs + +**GRM application logs** (Python CLI output): + +```bash +# Application log file (all messages including DEBUG) +cat ~/.local/state/grm/logs/grm.log + +# Enable debug logging in the current session +GRM_LOG_LEVEL=DEBUG grm install 192.168.1.10 --user ubuntu --name prod-runner +``` + +**Runner logs** (on the remote host): + +```bash +# Runner logs (via systemd user service) +sudo -u grm- journalctl --user -u gitea-runner -f +``` + +The GRM application writes to two destinations: + +| Destination | Level | Content | +|-------------|-------|---------| +| Console (stdout) | `GRM_LOG_LEVEL` (default: INFO) | Colorised user-facing messages and operation reports | +| `~/.local/state/grm/logs/grm.log` | DEBUG | All messages with timestamps and severity | + +Set `GRM_LOG_LEVEL` to one of `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL` to control console verbosity. The log file always captures everything at DEBUG level regardless of the console setting. + +Console output is automatically colorised via ``click.echo``: operation headers in bright cyan, completed steps in green, failures in red, and status updates in yellow. diff --git a/docs/user/installation.md b/docs/user/installation.md new file mode 100644 index 0000000..23617c1 --- /dev/null +++ b/docs/user/installation.md @@ -0,0 +1,55 @@ +# Installation + +## Prerequisites + +- **SSH key authentication** — The remote host must be reachable via SSH using the user specified with `--user` and the private key specified with `--key`. GRM uses Ansible under the hood, which connects to the target host over SSH to execute all installation and configuration tasks. Without valid SSH credentials, Ansible cannot establish a connection and the deployment will fail. +- **Sudo access** — GRM requires root privileges on the remote host to create system users, install packages, and configure rootless Docker. By default, you will be prompted interactively for the sudo password. For automation or uninterrupted workflows, configure passwordless sudo on the remote host and pass `--no-ask-become-pass`. + +## Supported Operating Systems + +- Arch Linux +- Ubuntu 22.04 / 24.04 +- Debian 12 + +All supported OSes are tested in CI via molecule scenarios on every PR. + +## Quick Start Install + +Using the CLI (you will be prompted for the sudo password by default): + +```bash +grm install 192.168.1.10 --user ubuntu --key ~/.ssh/id_ed25519 --name prod-runner +``` + +> **Automation tip:** Configure passwordless sudo on the remote host and pass `--no-ask-become-pass` to skip the password prompt. This is recommended for CI/CD pipelines. + +## Make Install + +Using Make: + +```bash +make install HOST=192.168.1.10 USER=ubuntu KEY=~/.ssh/id_ed25519 NAME=prod-runner +``` + +## Runner Registry + +After installation, GRM stores each runner's connection details (host, user, SSH key, Gitea URL) in a local JSON registry at `~/.local/share/grm/runners.json`. This means you rarely need to repeat connection arguments: + +```bash +# List all registered runners with live systemd status +grm list +``` + +## Multiple Instances on the Same Host + +Each runner instance is fully isolated with its own system user, rootless Docker daemon, data directory, and systemd user service: + +```bash +# Install two runners on the same host +grm install 192.168.1.10 --user ubuntu --name workflow-runner +grm install 192.168.1.10 --user ubuntu --name build-runner + +# Manage them independently by name +grm stop workflow-runner +grm status build-runner +``` diff --git a/docs/user/troubleshooting.md b/docs/user/troubleshooting.md new file mode 100644 index 0000000..0702729 --- /dev/null +++ b/docs/user/troubleshooting.md @@ -0,0 +1,50 @@ +# Troubleshooting + +## "Event loop is closed" warning + +This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure. + +## Runner appears offline after installation + +- Check that the `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` environment variables are correct. +- Verify the runner service is running: `sudo -u grm- systemctl --user status gitea-runner`. +- Check logs for registration errors. + +## Integration test fails + +The test checks two things: + +1. **`.runner` file missing or invalid** — Registration failed. Check: + - `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` are correct + - Runner logs for registration errors + - The `.runner` file should exist at `/var/lib/gitea-runner//.runner` + +2. **Service not running** — Daemon failed to start. Check: + - `sudo -u grm- systemctl --user status gitea-runner` + - Logs for connection errors + +## Rootless Docker: service fails to start + +- Check the service status: `sudo -u grm- systemctl --user status gitea-runner`. +- Verify the rootless Docker daemon is running: `sudo -u grm- systemctl --user status docker`. +- Verify the Docker socket exists: `ls /run/user/$(id -u grm-)/docker.sock`. +- Check logs: `sudo -u grm- journalctl --user -u gitea-runner -f`. +- Ensure lingering is enabled for the runner user: `loginctl show-user grm- | grep Linger`. + +## Common Issues Reference Table + +| Symptom | Likely Cause | Solution | +|---------|-------------|----------| +| Pre-commit rejects commit message | Missing conventional format or GRM-N prefix present | Use `feat: description` format without `GRM-N:` | +| `make molecule` fails with `runner_name is undefined` | Verify playbook missing variable | Fixed in Phase 1.1; ensure you're on latest master | +| CI molecule job fails | Docker not available on runner host | Ensure Gitea runner host has Docker installed and running | +| Auto-merge doesn't trigger | Label not exactly `ready-to-merge` or CI checks not all green | Verify label spelling; check CI status | +| Vikunja task not updated after merge | VIKUNJA_TOKEN expired or task ID missing from commit | Regenerate token; verify merge commit has `GRM-N:` prefix | +| Post-merge can't find Vikunja task | Task not in project 6 or identifier mismatch | Verify task exists in Vikunja project 6 with correct identifier | +| `make pytest-cov` fails | Coverage below 100% | Add tests for new code paths | +| `scripts/configure_repo.py` fails | REPO_TOKEN missing or invalid | Set token with repo admin scope and re-run | +| `configure_repo.py` sets wrong status checks | Stale `BRANCH_PROTECTION_CONFIG` | Updated to include `(pull_request)` suffix; re-run `configure_repo.py` | +| Token visible in `ps aux` during install | Old version passed tokens via command line | Fixed: tokens now passed via temp file with `0600` permissions | +| `remove-runner.yml` leaves lingering enabled | Old version didn't disable lingering | Fixed: now runs `loginctl disable-linger` and removes subuid/subgid | +| apt cache update always reports `changed` | `cache_valid_time: 0` forced update every run | Fixed: changed to `cache_valid_time: 3600` | +| Prune/service templates created even when `docker_rootless_setup: false` | Template tasks not guarded | Fixed: template creation now guarded by `docker_rootless_setup` | diff --git a/scripts/doc_coverage.py b/scripts/doc_coverage.py new file mode 100644 index 0000000..efbe4a0 --- /dev/null +++ b/scripts/doc_coverage.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Check documentation coverage for CLI commands and major modules. + +Parses Click commands from the CLI source code and checks if each command +has corresponding documentation in the wiki/docs. Reports missing +documentation as warnings and exits with non-zero if coverage is below 100%. + +Usage: + python3 scripts/doc_coverage.py [--docs-dir docs/] [--fail-on-missing] +""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path + +import click + +from gitea_runner_manager.i18n import _ + +DOCS_DIR = Path(__file__).resolve().parent.parent / "docs" +CLI_FILE = Path(__file__).resolve().parent.parent / "src" / "gitea_runner_manager" / "cli.py" + +# Major modules that should be documented in tech/architecture.md +REQUIRED_MODULES = [ + "cli.py", + "runner_manager.py", + "executor.py", + "registry.py", + "i18n.py", + "exceptions.py", + "api_clients.py", + "config.py", +] + +# CI scripts that should be documented in tech/ci-cd-workflow.md +REQUIRED_SCRIPTS = [ + "auto_merge.py", + "release.py", + "publish.py", + "review_pr.py", + "notify_failure.py", + "post_merge.py", +] + + +def extract_cli_commands() -> list[str]: + """Extract command names from the CLI source file.""" + content = CLI_FILE.read_text() + commands: list[str] = [] + # Find all @cli.command(...) occurrences, then the next def statement + for match in re.finditer(r'@cli\.command\b', content): + # Check for explicit name="..." in the decorator arguments + decorator_end = content.find(")", match.start()) + decorator_text = content[match.start() : decorator_end + 1] + name_match = re.search(r'name\s*=\s*"([^"]+)"', decorator_text) + if name_match: + commands.append(name_match.group(1)) + continue + # Find the next def statement after this decorator + after = content[decorator_end:] + def_match = re.search(r'def\s+(\w+)\s*\(', after) + if def_match: + commands.append(def_match.group(1)) + return commands + + +def check_command_documented(command: str, docs_content: str) -> bool: + """Check if a CLI command is documented in the docs content.""" + # Look for the command name as a heading or in code blocks + patterns = [ + rf"##.*\b{re.escape(command)}\b", + rf"`grm\s+{re.escape(command)}\b", + rf"\bgrm\s+{re.escape(command)}\b", + rf"###.*\b{re.escape(command)}\b", + ] + return any(re.search(p, docs_content, re.IGNORECASE) for p in patterns) + + +def check_module_documented(module: str, docs_content: str) -> bool: + """Check if a module is mentioned in the docs content.""" + return module in docs_content + + +@click.command() +@click.option("--docs-dir", default=str(DOCS_DIR), help="Path to the docs directory.") +@click.option( + "--fail-on-missing", + is_flag=True, + default=False, + help="Exit with non-zero status if any documentation is missing.", +) +def main(docs_dir: str, fail_on_missing: bool) -> None: + docs_path = Path(docs_dir) + cli_commands_file = docs_path / "user" / "cli-commands.md" + architecture_file = docs_path / "tech" / "architecture.md" + ci_cd_file = docs_path / "tech" / "ci-cd-workflow.md" + + missing: list[str] = [] + total = 0 + + # Check CLI commands + click.echo(_("Checking CLI command documentation...")) + commands = extract_cli_commands() + total += len(commands) + cli_docs = cli_commands_file.read_text() if cli_commands_file.exists() else "" + for cmd in commands: + if check_command_documented(cmd, cli_docs): + click.echo(_(" OK: grm {cmd}", cmd=cmd)) + else: + click.echo(_(" MISSING: grm {cmd}", cmd=cmd)) + missing.append(f"CLI command: grm {cmd}") + + # Check modules in architecture.md + click.echo(_("\nChecking module documentation in architecture.md...")) + total += len(REQUIRED_MODULES) + arch_docs = architecture_file.read_text() if architecture_file.exists() else "" + for module in REQUIRED_MODULES: + if check_module_documented(module, arch_docs): + click.echo(_(" OK: {module}", module=module)) + else: + click.echo(_(" MISSING: {module}", module=module)) + missing.append(f"Module: {module}") + + # Check CI scripts in ci-cd-workflow.md + click.echo(_("\nChecking CI script documentation in ci-cd-workflow.md...")) + total += len(REQUIRED_SCRIPTS) + ci_docs = ci_cd_file.read_text() if ci_cd_file.exists() else "" + for script in REQUIRED_SCRIPTS: + if check_module_documented(script, ci_docs): + click.echo(_(" OK: {script}", script=script)) + else: + click.echo(_(" MISSING: {script}", script=script)) + missing.append(f"CI script: {script}") + + # Report + covered = total - len(missing) + percentage = (covered / total * 100) if total > 0 else 100.0 + click.echo( + _( + "\nDoc coverage: {covered}/{total} ({pct}%)", + covered=covered, + total=total, + pct=f"{percentage:.0f}%", + ) + ) + + if missing: + click.echo(_("\nMissing documentation:")) + for item in missing: + click.echo(f" - {item}") + + if missing and fail_on_missing: + click.echo(_("\nERROR: Documentation coverage is not 100%. Use --fail-on-missing to enforce.")) + sys.exit(1) + + if not missing: + click.echo(_("\nAll documentation coverage checks passed!")) + + +if __name__ == "__main__": # pragma: no cover + main() diff --git a/scripts/sync_wiki.py b/scripts/sync_wiki.py new file mode 100644 index 0000000..331787c --- /dev/null +++ b/scripts/sync_wiki.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 +"""Sync documentation from /docs/ to the Gitea wiki via API. + +Reads markdown files from the ``docs/`` directory, uses ``mapping.json`` to +map file paths to wiki page titles, and creates/updates wiki pages via the +Gitea API. Pages that exist in the wiki but not in the mapping are left +untouched (not deleted). + +Usage: + REPO_TOKEN= python3 scripts/sync_wiki.py [--dry-run] [--repo owner/repo] +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +import click +from dotenv import load_dotenv # pyright: ignore[reportMissingImports,reportUnknownVariableType] + +from gitea_runner_manager.api_clients import GiteaClient +from gitea_runner_manager.config import GITEA_API_URL +from gitea_runner_manager.exceptions import APIError +from gitea_runner_manager.i18n import _ + +load_dotenv(override=True) + +DOCS_DIR = Path(__file__).resolve().parent.parent / "docs" +MAPPING_FILE = DOCS_DIR / "mapping.json" + + +def load_mapping() -> dict[str, str]: + """Load the file-to-wiki-page mapping from mapping.json.""" + with open(MAPPING_FILE) as f: + return json.load(f) + + +def read_doc_content(file_path: str) -> str: + """Read markdown content from a docs file.""" + full_path = DOCS_DIR / file_path + with open(full_path) as f: + return f.read() + + +def list_wiki_pages(client: GiteaClient) -> dict[str, str]: + """List existing wiki pages, returning {title: page_name}.""" + try: + pages = client._request("GET", "/wiki/pages").json() + except APIError: + return {} + return {page.get("title", ""): page.get("page_name", page.get("title", "")) for page in pages} + + +def sync_page( + client: GiteaClient, + page_title: str, + content: str, + existing_pages: dict[str, str], + dry_run: bool, +) -> str: + """Create or update a single wiki page. + + Returns "created", "updated", or "skipped" (if dry-run). + """ + if dry_run: + click.echo(_("[dry-run] Would sync page: {title} ({chars} chars)", title=page_title, chars=len(content))) + return "skipped" + + if page_title in existing_pages: + # Update existing page + page_name = existing_pages[page_title] + client._request( + "PUT", + f"/wiki/page/{page_name}", + json={"content": content, "message": f"Sync from docs/ — update {page_title}"}, + ) + return "updated" + + # Create new page + client._request( + "POST", + "/wiki/page", + json={"title": page_title, "content": content, "message": f"Sync from docs/ — create {page_title}"}, + ) + return "created" + + +@click.command() +@click.option("--dry-run", is_flag=True, default=False, help="Show what would happen without making changes.") +@click.option("--repo", default=None, help="Repository in owner/name format (auto-detected if omitted).") +def main(dry_run: bool, repo: str | None) -> None: + token = os.environ.get("REPO_TOKEN", "") + if not token: + raise click.ClickException(_("ERROR: REPO_TOKEN is not set.")) + + if repo is None: + owner = os.environ.get("GRM_REPO_OWNER", "oblachno-oss") + repo_name = os.environ.get("GRM_REPO_NAME", "grm") + else: + owner, repo_name = repo.split("/") + + if not MAPPING_FILE.exists(): + raise click.ClickException(_("ERROR: mapping.json not found at {path}", path=MAPPING_FILE)) + + mapping = load_mapping() + client = GiteaClient(GITEA_API_URL, token, owner, repo_name) + + click.echo(_("Syncing {count} documentation pages to wiki...", count=len(mapping))) + + existing_pages = list_wiki_pages(client) + if existing_pages: + click.echo(_("Found {count} existing wiki pages.", count=len(existing_pages))) + + created = 0 + updated = 0 + skipped = 0 + + for file_path, page_title in sorted(mapping.items()): + try: + content = read_doc_content(file_path) + except FileNotFoundError: + click.echo(_("WARNING: File {file} not found — skipping.", file=file_path)) + skipped += 1 + continue + + result = sync_page(client, page_title, content, existing_pages, dry_run) + if result == "created": + created += 1 + click.echo(_(" Created: {title}", title=page_title)) + elif result == "updated": + updated += 1 + click.echo(_(" Updated: {title}", title=page_title)) + else: + skipped += 1 + + click.echo( + _( + "\nDone! Created: {created}, Updated: {updated}, Skipped: {skipped}", + created=created, + updated=updated, + skipped=skipped, + ) + ) + + +if __name__ == "__main__": # pragma: no cover + main() diff --git a/tests/unit/test_doc_coverage.py b/tests/unit/test_doc_coverage.py new file mode 100644 index 0000000..2dc149d --- /dev/null +++ b/tests/unit/test_doc_coverage.py @@ -0,0 +1,109 @@ +"""Unit tests for scripts/doc_coverage.py.""" + +from pathlib import Path + +from click.testing import CliRunner + +from scripts.doc_coverage import ( + check_command_documented, + check_module_documented, + extract_cli_commands, + main, +) + + +class TestExtractCliCommands: + def test_extracts_commands(self) -> None: + commands = extract_cli_commands() + # Should find all 9 CLI commands + assert "install" in commands + assert "update" in commands + assert "start" in commands + assert "stop" in commands + assert "enable" in commands + assert "disable" in commands + assert "status" in commands + assert "remove" in commands + assert "list" in commands + + def test_returns_list(self) -> None: + commands = extract_cli_commands() + assert isinstance(commands, list) + assert len(commands) == 9 + + +class TestCheckCommandDocumented: + def test_finds_command_in_heading(self) -> None: + content = "## install\n\nInstall a runner." + assert check_command_documented("install", content) is True + + def test_finds_command_in_code_block(self) -> None: + content = "```bash\ngrm install 192.168.1.10\n```" + assert check_command_documented("install", content) is True + + def test_finds_command_with_grm_prefix(self) -> None: + content = "Use `grm start prod-runner` to start." + assert check_command_documented("start", content) is True + + def test_missing_command(self) -> None: + content = "## Other stuff\n\nNo commands here." + assert check_command_documented("install", content) is False + + +class TestCheckModuleDocumented: + def test_finds_module(self) -> None: + content = "The cli.py module handles..." + assert check_module_documented("cli.py", content) is True + + def test_missing_module(self) -> None: + content = "No modules mentioned." + assert check_module_documented("cli.py", content) is False + + +class TestMain: + def test_all_present(self, tmp_path: Path) -> None: + """When all docs exist and cover all commands/modules, exit 0.""" + docs = tmp_path / "docs" + (docs / "user").mkdir(parents=True) + (docs / "tech").mkdir(parents=True) + # Write cli-commands.md with all commands + (docs / "user" / "cli-commands.md").write_text( + "## install\n## update\n## start\n## stop\n## enable\n## disable\n## status\n## remove\n## list\n" + ) + # Write architecture.md with all modules + (docs / "tech" / "architecture.md").write_text( + "cli.py runner_manager.py executor.py registry.py i18n.py exceptions.py api_clients.py config.py" + ) + # Write ci-cd-workflow.md with all scripts + (docs / "tech" / "ci-cd-workflow.md").write_text( + "auto_merge.py release.py publish.py review_pr.py notify_failure.py post_merge.py" + ) + runner = CliRunner() + result = runner.invoke(main, ["--docs-dir", str(docs)]) + assert result.exit_code == 0 + assert "100%" in result.output + + def test_missing_docs_fail(self, tmp_path: Path) -> None: + """When docs are missing and --fail-on-missing is set, exit 1.""" + docs = tmp_path / "docs" + (docs / "user").mkdir(parents=True) + (docs / "tech").mkdir(parents=True) + (docs / "user" / "cli-commands.md").write_text("No commands here.") + (docs / "tech" / "architecture.md").write_text("No modules here.") + (docs / "tech" / "ci-cd-workflow.md").write_text("No scripts here.") + runner = CliRunner() + result = runner.invoke(main, ["--docs-dir", str(docs), "--fail-on-missing"]) + assert result.exit_code == 1 + + def test_missing_docs_warn_only(self, tmp_path: Path) -> None: + """Without --fail-on-missing, missing docs only warn (exit 0).""" + docs = tmp_path / "docs" + (docs / "user").mkdir(parents=True) + (docs / "tech").mkdir(parents=True) + (docs / "user" / "cli-commands.md").write_text("No commands here.") + (docs / "tech" / "architecture.md").write_text("No modules here.") + (docs / "tech" / "ci-cd-workflow.md").write_text("No scripts here.") + runner = CliRunner() + result = runner.invoke(main, ["--docs-dir", str(docs)]) + assert result.exit_code == 0 + assert "MISSING" in result.output diff --git a/tests/unit/test_sync_wiki.py b/tests/unit/test_sync_wiki.py new file mode 100644 index 0000000..25e7d7c --- /dev/null +++ b/tests/unit/test_sync_wiki.py @@ -0,0 +1,186 @@ +"""Unit tests for scripts/sync_wiki.py.""" + +import json +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest +from click.testing import CliRunner + +from scripts.sync_wiki import ( + list_wiki_pages, + load_mapping, + main, + read_doc_content, + sync_page, +) + + +class TestLoadMapping: + def test_loads_mapping(self, tmp_path: Path) -> None: + mapping_file = tmp_path / "mapping.json" + mapping_file.write_text(json.dumps({"user/getting-started.md": "Getting-Started"})) + with patch("scripts.sync_wiki.MAPPING_FILE", mapping_file): + result = load_mapping() + assert result == {"user/getting-started.md": "Getting-Started"} + + def test_missing_mapping_raises(self, tmp_path: Path) -> None: + with patch("scripts.sync_wiki.MAPPING_FILE", tmp_path / "nonexistent.json"): + with pytest.raises(FileNotFoundError): + load_mapping() + + +class TestReadDocContent: + def test_reads_file(self, tmp_path: Path) -> None: + docs_dir = tmp_path / "docs" + docs_dir.mkdir() + (docs_dir / "test.md").write_text("# Test\n\nContent") + with patch("scripts.sync_wiki.DOCS_DIR", docs_dir): + content = read_doc_content("test.md") + assert content == "# Test\n\nContent" + + def test_missing_file_raises(self, tmp_path: Path) -> None: + with patch("scripts.sync_wiki.DOCS_DIR", tmp_path): + with pytest.raises(FileNotFoundError): + read_doc_content("nonexistent.md") + + +class TestListWikiPages: + def test_returns_empty_on_api_error(self) -> None: + from gitea_runner_manager.exceptions import APIError + + client = MagicMock() + client._request.side_effect = APIError(404, "not found") + result = list_wiki_pages(client) + assert result == {} + + def test_returns_page_dict(self) -> None: + client = MagicMock() + client._request.return_value.json.return_value = [ + {"title": "Home", "page_name": "Home"}, + {"title": "Getting-Started", "page_name": "Getting-Started"}, + ] + result = list_wiki_pages(client) + assert result == {"Home": "Home", "Getting-Started": "Getting-Started"} + + +class TestSyncPage: + def test_dry_run_skips(self) -> None: + client = MagicMock() + result = sync_page(client, "Test-Page", "# Content", {}, dry_run=True) + assert result == "skipped" + client._request.assert_not_called() + + def test_creates_new_page(self) -> None: + client = MagicMock() + result = sync_page(client, "New-Page", "# Content", {}, dry_run=False) + assert result == "created" + client._request.assert_called_once() + call_args = client._request.call_args + assert call_args.args[0] == "POST" + assert call_args.args[1] == "/wiki/page" + + def test_updates_existing_page(self) -> None: + client = MagicMock() + existing = {"Existing-Page": "Existing-Page"} + result = sync_page(client, "Existing-Page", "# Updated", existing, dry_run=False) + assert result == "updated" + client._request.assert_called_once() + call_args = client._request.call_args + assert call_args.args[0] == "PUT" + + +class TestMain: + @patch.dict("os.environ", {"REPO_TOKEN": "tok"}) + @patch("scripts.sync_wiki.MAPPING_FILE") + @patch("scripts.sync_wiki.DOCS_DIR") + @patch("scripts.sync_wiki.GiteaClient") + def test_dry_run(self, mock_client_cls: MagicMock, mock_docs_dir: Path, mock_mapping_file: Path) -> None: + mock_mapping_file.exists.return_value = True + mock_mapping_file.__str__ = lambda _: "/docs/mapping.json" + with patch("scripts.sync_wiki.load_mapping", return_value={"index.md": "Home"}): + with patch("scripts.sync_wiki.read_doc_content", return_value="# Home"): + with patch("scripts.sync_wiki.list_wiki_pages", return_value={}): + runner = CliRunner() + result = runner.invoke(main, ["--dry-run", "--repo", "owner/repo"]) + assert result.exit_code == 0 + assert "dry-run" in result.output + + @patch.dict("os.environ", {"REPO_TOKEN": ""}, clear=True) + def test_missing_token_exits(self) -> None: + runner = CliRunner() + result = runner.invoke(main, ["--repo", "owner/repo"]) + assert result.exit_code == 1 + assert "REPO_TOKEN" in result.output + + @patch.dict("os.environ", {"REPO_TOKEN": "tok", "GRM_REPO_OWNER": "me", "GRM_REPO_NAME": "myrepo"}, clear=True) + @patch("scripts.sync_wiki.GiteaClient") + def test_auto_detect_repo(self, mock_client_cls: MagicMock) -> None: + """Test that repo is auto-detected from env vars when --repo is not passed.""" + with patch("scripts.sync_wiki.MAPPING_FILE") as mock_mapping: + mock_mapping.exists.return_value = True + with patch("scripts.sync_wiki.load_mapping", return_value={"index.md": "Home"}): + with patch("scripts.sync_wiki.read_doc_content", return_value="# Home"): + with patch("scripts.sync_wiki.list_wiki_pages", return_value={}): + runner = CliRunner() + result = runner.invoke(main, ["--dry-run"]) + assert result.exit_code == 0 + mock_client_cls.assert_called_once() + + @patch.dict("os.environ", {"REPO_TOKEN": "tok"}, clear=True) + @patch("scripts.sync_wiki.GiteaClient") + def test_missing_mapping_file(self, mock_client_cls: MagicMock) -> None: + """Test that missing mapping.json exits with error.""" + with patch("scripts.sync_wiki.MAPPING_FILE") as mock_mapping: + mock_mapping.exists.return_value = False + runner = CliRunner() + result = runner.invoke(main, ["--repo", "owner/repo"]) + assert result.exit_code == 1 + assert "mapping.json" in result.output + + @patch.dict("os.environ", {"REPO_TOKEN": "tok"}, clear=True) + @patch("scripts.sync_wiki.GiteaClient") + def test_existing_pages_message(self, mock_client_cls: MagicMock) -> None: + """Test that existing wiki pages are reported.""" + with patch("scripts.sync_wiki.MAPPING_FILE") as mock_mapping: + mock_mapping.exists.return_value = True + with patch("scripts.sync_wiki.load_mapping", return_value={"index.md": "Home"}): + with patch("scripts.sync_wiki.read_doc_content", return_value="# Home"): + with patch("scripts.sync_wiki.list_wiki_pages", return_value={"Home": "Home"}): + runner = CliRunner() + result = runner.invoke(main, ["--dry-run", "--repo", "owner/repo"]) + assert result.exit_code == 0 + assert "existing wiki pages" in result.output + + @patch.dict("os.environ", {"REPO_TOKEN": "tok"}, clear=True) + @patch("scripts.sync_wiki.GiteaClient") + def test_file_not_found_warning(self, mock_client_cls: MagicMock) -> None: + """Test that missing doc files are skipped with a warning.""" + with patch("scripts.sync_wiki.MAPPING_FILE") as mock_mapping: + mock_mapping.exists.return_value = True + with patch("scripts.sync_wiki.load_mapping", return_value={"missing.md": "Missing"}): + with patch("scripts.sync_wiki.read_doc_content", side_effect=FileNotFoundError): + with patch("scripts.sync_wiki.list_wiki_pages", return_value={}): + runner = CliRunner() + result = runner.invoke(main, ["--dry-run", "--repo", "owner/repo"]) + assert result.exit_code == 0 + assert "not found" in result.output + assert "Skipped: 1" in result.output + + @patch.dict("os.environ", {"REPO_TOKEN": "tok"}, clear=True) + @patch("scripts.sync_wiki.GiteaClient") + def test_create_and_update(self, mock_client_cls: MagicMock) -> None: + """Test that pages are created and updated correctly (non-dry-run).""" + mock_client = MagicMock() + mock_client_cls.return_value = mock_client + with patch("scripts.sync_wiki.MAPPING_FILE") as mock_mapping: + mock_mapping.exists.return_value = True + mapping = {"new.md": "New-Page", "existing.md": "Existing-Page"} + with patch("scripts.sync_wiki.load_mapping", return_value=mapping): + with patch("scripts.sync_wiki.read_doc_content", return_value="# Content"): + with patch("scripts.sync_wiki.list_wiki_pages", return_value={"Existing-Page": "Existing-Page"}): + runner = CliRunner() + result = runner.invoke(main, ["--repo", "owner/repo"]) + assert result.exit_code == 0 + assert "Created: 1" in result.output + assert "Updated: 1" in result.output