129 lines
7.2 KiB
Markdown
129 lines
7.2 KiB
Markdown
# Testing Strategy
|
|
|
|
GRM employs a multi-layered testing strategy: unit tests with 100% coverage enforcement, integration tests for the CLI lifecycle, and Molecule scenarios for Ansible role validation across multiple OS platforms.
|
|
|
|
## Unit Tests
|
|
|
|
```bash
|
|
make test-unit # Without coverage
|
|
make pytest-cov # With 100% coverage enforcement
|
|
```
|
|
|
|
Runs pytest with 100% coverage requirement.
|
|
|
|
From the `Makefile`:
|
|
|
|
- `test-unit` — `pytest tests/unit/ -v --no-cov` (unit tests without coverage)
|
|
- `pytest-cov` — `pytest tests/ -v --cov=src/grm --cov-report=term-missing --cov-fail-under=100` (unit tests with 100% coverage enforcement)
|
|
|
|
The coverage requirement is `--cov-fail-under=100` — 100% test coverage is required for all code in `src/grm/`. The CI validate job runs `make pytest-cov` on every PR, and the release step in the release-and-maintain job runs it again before tagging a release.
|
|
|
|
### Test speed verification
|
|
|
|
The CI validate job also runs `python -m devx.tools.check_test_speed --max-seconds 10` to verify that unit tests run fast (under 10 seconds total). This catches performance regressions early.
|
|
|
|
## Integration Tests
|
|
|
|
```bash
|
|
make test-integration
|
|
```
|
|
|
|
Tests the full CLI lifecycle commands end-to-end with a mocked executor boundary. This verifies that the CLI correctly parses arguments, resolves runners from the registry, constructs the right Ansible commands, and handles errors — all without actually connecting to remote hosts.
|
|
|
|
From the `Makefile`:
|
|
|
|
- `test-integration` — `pytest tests/integration/ -v --no-cov`
|
|
|
|
Integration tests are marked with `@pytest.mark.integration` and are not counted toward coverage.
|
|
|
|
## Molecule Tests
|
|
|
|
```bash
|
|
make molecule # Quick: all 6 scenarios on Ubuntu 22.04
|
|
make molecule-all # Full: all 6 scenarios on all 4 supported OSes
|
|
```
|
|
|
|
Molecule tests validate the Ansible role (`ansible/roles/gitea_runner/`) by creating Docker containers as test hosts, applying the role, and verifying the results. Each scenario tests a specific aspect of the role.
|
|
|
|
### Scenarios
|
|
|
|
Seven Molecule scenarios are defined under `ansible/roles/gitea_runner/molecule/`:
|
|
|
|
| Scenario | Description | What it verifies |
|
|
|----------|-------------|------------------|
|
|
| `default` | Rootless Docker runner installation | Basic role convergence — user creation, directory structure, binary download, config file, systemd service template, prune timer templates |
|
|
| `multi-instance` | Two isolated runner instances on the same host | Two separate converge plays with different runner names; verifies both instances coexist with independent users, data directories, and service files |
|
|
| `lifecycle` | Stop, disable, re-enable, and start sequence | Converge, then side_effect stops and disables the service, then re-enables and starts it; verify confirms the service is active again |
|
|
| `template-content` | Verify rendered systemd and prune templates | Checks that the systemd user service file contains expected directives (`Type=simple`, `ExecStart`, `Restart=on-failure`, `DOCKER_HOST`, `XDG_RUNTIME_DIR`), and that the prune service and timer templates are correctly rendered |
|
|
| `deregister` | Runner deregistration | Creates a fake `.runner` file, then runs the deregister tasks; verifies the `.runner` file is removed |
|
|
| `update` | Runner binary update | Converge, then side_effect runs the update playbook; verifies the binary is updated |
|
|
| `remove` | Runner removal | Converge, then side_effect runs the remove playbook; verifies the user, directories, and service files are cleaned up |
|
|
|
|
All scenarios test idempotence (second run produces zero changes), which is a core requirement of the Ansible role.
|
|
|
|
### Common scenario configuration
|
|
|
|
All scenarios use `docker_rootless_setup: false` and `skip_runner_registration: true` in their converge playbooks. This is because:
|
|
|
|
- **Rootless Docker** requires kernel user namespace support, which is not available in all Docker-in-Docker CI environments. The role handles this gracefully via the `docker_rootless_setup` guard.
|
|
- **Runner registration** requires a real Gitea instance. The role handles this via the `skip_runner_registration` flag, which skips the `register.yml` and `integration_test.yml` tasks.
|
|
|
|
### Platforms
|
|
|
|
4 platforms are tested:
|
|
|
|
| Platform | Docker image |
|
|
|----------|-------------|
|
|
| `ubuntu-2204` | `geerlingguy/docker-ubuntu2204-ansible` |
|
|
| `ubuntu-2404` | `geerlingguy/docker-ubuntu2404-ansible` |
|
|
| `debian-12` | `geerlingguy/docker-debian12-ansible` |
|
|
| `archlinux` | `archlinux:latest` |
|
|
|
|
The platform list is defined in `devx.molecule.platforms` (single source of truth), shared between `devx.molecule.distribute_molecule` (CI) and `devx.molecule.molecule_all` (local dev tool).
|
|
|
|
### CI Test Distribution
|
|
|
|
CI runs all 7 scenarios x 4 platforms (28 test pairs) distributed across available Gitea Actions runners.
|
|
|
|
The `discover-runners` job runs `devx.molecule.discover_runners` which queries the Gitea API for registered runners at three levels (repo, org, instance) and generates a dynamic matrix. If the API query fails (e.g., no admin access for instance-level runners), it falls back to the `MOLECULE_RUNNERS` repo variable, then to a default of 3.
|
|
|
|
The `molecule-tests` job uses `fromJSON()` to consume the dynamic matrix, and passes the runner count to `python -m devx.molecule.distribute_molecule --max-runners` so test pairs are evenly distributed.
|
|
|
|
`devx.molecule.distribute_molecule` discovers all molecule scenarios under `ansible/roles/*/molecule/` and crosses them with the supported OS platform matrix, then splits the resulting test pairs evenly across the requested number of runners. Each pair is encoded as `scenario|platform_name|platform_image|platform_command`.
|
|
|
|
The CI workflow runs each test pair sequentially via a shell loop that sets the appropriate `MOLECULE_PLATFORM_*` environment variables and invokes `molecule test` directly.
|
|
|
|
### Path-based CI filtering
|
|
|
|
The CI workflow includes a `detect-changes` job that checks whether any files under `ansible/` or `.ansible-lint` have changed. If no Ansible files are changed, molecule tests are skipped — this prevents non-Ansible changes (e.g., Python scripts, workflow YAML, docs) from being blocked by molecule test infrastructure flakiness.
|
|
|
|
## Full Test Suite
|
|
|
|
```bash
|
|
make test-all # Runs pytest-cov + molecule (Ubuntu 22.04)
|
|
```
|
|
|
|
From the `Makefile`:
|
|
|
|
- `test-all` — `pytest-cov + molecule` (unit tests with coverage + all 6 molecule scenarios on Ubuntu 22.04)
|
|
|
|
For a complete test across all platforms, use `make molecule-all` separately.
|
|
|
|
## Build & Test Commands Summary
|
|
|
|
```bash
|
|
make setup # Create venv, install deps, set up hooks, install CI tools
|
|
make lint-all # ruff + pyright + bandit + ansible-lint + checkmake + actionlint
|
|
make pytest-cov # Unit tests with 100% coverage enforcement
|
|
make test-unit # Unit tests without coverage
|
|
make test-integration # Integration tests
|
|
make molecule # All 6 scenarios on Ubuntu 22.04
|
|
make molecule-all # All 6 scenarios on all 4 supported OSes
|
|
make test-all # pytest-cov + molecule
|
|
```
|
|
|
|
## Known Issues
|
|
|
|
- `ansible-lint` may warn about `command-instead-of-module` for `systemctl --user` calls — this is expected (systemd module doesn't support user services) and skipped in `.ansible-lint`
|
|
- Molecule Docker driver may print "Event loop is closed" warnings on interrupt — harmless
|