grm/docs/tech/testing-strategy.md

Testing Strategy

GRM employs a multi-layered testing strategy: unit tests with 100% coverage enforcement, integration tests for the CLI lifecycle, and Molecule scenarios for Ansible role validation across multiple OS platforms.

Unit Tests

make test-unit      # Without coverage
make pytest-cov     # With 100% coverage enforcement

Runs pytest with 100% coverage requirement.

From the Makefile:

• test-unit — pytest tests/unit/ -v --no-cov (unit tests without coverage)
• pytest-cov — pytest tests/ -v --cov=src/grm --cov-report=term-missing --cov-fail-under=100 (unit tests with 100% coverage enforcement)

The coverage requirement is --cov-fail-under=100 — 100% test coverage is required for all code in src/grm/. The CI validate job runs make pytest-cov on every PR, and the release step in the release-and-maintain job runs it again before tagging a release.

Test speed verification

The CI validate job also runs python -m devx.tools.check_test_speed --max-seconds 10 to verify that unit tests run fast (under 10 seconds total). This catches performance regressions early.

Integration Tests

make test-integration

Tests the full CLI lifecycle commands end-to-end with a mocked executor boundary. This verifies that the CLI correctly parses arguments, resolves runners from the registry, constructs the right Ansible commands, and handles errors — all without actually connecting to remote hosts.

From the Makefile:

• test-integration — pytest tests/integration/ -v --no-cov

Integration tests are marked with @pytest.mark.integration and are not counted toward coverage.

Molecule Tests

make molecule      # Quick: all 6 scenarios on Ubuntu 22.04
make molecule-all  # Full: all 6 scenarios on all 4 supported OSes

Molecule tests validate the Ansible role (ansible/roles/gitea_runner/) by creating Docker containers as test hosts, applying the role, and verifying the results. Each scenario tests a specific aspect of the role.

Scenarios

Seven Molecule scenarios are defined under ansible/roles/gitea_runner/molecule/:

Scenario	Description	What it verifies
default	Rootless Docker runner installation	Basic role convergence — user creation, directory structure, binary download, config file, systemd service template, prune timer templates
multi-instance	Two isolated runner instances on the same host	Two separate converge plays with different runner names; verifies both instances coexist with independent users, data directories, and service files
lifecycle	Stop, disable, re-enable, and start sequence	Converge, then side_effect stops and disables the service, then re-enables and starts it; verify confirms the service is active again
template-content	Verify rendered systemd and prune templates	Checks that the systemd user service file contains expected directives (Type=simple, ExecStart, Restart=on-failure, DOCKER_HOST, XDG_RUNTIME_DIR), and that the prune service and timer templates are correctly rendered
deregister	Runner deregistration	Creates a fake .runner file, then runs the deregister tasks; verifies the .runner file is removed
update	Runner binary update	Converge, then side_effect runs the update playbook; verifies the binary is updated
remove	Runner removal	Converge, then side_effect runs the remove playbook; verifies the user, directories, and service files are cleaned up

All scenarios test idempotence (second run produces zero changes), which is a core requirement of the Ansible role.

Common scenario configuration

All scenarios use docker_rootless_setup: false and skip_runner_registration: true in their converge playbooks. This is because:

• Rootless Docker requires kernel user namespace support, which is not available in all Docker-in-Docker CI environments. The role handles this gracefully via the docker_rootless_setup guard.
• Runner registration requires a real Gitea instance. The role handles this via the skip_runner_registration flag, which skips the register.yml and integration_test.yml tasks.

Platforms

4 platforms are tested:

Platform	Docker image
ubuntu-2204	geerlingguy/docker-ubuntu2204-ansible
ubuntu-2404	geerlingguy/docker-ubuntu2404-ansible
debian-12	geerlingguy/docker-debian12-ansible
archlinux	archlinux:latest

The platform list is defined in devx.molecule.platforms (single source of truth), shared between devx.molecule.distribute_molecule (CI) and devx.molecule.molecule_all (local dev tool).

CI Test Distribution

CI runs all 7 scenarios x 4 platforms (28 test pairs) distributed across available Gitea Actions runners.

The discover-runners job runs devx.molecule.discover_runners which queries the Gitea API for registered runners at three levels (repo, org, instance) and generates a dynamic matrix. If the API query fails (e.g., no admin access for instance-level runners), it falls back to the MOLECULE_RUNNERS repo variable, then to a default of 3.

The molecule-tests job uses fromJSON() to consume the dynamic matrix, and passes the runner count to python -m devx.molecule.distribute_molecule --max-runners so test pairs are evenly distributed.

devx.molecule.distribute_molecule discovers all molecule scenarios under ansible/roles/*/molecule/ and crosses them with the supported OS platform matrix, then splits the resulting test pairs evenly across the requested number of runners. Each pair is encoded as scenario|platform_name|platform_image|platform_command.

The CI workflow runs each test pair sequentially via a shell loop that sets the appropriate MOLECULE_PLATFORM_* environment variables and invokes molecule test directly.

Path-based CI filtering

The CI workflow includes a detect-changes job that checks whether any files under ansible/ or .ansible-lint have changed. If no Ansible files are changed, molecule tests are skipped — this prevents non-Ansible changes (e.g., Python scripts, workflow YAML, docs) from being blocked by molecule test infrastructure flakiness.

Full Test Suite

make test-all      # Runs pytest-cov + molecule (Ubuntu 22.04)

From the Makefile:

• test-all — pytest-cov + molecule (unit tests with coverage + all 6 molecule scenarios on Ubuntu 22.04)

For a complete test across all platforms, use make molecule-all separately.

Build & Test Commands Summary

make setup          # Create venv, install deps, set up hooks, install CI tools
make lint-all       # ruff + pyright + bandit + ansible-lint + checkmake + actionlint
make pytest-cov     # Unit tests with 100% coverage enforcement
make test-unit      # Unit tests without coverage
make test-integration # Integration tests
make molecule       # All 6 scenarios on Ubuntu 22.04
make molecule-all   # All 6 scenarios on all 4 supported OSes
make test-all       # pytest-cov + molecule

Known Issues

• ansible-lint may warn about command-instead-of-module for systemctl --user calls — this is expected (systemd module doesn't support user services) and skipped in .ansible-lint
• Molecule Docker driver may print "Event loop is closed" warnings on interrupt — harmless