grm/README.md

# Gitea Runner Manager (GRM)

A lean command-line tool to automate the installation, configuration, and lifecycle management of Gitea Actions runners on Arch Linux, Ubuntu, and Debian hosts.

Each runner runs in an isolated **rootless Docker** environment under a dedicated system user, enabling multiple runners to operate in parallel on the same host without conflicts. The runner binary (`gitea_runner`) is installed directly and managed as a systemd user service.

> **Pronunciation note:** GRM is short for *Gitea Runner Manager*, but say it like **ГРЪМ** (roughly "GRUM" in Latin letters) — the Bulgarian word for **thunder**. Wherever there are clouds, there may be thunders. This is an open-source project from **Oblachno** (облачно means *cloudy* in Bulgarian).

[![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/grm/actions)

## Commit Convention & Branch Naming

This project uses **conventional commits** and **GRM-N branch prefixes**. See [AGENTS.md](AGENTS.md) for the full workflow.

| What | Format | Example |
|------|--------|---------|
| Branch name | `GRM-N-short-description` | `GRM-33-add-pr-review-step` |
| Branch commits | `<conventional commit>` | `feat: add review script` |
| PR title | `GRM-N: <vikunja task title>` | `GRM-33: Add mandatory PR review step` |
| Merge commit | `GRM-N <conventional commit>` | `GRM-33 feat: add review script` |

### PR Workflow

Every change to master goes through a mandatory review workflow:

1. **Create Vikunja task** — get a `GRM-N` identifier
2. **Create branch** — `GRM-N-short-description`
3. **Implement** — write code, tests (100% coverage), update docs
4. **Commit** — conventional commits (no `GRM-N:` prefix on branch)
5. **Push & create PR** — title: `GRM-N: <vikunja task title>`
6. **Review** — review the full diff focusing on: functional completeness, edge cases, technical excellence (architecture, SRP, deduplication, code smells, best practices, code quality, reusability, clean code, readability, maintainability, extensibility), performance, security, UX, documentation completeness/relevance. Post review comments via `scripts/review_pr.py`.
7. **Address comments** — fix each comment, commit, push, re-review
8. **Approve** — post an `APPROVE` review via `scripts/review_pr.py`
9. **Add `ready-to-merge` label** — auto-merge workflow squash-merges with title `GRM-N <conventional commit message>`, post-merge workflow marks the Vikunja task as done, release workflow automatically versions and tags

### Automated Versioning & Releases

Versioning is fully automated using [git-cliff](https://git-cliff.org):

1. **After merge to master** — the release workflow runs `scripts/release.py`
2. **git-cliff calculates the next version** from conventional commits since the last tag
3. **Version file is updated** (`__init__.py`) and a `chore(release): prepare for vX.Y.Z` commit is created
4. **An annotated tag `vX.Y.Z`** is pushed with the changelog as the tag message
5. **The publish workflow triggers** on the tag — builds the package, optionally publishes to PyPI, and creates a Gitea release with generated release notes

| Commit type | Version bump |
|-------------|-------------|
| `feat:` | minor |
| `fix:` | patch |
| `feat!:` / `BREAKING CHANGE` | minor (pre-1.0) |
| `chore:`, `ci:`, `docs:` | no bump |

`grm --version` reports the current version from `__init__.py`.

## Features

- **Simple and focused** — no unnecessary features.
- **Secure** — no hardcoded secrets, uses scoped tokens.
- **Idempotent** — can be run multiple times safely.
- **Flexible** — accepts a plain IP address or hostname, and allows specifying the SSH user and private key.
- **Runner registry** — stores runner connection metadata locally after installation. Subsequent commands need only the runner name.
- **Lifecycle management** — start, stop, enable, disable, status, and remove runners via CLI.
- **Multi-instance** — run multiple isolated runners on the same host, each with its own system user, rootless Docker daemon, data directory, and systemd user service.
- **Rootless Docker** — each runner gets its own rootless Docker daemon, avoiding conflicts with the host's Docker installation and enabling true parallel execution.
- **Systemd-managed** — runners run as systemd user services (`gitea-runner.service`) under dedicated per-runner system users.

## Supported Operating Systems

- Arch Linux
- Ubuntu 22.04 / 24.04
- Debian 12

All supported OSes are tested in CI via molecule scenarios on every PR.

## Prerequisites

- **SSH key authentication** — The remote host must be reachable via SSH using the user specified with `--user` and the private key specified with `--key`. GRM uses Ansible under the hood, which connects to the target host over SSH to execute all installation and configuration tasks. Without valid SSH credentials, Ansible cannot establish a connection and the deployment will fail.
- **Sudo access** — GRM requires root privileges on the remote host to create system users, install packages, and configure rootless Docker. By default, you will be prompted interactively for the sudo password. For automation or uninterrupted workflows, configure passwordless sudo on the remote host and pass `--no-ask-become-pass`.

## Quick Start

### Developer Setup

```bash
git clone https://git.oblachno.oblachno.com/oblachno/gitea-runner-manager.git
cd gitea-runner-manager
pyenv install 3.12
pyenv local 3.12
make setup
```

### Configure Gitea Credentials

```bash
cp .env.example .env
# Edit .env:
# GITEA_URL=https://git.example.com
# GITEA_REGISTRATION_TOKEN=your-registration-token
```

`GITEA_REGISTRATION_TOKEN` is the runner registration token obtained from your Gitea instance (Admin → Actions → Runners → Create Registration Token).

#### Admin API Token (optional)

Set `GITEA_ADMIN_TOKEN` to enable informational API checks during integration test. This is **optional** — the test primarily verifies the runner by checking:

1. **`.runner` registration file** exists and contains valid JSON (proves successful registration)
2. **Systemd user service** is active (proves daemon is polling for jobs)

API checks, if enabled, are purely informational and do not affect pass/fail.

### Install a Runner

Using the CLI (you will be prompted for the sudo password by default):

```bash
grm install 192.168.1.10 --user ubuntu --key ~/.ssh/id_ed25519 --name prod-runner
```

> **Automation tip:** Configure passwordless sudo on the remote host and pass `--no-ask-become-pass` to skip the password prompt. This is recommended for CI/CD pipelines.

Using Make:

```bash
make install HOST=192.168.1.10 USER=ubuntu KEY=~/.ssh/id_ed25519 NAME=prod-runner
```

### Runner Registry

After installation, GRM stores each runner's connection details (host, user, SSH key, Gitea URL) in a local JSON registry at `~/.local/share/grm/runners.json`. This means you rarely need to repeat connection arguments:

```bash
# List all registered runners with live systemd status
grm list
```

### Manage Runner Lifecycle

Once a runner is installed, lifecycle commands work by runner name only:

```bash
# Start a runner
grm start prod-runner

# Stop a runner
grm stop prod-runner

# Enable a runner to start on boot
grm enable prod-runner

# Disable a runner (stops, deregisters, and disables systemd)
grm disable prod-runner --token <token>

# Check runner status
grm status prod-runner

# Remove a runner completely
grm remove prod-runner --token <token>
```

You can override any stored value by passing the corresponding flag:

```bash
grm start prod-runner --host 192.168.1.11 --user root
```

> **Automation tip:** If the remote host has passwordless sudo configured, pass `--no-ask-become-pass`.

### Multiple Instances on the Same Host

Each runner instance is fully isolated with its own system user, rootless Docker daemon, data directory, and systemd user service:

```bash
# Install two runners on the same host
grm install 192.168.1.10 --user ubuntu --name workflow-runner
grm install 192.168.1.10 --user ubuntu --name build-runner

# Manage them independently by name
grm stop workflow-runner
grm status build-runner
```

### Verify Runner

The installer performs an automated integration test that verifies:

1. **`.runner` file exists** with valid JSON containing `id`, `uuid`, `token`, `address` — this proves successful registration with Gitea
2. **Systemd user service is active** — this proves the daemon is polling for jobs

You can also check the Gitea UI under **Actions → Runners** to confirm the runner appears as **Online**.

Optional: If `GITEA_ADMIN_TOKEN` is set, the installer will also query the Gitea API and report whether the runner appears in the admin or repo runners list. This is purely informational.

### View Logs

**GRM application logs** (Python CLI output):

```bash
# Application log file (all messages including DEBUG)
cat ~/.local/state/grm/logs/grm.log

# Enable debug logging in the current session
GRM_LOG_LEVEL=DEBUG grm install 192.168.1.10 --user ubuntu --name prod-runner
```

**Runner logs** (on the remote host):

```bash
# Runner logs (via systemd user service)
sudo -u grm-<name> journalctl --user -u gitea-runner -f
```

The GRM application writes to two destinations:

| Destination | Level | Content |
|-------------|-------|---------|
| Console (stdout) | `GRM_LOG_LEVEL` (default: INFO) | Colorised user-facing messages and operation reports |
| `~/.local/state/grm/logs/grm.log` | DEBUG | All messages with timestamps and severity |

Set `GRM_LOG_LEVEL` to one of `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL` to control console verbosity. The log file always captures everything at DEBUG level regardless of the console setting.

Console output is automatically colorised via ``click.echo``: operation headers in bright cyan, completed steps in green, failures in red, and status updates in yellow.

## Architecture

GRM consists of two layers:

1. **Python CLI** (`src/gitea_runner_manager/`) — built with Click, handles argument parsing, environment loading, i18n translations, and delegates to Ansible via the `ansible-playbook` subprocess.
2. **Ansible Role** (`ansible/roles/gitea-runner/`) — idempotent role that creates a dedicated system user, sets up rootless Docker, installs the runner binary, creates a systemd user service, and registers the runner with Gitea.

```
grm install <host>
  └── RunnerManager.install()
      └── ansible-playbook ansible/install-runner.yml
          └── role: gitea-runner
              ├── user_setup.yml       (create per-runner system user + lingering)
              ├── rootless_docker.yml  (rootless Docker setup under runner user)
              ├── install_runner.yml   (download binary, config, register, service)
              ├── prune.yml            (Docker prune timer)
              └── integration_test.yml (validate service is active)
```

Each runner runs as a systemd user service under a dedicated system user (`grm-<name>`). Each instance has fully isolated resources:

- **User**: `grm-<name>` (dedicated system user with lingering enabled)
- **Home**: `/home/grm-<name>/`
- **Data**: `/var/lib/gitea-runner/<name>/`
- **Config**: `/etc/gitea-runner/<name>/`
- **Service**: `gitea-runner.service` (systemd user service)
- **Docker socket**: `/run/user/<UID>/docker.sock` (rootless, per-runner)

## Configuration

All tunable values are exposed as Ansible variables in `ansible/roles/gitea-runner/defaults/main.yml`:

| Variable | Default | Description |
|----------|---------|-------------|
| `gitea_runner_version` | `1.0.8` | Runner binary version |
| `runner_labels` | `docker,ubuntu-latest:docker://runner-images:ubuntu-22.04` | Runner labels |
| `skip_runner_registration` | `false` | Skip API registration (useful for tests) |
| `gitea_runner_user_prefix` | `grm-` | Prefix for per-runner system users |
| `gitea_runner_base_home` | `/home` | Base directory for runner user homes |
| `gitea_runner_service_user` | `{{ prefix }}{{ runner_name }}` | Per-runner system user |
| `gitea_runner_home` | `{{ base_home }}/{{ service_user }}` | Runner user home directory |
| `gitea_runner_base_data_dir` | `/var/lib/gitea-runner` | Base data directory (instance-scoped) |
| `gitea_runner_base_config_dir` | `/etc/gitea-runner` | Base config directory (instance-scoped) |
| `gitea_runner_data_dir` | `{{ base }}/{{ runner_name }}` | Runtime data directory per instance |
| `gitea_runner_config_dir` | `{{ base }}/{{ runner_name }}` | Config directory per instance |
| `gitea_runner_binary_path` | `/usr/local/bin/gitea_runner` | Binary install path |
| `gitea_runner_prune_until` | `24h` | Prune resources older than this |
| `gitea_runner_prune_schedule` | `daily` | systemd timer schedule |
| `gitea_runner_prune_label` | `gitea-runner=true` | Docker label for pruning |
| `gitea_runner_service_restart_sec` | `5` | systemd RestartSec value |
| `gitea_runner_log_level` | `info` | Runner log level |
| `gitea_runner_container_label` | `gitea-runner=true` | Container label |
| `docker_gpg_key_path` | `/etc/apt/keyrings/docker.asc` | Docker GPG key path |
| `GRM_LANG` | `en` | CLI language: `en`, `bg`, `de`, `ru`, `zh` |
| `GRM_LOG_LEVEL` | `INFO` | Console verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| `GRM_GITEA_API_URL` | `https://git.oblachno.oblachno.fyi/api/v1` | Gitea API URL for CI scripts |
| `GRM_VIKUNJA_API_URL` | `https://work.oblachno.oblachno.fyi/api/v1` | Vikunja API URL for post-merge scripts |
| `GRM_REPO_OWNER` | `oblachno-oss` | Repository owner for CI scripts |
| `GRM_REPO_NAME` | `grm` | Repository name for CI scripts |
| `GRM_VIKUNJA_PROJECT_ID` | `6` | Vikunja project ID for task tracking |

Override any variable by passing it to the CLI with `--extra-vars` or by setting it in your Ansible inventory.

## Development

### Project Structure

```
.
├── src/gitea_runner_manager/      # Python CLI source
│   ├── cli.py                     # Click commands
│   ├── runner_manager.py          # Ansible orchestration + registry integration
│   ├── executor.py                # Ansible subprocess execution
│   ├── registry.py                # Local JSON runner registry
│   ├── i18n.py                    # Translations (en, bg, de, ru, zh)
│   └── exceptions.py              # Custom exceptions
├── ansible/
│   ├── roles/gitea-runner/        # Main Ansible role
│   │   ├── defaults/main.yml      # Default variables
│   │   ├── tasks/                 # Task files
│   │   ├── templates/             # Jinja2 templates
│   │   └── molecule/              # Test scenarios
│   ├── install-runner.yml         # Install playbook
│   ├── update-runner.yml          # Update playbook
│   ├── start-runner.yml           # Start playbook
│   ├── stop-runner.yml            # Stop playbook
│   ├── enable-runner.yml          # Enable playbook
│   ├── disable-runner.yml         # Disable playbook
│   ├── status-runner.yml          # Status playbook
│   └── remove-runner.yml          # Remove playbook
├── tests/
│   ├── unit/                      # Unit tests
│   └── integration/               # Integration tests
├── Makefile                       # Build & test automation
└── pyproject.toml                # Python project metadata
```

### Setup Development Environment

```bash
make setup        # Creates venv, installs deps, sets up hooks
source .venv/bin/activate
```

### Running Linters

```bash
make lint          # Python (ruff + pyright + bandit)
make lint-bandit   # Security scan only
make ansible-lint  # Ansible
make makefile-lint # Makefile
```

## Testing

### Unit Tests

```bash
make test-unit
```

Runs pytest with 100% coverage requirement.

### Molecule Tests

```bash
make molecule      # Quick: all 6 scenarios on Ubuntu 22.04
make molecule-all  # Full: all 6 scenarios on all 4 supported OSes
```

Runs six scenarios:

- **default** — Rootless Docker runner installation
- **multi-instance** — Two isolated runner instances on the same host
- **lifecycle** — Stop, disable, re-enable, and start sequence
- **template-content** — Verify rendered systemd user service and prune templates
- **deregister** — Runner deregistration
- **update** — Runner binary update

All scenarios test idempotence (second run produces zero changes).

CI runs all 6 scenarios × 4 platforms (24 test pairs) distributed across 3 parallel runners.

### Integration Tests

```bash
make test-integration
```

Tests the full CLI lifecycle commands end-to-end ( mocked executor boundary).

### Full Test Suite

```bash
make test-all      # Runs unit tests + linters + molecule
```

## Troubleshooting

### "Event loop is closed" warning

This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure.

### Runner appears offline after installation

- Check that the `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` environment variables are correct.
- Verify the runner service is running: `sudo -u grm-<name> systemctl --user status gitea-runner`.
- Check logs for registration errors.

### Integration test fails

The test checks two things:

1. **`.runner` file missing or invalid** — Registration failed. Check:
   - `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` are correct
   - Runner logs for registration errors
   - The `.runner` file should exist at `/var/lib/gitea-runner/<name>/.runner`

2. **Service not running** — Daemon failed to start. Check:
   - `sudo -u grm-<name> systemctl --user status gitea-runner`
   - Logs for connection errors

### Rootless Docker: service fails to start

- Check the service status: `sudo -u grm-<name> systemctl --user status gitea-runner`.
- Verify the rootless Docker daemon is running: `sudo -u grm-<name> systemctl --user status docker`.
- Verify the Docker socket exists: `ls /run/user/$(id -u grm-<name>)/docker.sock`.
- Check logs: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`.
- Ensure lingering is enabled for the runner user: `loginctl show-user grm-<name> | grep Linger`.

## Makefile Targets

| Target | Description |
|--------|-------------|
| `setup` | Full environment setup |
| `install` | Installs a runner on a host |
| `update` | Updates a runner on a host |
| `start` | Starts a runner instance |
| `stop` | Stops a runner instance |
| `enable` | Enables a runner to start on boot |
| `disable` | Disables and deregisters a runner |
| `status` | Checks runner status |
| `remove` | Removes a runner completely |
| `list` | Lists registered runners with live status |
| `lint` | Runs Python linters (ruff, pyright, bandit) |
| `lint-bandit` | Runs `bandit` security scanner |
| `ansible-lint` | Runs `ansible-lint` |
| `test-unit` | Runs unit tests with coverage |
| `test-integration` | Runs integration tests |
| `molecule` | Runs Ansible Molecule tests |
| `test-all` | Runs all tests |

## License

GPL-3.0