grm/docs/user/troubleshooting.md

# Troubleshooting

## Installation Issues

### Ansible connection fails (UNREACHABLE)

**Symptom:** Ansible reports `UNREACHABLE` when trying to connect to the target host.

**Causes and solutions:**

- **SSH key not found or wrong path** — Verify the key path with `--key`. The key must be readable by the user running `grm`.
- **SSH user does not exist on the target** — Verify the `--user` argument. The user must exist on the remote host and have sudo privileges.
- **Host is not reachable** — Verify the host IP/hostname with `ping` and `ssh -u <user> <host>`.
- **SSH port is not 22** — GRM uses the default SSH port. If your host uses a different port, you may need to configure SSH config (`~/.ssh/config`) with the appropriate port.

### Sudo password prompt fails or is not displayed

**Symptom:** The sudo password prompt does not appear or the command hangs.

**Causes and solutions:**

- **Non-interactive session** — If running in a CI/CD pipeline or script without a TTY, the password prompt cannot be displayed. Configure passwordless sudo on the remote host and pass `--no-ask-become-pass`.
- **Wrong sudo password** — Ensure you are entering the correct sudo password for the remote user.

### GITEA_URL must be set

**Symptom:** Error message `GITEA_URL must be set (or pass --url)`.

**Solution:** Set `GITEA_URL` in your `.env` file or pass it via `--url`:

```bash
# In .env
GITEA_URL=https://git.example.com

# Or on the command line
grm install 192.168.1.10 --user ubuntu --url https://git.example.com
```

### GITEA_REGISTRATION_TOKEN must be set

**Symptom:** Error message `GITEA_REGISTRATION_TOKEN must be set (or pass --token)`.

**Solution:** Set `GITEA_REGISTRATION_TOKEN` in your `.env` file or pass it via `--token`. The token must be a valid registration token from your Gitea instance (it starts with `GR`).

## Runner Issues

### Runner appears offline after installation

- Check that the `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` environment variables are correct.
- Verify the registration token has not expired — generate a new one from Gitea if needed (Site Administration → Actions → Runners → Create Registration Token).
- Verify the runner service is running: `sudo -u grm-<name> systemctl --user status gitea-runner`.
- Check logs for registration errors: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`.
- Confirm the runner appears in the Gitea UI under **Actions → Runners**. If it shows as offline, the runner daemon may not be polling — check network connectivity between the runner host and Gitea.

### Runner service fails to start

- Check the service status: `sudo -u grm-<name> systemctl --user status gitea-runner`
- Check logs: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`
- Verify the runner binary exists: `ls -la /usr/local/bin/gitea_runner`
- Verify the config file exists: `ls -la /etc/gitea-runner/<name>/config.yaml`
- Verify the `.runner` registration file exists: `ls -la /var/lib/gitea-runner/<name>/.runner`

### Rootless Docker: service fails to start

- Check the service status: `sudo -u grm-<name> systemctl --user status gitea-runner`.
- Verify the rootless Docker daemon is running: `sudo -u grm-<name> systemctl --user status docker`.
- Verify the Docker socket exists: `ls /run/user/$(id -u grm-<name>)/docker.sock`.
- Check logs: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`.
- Ensure lingering is enabled for the runner user: `loginctl show-user grm-<name> | grep Linger`. If not enabled, run `sudo loginctl enable-linger grm-<name>`.
- Verify subuid/subgid entries exist: `grep grm-<name> /etc/subuid /etc/subgid`. If missing, the rootless Docker setup will fail.

### Runner not found in registry

**Symptom:** Error message `Runner '<name>' not found in registry. Use 'grm install' first or provide --host and --user.`

**Solution:** The runner was not installed via `grm install`, or the registry file was deleted. Either:

1. Install the runner first: `grm install <host> --user <user> --name <name>`
2. Or provide explicit connection details: `grm status <name> --host <host> --user <user>`

## Integration Test Issues

### Integration test fails

The test checks two things:

1. **`.runner` file missing or invalid** — Registration failed. Check:
   - `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` are correct
   - The registration token is valid and has not expired
   - Runner logs for registration errors: `sudo -u grm-<name> journalctl --user -u gitea-runner`
   - The `.runner` file should exist at `/var/lib/gitea-runner/<name>/.runner`
   - The `.runner` file should contain valid JSON with `id`, `uuid`, `token`, `address` fields

2. **Service not running** — Daemon failed to start. Check:
   - `sudo -u grm-<name> systemctl --user status gitea-runner`
   - Logs for connection errors: `sudo -u grm-<name> journalctl --user -u gitea-runner`
   - Verify the rootless Docker daemon is running (see above)

### API verification shows error status

If `CI_GITEA_TOKEN` is set, the integration test queries the Gitea API. If the API returns `401` or `403`, the token does not have sufficient permissions. This is **informational only** and does not affect pass/fail. The test passes as long as the `.runner` file exists and the systemd service is active.

## Logging and Diagnostics

### Enable debug logging

```bash
GRM_LOG_LEVEL=DEBUG grm install 192.168.1.10 --user ubuntu --name prod-runner
```

This prints all debug messages to the console. The log file at `~/.local/state/grm/logs/grm.log` always captures DEBUG level regardless of this setting.

### View Ansible execution logs

Each `grm` command that invokes Ansible creates a timestamped log file:

```bash
ls ~/.local/state/grm/logs/ansible-*.log
cat ~/.local/state/grm/logs/ansible-<timestamp>.log
```

These logs contain the full Ansible output, including task results, changed/failed counts, and any error messages.

### View runner logs on the remote host

```bash
# Runner daemon logs
sudo -u grm-<name> journalctl --user -u gitea-runner -f

# Rootless Docker daemon logs
sudo -u grm-<name> journalctl --user -u docker -f

# Docker prune timer logs
sudo -u grm-<name> journalctl --user -u docker-prune.service
```

## Development Issues

### "Event loop is closed" warning

This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure.

### Pre-commit rejects commit message

The pre-commit hook validates that commit messages follow conventional commit format (`feat:`, `fix:`, `docs:`, etc.). The `GRM-N:` prefix is not allowed on branch commits — use it only in PR titles.

**Correct:**
```
feat: add new runner label option
```

**Incorrect:**
```
GRM-33: add new runner label option
update README
```

### `make pytest-cov` fails with coverage below 100%

Add tests for any new code paths. The coverage requirement is strict (`--cov-fail-under=100`). Run `make pytest-cov` locally to see which lines are not covered:

```bash
make pytest-cov
# The output shows "Missing" lines for each file
```

### `make molecule` fails with Docker not available

Molecule requires Docker to be installed and running on your machine. Verify:

```bash
docker info  # Should print Docker server info
```

If Docker is not installed, install it via your package manager or [Docker's official installation guide](https://docs.docker.com/get-docker/).

## Common Issues Reference Table

| Symptom | Likely Cause | Solution |
|---------|-------------|----------|
| Ansible UNREACHABLE | SSH connection failed | Verify `--user`, `--key`, and host reachability |
| `GITEA_URL must be set` | Missing environment variable | Set `GITEA_URL` in `.env` or pass `--url` |
| `GITEA_REGISTRATION_TOKEN must be set` | Missing environment variable | Set `GITEA_REGISTRATION_TOKEN` in `.env` or pass `--token` |
| Runner appears offline | Registration failed or service not running | Check GITEA_URL, token validity, and service status |
| Rootless Docker fails to start | subuid/subgid missing or lingering disabled | Verify `/etc/subuid`, `/etc/subgid`, and `loginctl show-user` |
| Runner not found in registry | Runner not installed or registry deleted | Run `grm install` or provide `--host` and `--user` |
| Pre-commit rejects commit message | Missing conventional format or GRM-N prefix present | Use `feat: description` format without `GRM-N:` |
| `make molecule` fails with `runner_name is undefined` | Verify playbook missing variable | Fixed in Phase 1.1; ensure you're on latest master |
| CI molecule job fails | Docker not available on runner host | Ensure Gitea runner host has Docker installed and running |
| Auto-merge doesn't trigger | Label not exactly `ready-to-merge` or CI checks not all green | Verify label spelling; check CI status |
| Vikunja task not updated after merge | VIKUNJA_TOKEN expired or task ID missing from commit | Regenerate token; verify merge commit has `GRM-N:` prefix |
| Post-merge can't find Vikunja task | Task not in project 6 or identifier mismatch | Verify task exists in Vikunja project 6 with correct identifier |
| `make pytest-cov` fails | Coverage below 100% | Add tests for new code paths |
| `devx.tools.configure_repo` fails | CI_GITEA_TOKEN missing or invalid | Set token with repo admin scope and re-run |
| `configure_repo` sets wrong status checks | Stale `BRANCH_PROTECTION_CONFIG` | Updated to include `(pull_request)` suffix; re-run `configure_repo` |
| Token visible in `ps aux` during install | Old version passed tokens via command line | Fixed: tokens now passed via temp file with `0600` permissions |
| `remove-runner.yml` leaves lingering enabled | Old version didn't disable lingering | Fixed: now runs `loginctl disable-linger` and removes subuid/subgid |
| apt cache update always reports `changed` | `cache_valid_time: 0` forced update every run | Fixed: changed to `cache_valid_time: 3600` |
| Prune/service templates created even when `docker_rootless_setup: false` | Template tasks not guarded | Fixed: template creation now guarded by `docker_rootless_setup` |