209 lines
11 KiB
Markdown
209 lines
11 KiB
Markdown
# Troubleshooting
|
|
|
|
## Installation Issues
|
|
|
|
### `No such file or directory: dockerd-rootless-setuptool.sh` (Arch Linux)
|
|
|
|
**Symptom:** `grm install` fails on an Arch Linux host during rootless Docker setup with `Error executing command: [Errno 2] No such file or directory: b'dockerd-rootless-setuptool.sh'`.
|
|
|
|
**Cause:** Arch's `docker` package does not ship the rootless setup scripts (unlike Debian/Ubuntu's `docker-ce-rootless-extras`), and `rootlesskit` is not a dependency of the `docker` package.
|
|
|
|
**Solution:** This is handled automatically by GRM ≥ v0.19.0. Ensure you are on the latest release (`git checkout $(git describe --tags --abbrev=0)`). The role fetches the scripts from upstream `moby/moby` and installs `rootlesskit` automatically. If it still fails, verify the host can reach `raw.githubusercontent.com` (the script source) and check the Ansible log under `~/.local/state/grm/logs/`.
|
|
|
|
### Ansible connection fails (UNREACHABLE)
|
|
|
|
**Symptom:** Ansible reports `UNREACHABLE` when trying to connect to the target host.
|
|
|
|
**Causes and solutions:**
|
|
|
|
- **SSH key not found or wrong path** — Verify the key path with `--key`. The key must be readable by the user running `grm`.
|
|
- **SSH user does not exist on the target** — Verify the `--user` argument. The user must exist on the remote host and have sudo privileges.
|
|
- **Host is not reachable** — Verify the host IP/hostname with `ping` and `ssh -u <user> <host>`.
|
|
- **SSH port is not 22** — GRM uses the default SSH port. If your host uses a different port, you may need to configure SSH config (`~/.ssh/config`) with the appropriate port.
|
|
|
|
### Sudo password prompt fails or is not displayed
|
|
|
|
**Symptom:** The sudo password prompt does not appear or the command hangs.
|
|
|
|
**Causes and solutions:**
|
|
|
|
- **Non-interactive session** — If running in a CI/CD pipeline or script without a TTY, the password prompt cannot be displayed. Configure passwordless sudo on the remote host and pass `--no-ask-become-pass`.
|
|
- **Wrong sudo password** — Ensure you are entering the correct sudo password for the remote user.
|
|
|
|
### GITEA_URL must be set
|
|
|
|
**Symptom:** Error message `GITEA_URL must be set (or pass --url)`.
|
|
|
|
**Solution:** Set `GITEA_URL` in your `.env` file or pass it via `--url`:
|
|
|
|
```bash
|
|
# In .env
|
|
GITEA_URL=https://git.example.com
|
|
|
|
# Or on the command line
|
|
grm install 192.168.1.10 --user ubuntu --url https://git.example.com
|
|
```
|
|
|
|
### GITEA_REGISTRATION_TOKEN must be set
|
|
|
|
**Symptom:** Error message `GITEA_REGISTRATION_TOKEN must be set (or pass --token)`.
|
|
|
|
**Solution:** Set `GITEA_REGISTRATION_TOKEN` in your `.env` file or pass it via `--token`. The token must be a valid registration token from your Gitea instance (it starts with `GR`).
|
|
|
|
## Runner Issues
|
|
|
|
### Runner appears offline after installation
|
|
|
|
- Check that the `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` environment variables are correct.
|
|
- Verify the registration token has not expired — generate a new one from Gitea if needed (Site Administration → Actions → Runners → Create Registration Token).
|
|
- Verify the runner service is running: `sudo -u grm-<name> systemctl --user status gitea-runner`.
|
|
- Check logs for registration errors: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`.
|
|
- Confirm the runner appears in the Gitea UI under **Actions → Runners**. If it shows as offline, the runner daemon may not be polling — check network connectivity between the runner host and Gitea.
|
|
|
|
### Runner service fails to start
|
|
|
|
- Check the service status: `sudo -u grm-<name> systemctl --user status gitea-runner`
|
|
- Check logs: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`
|
|
- Verify the runner binary exists: `ls -la /usr/local/bin/gitea_runner`
|
|
- Verify the config file exists: `ls -la /etc/gitea-runner/<name>/config.yaml`
|
|
- Verify the `.runner` registration file exists: `ls -la /var/lib/gitea-runner/<name>/.runner`
|
|
|
|
### Rootless Docker: service fails to start
|
|
|
|
- Check the service status: `sudo -u grm-<name> systemctl --user status gitea-runner`.
|
|
- Verify the rootless Docker daemon is running: `sudo -u grm-<name> systemctl --user status docker`.
|
|
- Verify the Docker socket exists: `ls /run/user/$(id -u grm-<name>)/docker.sock`.
|
|
- Check logs: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`.
|
|
- Ensure lingering is enabled for the runner user: `loginctl show-user grm-<name> | grep Linger`. If not enabled, run `sudo loginctl enable-linger grm-<name>`.
|
|
- Verify subuid/subgid entries exist: `grep grm-<name> /etc/subuid /etc/subgid`. If missing, the rootless Docker setup will fail.
|
|
|
|
### Runner not found in registry
|
|
|
|
**Symptom:** Error message `Runner '<name>' not found in registry. Use 'grm install' first or provide --host and --user.`
|
|
|
|
**Solution:** The runner was not installed via `grm install`, or the registry file was deleted. Either:
|
|
|
|
1. Install the runner first: `grm install <host> --user <user> --name <name>`
|
|
2. Or provide explicit connection details: `grm status <name> --host <host> --user <user>`
|
|
|
|
## Integration Test Issues
|
|
|
|
### Integration test fails
|
|
|
|
The test checks two things:
|
|
|
|
1. **`.runner` file missing or invalid** — Registration failed. Check:
|
|
- `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` are correct
|
|
- The registration token is valid and has not expired
|
|
- Runner logs for registration errors: `sudo -u grm-<name> journalctl --user -u gitea-runner`
|
|
- The `.runner` file should exist at `/var/lib/gitea-runner/<name>/.runner`
|
|
- The `.runner` file should contain valid JSON with `id`, `uuid`, `token`, `address` fields
|
|
|
|
2. **Service not running** — Daemon failed to start. Check:
|
|
- `sudo -u grm-<name> systemctl --user status gitea-runner`
|
|
- Logs for connection errors: `sudo -u grm-<name> journalctl --user -u gitea-runner`
|
|
- Verify the rootless Docker daemon is running (see above)
|
|
|
|
### API verification shows error status
|
|
|
|
If `CI_GITEA_TOKEN` is set, the integration test queries the Gitea API. If the API returns `401` or `403`, the token does not have sufficient permissions. This is **informational only** and does not affect pass/fail. The test passes as long as the `.runner` file exists and the systemd service is active.
|
|
|
|
## Logging and Diagnostics
|
|
|
|
### Enable debug logging
|
|
|
|
```bash
|
|
GRM_LOG_LEVEL=DEBUG grm install 192.168.1.10 --user ubuntu --name prod-runner
|
|
```
|
|
|
|
This prints all debug messages to the console. The log file at `~/.local/state/grm/logs/grm.log` always captures DEBUG level regardless of this setting.
|
|
|
|
### View Ansible execution logs
|
|
|
|
Each `grm` command that invokes Ansible creates a timestamped log file:
|
|
|
|
```bash
|
|
ls ~/.local/state/grm/logs/ansible-*.log
|
|
cat ~/.local/state/grm/logs/ansible-<timestamp>.log
|
|
```
|
|
|
|
These logs contain the full Ansible output, including task results, changed/failed counts, and any error messages.
|
|
|
|
### View runner logs on the remote host
|
|
|
|
```bash
|
|
# Runner daemon logs
|
|
sudo -u grm-<name> journalctl --user -u gitea-runner -f
|
|
|
|
# Rootless Docker daemon logs
|
|
sudo -u grm-<name> journalctl --user -u docker -f
|
|
|
|
# Docker prune timer logs
|
|
sudo -u grm-<name> journalctl --user -u docker-prune.service
|
|
```
|
|
|
|
## Development Issues
|
|
|
|
### "Event loop is closed" warning
|
|
|
|
This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure.
|
|
|
|
### Pre-commit rejects commit message
|
|
|
|
The pre-commit hook validates that commit messages follow conventional commit format (`feat:`, `fix:`, `docs:`, etc.). The `GRM-N:` prefix is not allowed on branch commits — use it only in PR titles.
|
|
|
|
**Correct:**
|
|
```text
|
|
feat: add new runner label option
|
|
```
|
|
|
|
**Incorrect:**
|
|
```text
|
|
GRM-33: add new runner label option
|
|
update README
|
|
```
|
|
|
|
### `make pytest-cov` fails with coverage below 100%
|
|
|
|
Add tests for any new code paths. The coverage requirement is strict (`--cov-fail-under=100`). Run `make pytest-cov` locally to see which lines are not covered:
|
|
|
|
```bash
|
|
make pytest-cov
|
|
# The output shows "Missing" lines for each file
|
|
```
|
|
|
|
### `make molecule` fails with Docker not available
|
|
|
|
Molecule requires Docker to be installed and running on your machine. Verify:
|
|
|
|
```bash
|
|
docker info # Should print Docker server info
|
|
```
|
|
|
|
If Docker is not installed, install it via your package manager or [Docker's official installation guide](https://docs.docker.com/get-docker/).
|
|
|
|
## Common Issues Reference Table
|
|
|
|
| Symptom | Likely Cause | Solution |
|
|
|---------|-------------|----------|
|
|
| Ansible UNREACHABLE | SSH connection failed | Verify `--user`, `--key`, and host reachability |
|
|
| `No such file ... dockerd-rootless-setuptool.sh` (Arch) | Arch `docker` pkg lacks rootless scripts + `rootlesskit` | Fixed in v0.19.0; update GRM to latest release |
|
|
| `GITEA_URL must be set` | Missing environment variable | Set `GITEA_URL` in `.env` or pass `--url` |
|
|
| `GITEA_REGISTRATION_TOKEN must be set` | Missing environment variable | Set `GITEA_REGISTRATION_TOKEN` in `.env` or pass `--token` |
|
|
| Runner appears offline | Registration failed or service not running | Check GITEA_URL, token validity, and service status |
|
|
| Rootless Docker fails to start | subuid/subgid missing or lingering disabled | Verify `/etc/subuid`, `/etc/subgid`, and `loginctl show-user` |
|
|
| Runner not found in registry | Runner not installed or registry deleted | Run `grm install` or provide `--host` and `--user` |
|
|
| Pre-commit rejects commit message | Missing conventional format or GRM-N prefix present | Use `feat: description` format without `GRM-N:` |
|
|
| `make molecule` fails with `runner_name is undefined` | Verify playbook missing variable | Fixed in Phase 1.1; ensure you're on latest master |
|
|
| CI molecule job fails | Docker not available on runner host | Ensure Gitea runner host has Docker installed and running |
|
|
| Auto-merge doesn't trigger | Label not exactly `ready-to-merge` or CI checks not all green | Verify label spelling; check CI status |
|
|
| Vikunja task not updated after merge | VIKUNJA_TOKEN expired or task ID missing from commit | Regenerate token; verify merge commit has `GRM-N:` prefix |
|
|
| Post-merge can't find Vikunja task | Task not in project 6 or identifier mismatch | Verify task exists in Vikunja project 6 with correct identifier |
|
|
| `make pytest-cov` fails | Coverage below 100% | Add tests for new code paths |
|
|
| `devx.tools.configure_repo` fails | CI_GITEA_TOKEN missing or invalid | Set token with repo admin scope and re-run |
|
|
| `configure_repo` sets wrong status checks | Stale `BRANCH_PROTECTION_CONFIG` | Updated to include `(pull_request)` suffix; re-run `configure_repo` |
|
|
| Token visible in `ps aux` during install | Old version passed tokens via command line | Fixed: tokens now passed via temp file with `0600` permissions |
|
|
| `remove-runner.yml` leaves lingering enabled | Old version didn't disable lingering | Fixed: now runs `loginctl disable-linger` and removes subuid/subgid |
|
|
| apt cache update always reports `changed` | `cache_valid_time: 0` forced update every run | Fixed: changed to `cache_valid_time: 3600` |
|
|
| Prune/service templates created even when `docker_rootless_setup: false` | Template tasks not guarded | Fixed: template creation now guarded by `docker_rootless_setup` |
|