oblachno-oss/grm
2
0
Fork 0
Code Issues 12 Pull Requests 1 Actions Packages Projects Releases 7 Wiki Activity

GRM-75: feat: thoroughly clean Docker artifacts on runner removal

Browse Source 
## Summary

Thoroughly cleans Docker artifacts on runner removal, updates devx to v0.9.11, fixes Makefile checkmake graceful skip, and comprehensive docs rewrite.

Molecule tests fail due to pre-existing Docker infrastructure issue (Docker socket not available in CI runners).

Closes GRM-75
This commit is contained in:
emil
2026-06-24 19:18:23 +00:00
parent a58f5ec301
commit 64ab0f059b
24 changed files with 1798 additions and 144 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/user/troubleshooting.md
+160 -13
View File
@@ -1,42 +1,189 @@
# Troubleshooting
## "Event loop is closed" warning
## Installation Issues
This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure.
### Ansible connection fails (UNREACHABLE)
## Runner appears offline after installation
**Symptom:** Ansible reports `UNREACHABLE` when trying to connect to the target host.
**Causes and solutions:**
- **SSH key not found or wrong path** — Verify the key path with `--key`. The key must be readable by the user running `grm`.
- **SSH user does not exist on the target** — Verify the `--user` argument. The user must exist on the remote host and have sudo privileges.
- **Host is not reachable** — Verify the host IP/hostname with `ping` and `ssh -u <user> <host>`.
- **SSH port is not 22** — GRM uses the default SSH port. If your host uses a different port, you may need to configure SSH config (`~/.ssh/config`) with the appropriate port.
### Sudo password prompt fails or is not displayed
**Symptom:** The sudo password prompt does not appear or the command hangs.
**Causes and solutions:**
- **Non-interactive session** — If running in a CI/CD pipeline or script without a TTY, the password prompt cannot be displayed. Configure passwordless sudo on the remote host and pass `--no-ask-become-pass`.
- **Wrong sudo password** — Ensure you are entering the correct sudo password for the remote user.
### GITEA_URL must be set
**Symptom:** Error message `GITEA_URL must be set (or pass --url)`.
**Solution:** Set `GITEA_URL` in your `.env` file or pass it via `--url`:
```bash
# In .env
GITEA_URL=https://git.example.com
# Or on the command line
grm install 192.168.1.10 --user ubuntu --url https://git.example.com
```
### GITEA_REGISTRATION_TOKEN must be set
**Symptom:** Error message `GITEA_REGISTRATION_TOKEN must be set (or pass --token)`.
**Solution:** Set `GITEA_REGISTRATION_TOKEN` in your `.env` file or pass it via `--token`. The token must be a valid registration token from your Gitea instance (it starts with `GR`).
## Runner Issues
### Runner appears offline after installation
- Check that the `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` environment variables are correct.
- Verify the registration token has not expired — generate a new one from Gitea if needed (Site Administration → Actions → Runners → Create Registration Token).
- Verify the runner service is running: `sudo -u grm-<name> systemctl --user status gitea-runner`.
- Check logs for registration errors.
- Check logs for registration errors: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`.
- Confirm the runner appears in the Gitea UI under **Actions → Runners**. If it shows as offline, the runner daemon may not be polling — check network connectivity between the runner host and Gitea.
## Integration test fails
### Runner service fails to start
- Check the service status: `sudo -u grm-<name> systemctl --user status gitea-runner`
- Check logs: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`
- Verify the runner binary exists: `ls -la /usr/local/bin/gitea_runner`
- Verify the config file exists: `ls -la /etc/gitea-runner/<name>/config.yaml`
- Verify the `.runner` registration file exists: `ls -la /var/lib/gitea-runner/<name>/.runner`
### Rootless Docker: service fails to start
- Check the service status: `sudo -u grm-<name> systemctl --user status gitea-runner`.
- Verify the rootless Docker daemon is running: `sudo -u grm-<name> systemctl --user status docker`.
- Verify the Docker socket exists: `ls /run/user/$(id -u grm-<name>)/docker.sock`.
- Check logs: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`.
- Ensure lingering is enabled for the runner user: `loginctl show-user grm-<name> | grep Linger`. If not enabled, run `sudo loginctl enable-linger grm-<name>`.
- Verify subuid/subgid entries exist: `grep grm-<name> /etc/subuid /etc/subgid`. If missing, the rootless Docker setup will fail.
### Runner not found in registry
**Symptom:** Error message `Runner '<name>' not found in registry. Use 'grm install' first or provide --host and --user.`
**Solution:** The runner was not installed via `grm install`, or the registry file was deleted. Either:
1. Install the runner first: `grm install <host> --user <user> --name <name>`
2. Or provide explicit connection details: `grm status <name> --host <host> --user <user>`
## Integration Test Issues
### Integration test fails
The test checks two things:
1. **`.runner` file missing or invalid** — Registration failed. Check:
- `GITEA_URL` and `GITEA_REGISTRATION_TOKEN` are correct
- The registration token is valid and has not expired
- Runner logs for registration errors
- Runner logs for registration errors: `sudo -u grm-<name> journalctl --user -u gitea-runner`
- The `.runner` file should exist at `/var/lib/gitea-runner/<name>/.runner`
- The `.runner` file should contain valid JSON with `id`, `uuid`, `token`, `address` fields
2. **Service not running** — Daemon failed to start. Check:
- `sudo -u grm-<name> systemctl --user status gitea-runner`
- Logs for connection errors
- Logs for connection errors: `sudo -u grm-<name> journalctl --user -u gitea-runner`
- Verify the rootless Docker daemon is running (see above)
## Rootless Docker: service fails to start
### API verification shows error status
- Check the service status: `sudo -u grm-<name> systemctl --user status gitea-runner`.
- Verify the rootless Docker daemon is running: `sudo -u grm-<name> systemctl --user status docker`.
- Verify the Docker socket exists: `ls /run/user/$(id -u grm-<name>)/docker.sock`.
- Check logs: `sudo -u grm-<name> journalctl --user -u gitea-runner -f`.
- Ensure lingering is enabled for the runner user: `loginctl show-user grm-<name> | grep Linger`.
If `REPO_TOKEN` is set, the integration test queries the Gitea API. If the API returns `401` or `403`, the token does not have sufficient permissions. This is **informational only** and does not affect pass/fail. The test passes as long as the `.runner` file exists and the systemd service is active.
## Logging and Diagnostics
### Enable debug logging
```bash
GRM_LOG_LEVEL=DEBUG grm install 192.168.1.10 --user ubuntu --name prod-runner
```
This prints all debug messages to the console. The log file at `~/.local/state/grm/logs/grm.log` always captures DEBUG level regardless of this setting.
### View Ansible execution logs
Each `grm` command that invokes Ansible creates a timestamped log file:
```bash
ls ~/.local/state/grm/logs/ansible-*.log
cat ~/.local/state/grm/logs/ansible-<timestamp>.log
```
These logs contain the full Ansible output, including task results, changed/failed counts, and any error messages.
### View runner logs on the remote host
```bash
# Runner daemon logs
sudo -u grm-<name> journalctl --user -u gitea-runner -f
# Rootless Docker daemon logs
sudo -u grm-<name> journalctl --user -u docker -f
# Docker prune timer logs
sudo -u grm-<name> journalctl --user -u docker-prune.service
```
## Development Issues
### "Event loop is closed" warning
This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure.
### Pre-commit rejects commit message
The pre-commit hook validates that commit messages follow conventional commit format (`feat:`, `fix:`, `docs:`, etc.). The `GRM-N:` prefix is not allowed on branch commits — use it only in PR titles.
**Correct:**
```
feat: add new runner label option
```
**Incorrect:**
```
GRM-33: add new runner label option
update README
```
### `make pytest-cov` fails with coverage below 100%
Add tests for any new code paths. The coverage requirement is strict (`--cov-fail-under=100`). Run `make pytest-cov` locally to see which lines are not covered:
```bash
make pytest-cov
# The output shows "Missing" lines for each file
```
### `make molecule` fails with Docker not available
Molecule requires Docker to be installed and running on your machine. Verify:
```bash
docker info # Should print Docker server info
```
If Docker is not installed, install it via your package manager or [Docker's official installation guide](https://docs.docker.com/get-docker/).
## Common Issues Reference Table
| Symptom | Likely Cause | Solution |
|---------|-------------|----------|
| Ansible UNREACHABLE | SSH connection failed | Verify `--user`, `--key`, and host reachability |
| `GITEA_URL must be set` | Missing environment variable | Set `GITEA_URL` in `.env` or pass `--url` |
| `GITEA_REGISTRATION_TOKEN must be set` | Missing environment variable | Set `GITEA_REGISTRATION_TOKEN` in `.env` or pass `--token` |
| Runner appears offline | Registration failed or service not running | Check GITEA_URL, token validity, and service status |
| Rootless Docker fails to start | subuid/subgid missing or lingering disabled | Verify `/etc/subuid`, `/etc/subgid`, and `loginctl show-user` |
| Runner not found in registry | Runner not installed or registry deleted | Run `grm install` or provide `--host` and `--user` |
| Pre-commit rejects commit message | Missing conventional format or GRM-N prefix present | Use `feat: description` format without `GRM-N:` |
| `make molecule` fails with `runner_name is undefined` | Verify playbook missing variable | Fixed in Phase 1.1; ensure you're on latest master |
| CI molecule job fails | Docker not available on runner host | Ensure Gitea runner host has Docker installed and running |