oblachno-oss/grm
3
0
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects Releases 14 Wiki Activity
Page: Troubleshooting
Pages
Architecture CI-CD-Workflow CLI-Commands Contributing-Guide Decision-Log Development-Setup FAQ Getting-Started Home Installation Testing-Strategy Troubleshooting
Clone
1
Troubleshooting
Devin CI edited this page 2026-08-04 14:04:14 +00:00
Table of Contents

Troubleshooting

Installation Issues

No such file or directory: dockerd-rootless-setuptool.sh (Arch Linux)

Symptom: grm install fails on an Arch Linux host during rootless Docker setup with Error executing command: [Errno 2] No such file or directory: b'dockerd-rootless-setuptool.sh'.

Cause: Arch's docker package does not ship the rootless setup scripts (unlike Debian/Ubuntu's docker-ce-rootless-extras), and rootlesskit is not a dependency of the docker package.

Solution: This is handled automatically by GRM ≥ v0.19.0. Ensure you are on the latest release (git checkout $(git describe --tags --abbrev=0)). The role fetches the scripts from upstream moby/moby and installs rootlesskit automatically. If it still fails, verify the host can reach raw.githubusercontent.com (the script source) and check the Ansible log under ~/.local/state/grm/logs/.

Ansible connection fails (UNREACHABLE)

Symptom: Ansible reports UNREACHABLE when trying to connect to the target host.

Causes and solutions:

  • SSH key not found or wrong path — Verify the key path with --key. The key must be readable by the user running grm.
  • SSH user does not exist on the target — Verify the --user argument. The user must exist on the remote host and have sudo privileges.
  • Host is not reachable — Verify the host IP/hostname with ping and ssh -u <user> <host>.
  • SSH port is not 22 — GRM uses the default SSH port. If your host uses a different port, you may need to configure SSH config (~/.ssh/config) with the appropriate port.

Sudo password prompt fails or is not displayed

Symptom: The sudo password prompt does not appear or the command hangs.

Causes and solutions:

  • Non-interactive session — If running in a CI/CD pipeline or script without a TTY, the password prompt cannot be displayed. Configure passwordless sudo on the remote host and pass --no-ask-become-pass.
  • Wrong sudo password — Ensure you are entering the correct sudo password for the remote user.

GITEA_URL must be set

Symptom: Error message GITEA_URL must be set (or pass --url).

Solution: Set GITEA_URL in your .env file or pass it via --url:

# In .env
GITEA_URL=https://git.example.com

# Or on the command line
grm install 192.168.1.10 --user ubuntu --url https://git.example.com

GITEA_REGISTRATION_TOKEN must be set

Symptom: Error message GITEA_REGISTRATION_TOKEN must be set (or pass --token).

Solution: Set GITEA_REGISTRATION_TOKEN in your .env file or pass it via --token. The token must be a valid registration token from your Gitea instance (it starts with GR).

Runner Issues

Runner appears offline after installation

  • Check that the GITEA_URL and GITEA_REGISTRATION_TOKEN environment variables are correct.
  • Verify the registration token has not expired — generate a new one from Gitea if needed (Site Administration → Actions → Runners → Create Registration Token).
  • Verify the runner service is running: sudo -u grm-<name> systemctl --user status gitea-runner.
  • Check logs for registration errors: sudo -u grm-<name> journalctl --user -u gitea-runner -f.
  • Confirm the runner appears in the Gitea UI under Actions → Runners. If it shows as offline, the runner daemon may not be polling — check network connectivity between the runner host and Gitea.

Runner service fails to start

  • Check the service status: sudo -u grm-<name> systemctl --user status gitea-runner
  • Check logs: sudo -u grm-<name> journalctl --user -u gitea-runner -f
  • Verify the runner binary exists: ls -la /usr/local/bin/gitea_runner
  • Verify the config file exists: ls -la /etc/gitea-runner/<name>/config.yaml
  • Verify the .runner registration file exists: ls -la /var/lib/gitea-runner/<name>/.runner

Rootless Docker: service fails to start

  • Check the service status: sudo -u grm-<name> systemctl --user status gitea-runner.
  • Verify the rootless Docker daemon is running: sudo -u grm-<name> systemctl --user status docker.
  • Verify the Docker socket exists: ls /run/user/$(id -u grm-<name>)/docker.sock.
  • Check logs: sudo -u grm-<name> journalctl --user -u gitea-runner -f.
  • Ensure lingering is enabled for the runner user: loginctl show-user grm-<name> | grep Linger. If not enabled, run sudo loginctl enable-linger grm-<name>.
  • Verify subuid/subgid entries exist: grep grm-<name> /etc/subuid /etc/subgid. If missing, the rootless Docker setup will fail.

Runner not found in registry

Symptom: Error message Runner '<name>' not found in registry. Use 'grm install' first or provide --host and --user.

Solution: The runner was not installed via grm install, or the registry file was deleted. Either:

  1. Install the runner first: grm install <host> --user <user> --name <name>
  2. Or provide explicit connection details: grm status <name> --host <host> --user <user>

Integration Test Issues

Integration test fails

The test checks two things:

  1. .runner file missing or invalid — Registration failed. Check:

    • GITEA_URL and GITEA_REGISTRATION_TOKEN are correct
    • The registration token is valid and has not expired
    • Runner logs for registration errors: sudo -u grm-<name> journalctl --user -u gitea-runner
    • The .runner file should exist at /var/lib/gitea-runner/<name>/.runner
    • The .runner file should contain valid JSON with id, uuid, token, address fields

  2. Service not running — Daemon failed to start. Check:

    • sudo -u grm-<name> systemctl --user status gitea-runner
    • Logs for connection errors: sudo -u grm-<name> journalctl --user -u gitea-runner
    • Verify the rootless Docker daemon is running (see above)

API verification shows error status

If CI_GITEA_TOKEN is set, the integration test queries the Gitea API. If the API returns 401 or 403, the token does not have sufficient permissions. This is informational only and does not affect pass/fail. The test passes as long as the .runner file exists and the systemd service is active.

Logging and Diagnostics

Enable debug logging

GRM_LOG_LEVEL=DEBUG grm install 192.168.1.10 --user ubuntu --name prod-runner

This prints all debug messages to the console. The log file at ~/.local/state/grm/logs/grm.log always captures DEBUG level regardless of this setting.

View Ansible execution logs

Each grm command that invokes Ansible creates a timestamped log file:

ls ~/.local/state/grm/logs/ansible-*.log
cat ~/.local/state/grm/logs/ansible-<timestamp>.log

These logs contain the full Ansible output, including task results, changed/failed counts, and any error messages.

View runner logs on the remote host

# Runner daemon logs
sudo -u grm-<name> journalctl --user -u gitea-runner -f

# Rootless Docker daemon logs
sudo -u grm-<name> journalctl --user -u docker -f

# Docker prune timer logs
sudo -u grm-<name> journalctl --user -u docker-prune.service

Development Issues

"Event loop is closed" warning

This is a harmless cleanup traceback from Molecule's Docker driver when the test process is interrupted. It does not indicate a test failure.

Pre-commit rejects commit message

The pre-commit hook validates that commit messages follow conventional commit format (feat:, fix:, docs:, etc.). The GRM-N: prefix is not allowed on branch commits — use it only in PR titles.

Correct:

feat: add new runner label option

Incorrect:

GRM-33: add new runner label option
update README

make pytest-cov fails with coverage below 100%

Add tests for any new code paths. The coverage requirement is strict (--cov-fail-under=100). Run make pytest-cov locally to see which lines are not covered:

make pytest-cov
# The output shows "Missing" lines for each file

make molecule fails with Docker not available

Molecule requires Docker to be installed and running on your machine. Verify:

docker info  # Should print Docker server info

If Docker is not installed, install it via your package manager or Docker's official installation guide.

Common Issues Reference Table

Symptom Likely Cause Solution
Ansible UNREACHABLE SSH connection failed Verify --user, --key, and host reachability
No such file ... dockerd-rootless-setuptool.sh (Arch) Arch docker pkg lacks rootless scripts + rootlesskit Fixed in v0.19.0; update GRM to latest release
GITEA_URL must be set Missing environment variable Set GITEA_URL in .env or pass --url
GITEA_REGISTRATION_TOKEN must be set Missing environment variable Set GITEA_REGISTRATION_TOKEN in .env or pass --token
Runner appears offline Registration failed or service not running Check GITEA_URL, token validity, and service status
Rootless Docker fails to start subuid/subgid missing or lingering disabled Verify /etc/subuid, /etc/subgid, and loginctl show-user
Runner not found in registry Runner not installed or registry deleted Run grm install or provide --host and --user
Pre-commit rejects commit message Missing conventional format or GRM-N prefix present Use feat: description format without GRM-N:
make molecule fails with runner_name is undefined Verify playbook missing variable Fixed in Phase 1.1; ensure you're on latest master
CI molecule job fails Docker not available on runner host Ensure Gitea runner host has Docker installed and running
Auto-merge doesn't trigger Label not exactly ready-to-merge or CI checks not all green Verify label spelling; check CI status
Vikunja task not updated after merge VIKUNJA_TOKEN expired or task ID missing from commit Regenerate token; verify merge commit has GRM-N: prefix
Post-merge can't find Vikunja task Task not in project 6 or identifier mismatch Verify task exists in Vikunja project 6 with correct identifier
make pytest-cov fails Coverage below 100% Add tests for new code paths
devx.tools.configure_repo fails CI_GITEA_TOKEN missing or invalid Set token with repo admin scope and re-run
configure_repo sets wrong status checks Stale BRANCH_PROTECTION_CONFIG Updated to include (pull_request) suffix; re-run configure_repo
Token visible in ps aux during install Old version passed tokens via command line Fixed: tokens now passed via temp file with 0600 permissions
remove-runner.yml leaves lingering enabled Old version didn't disable lingering Fixed: now runs loginctl disable-linger and removes subuid/subgid
apt cache update always reports changed cache_valid_time: 0 forced update every run Fixed: changed to cache_valid_time: 3600
Prune/service templates created even when docker_rootless_setup: false Template tasks not guarded Fixed: template creation now guarded by docker_rootless_setup