grm/docs/specs/GRM-173.md

GRM-173: Runner resource leases and scoped disk cleanup

Problem

Runner hosts protect in-flight work only via name-prefix and age heuristics:

• The healthcheck critical tier runs unfiltered docker system prune -af --volumes and docker volume prune -f — a job's freshly created but momentarily unused volume/network can be wiped mid-run, and warm base images are destroyed under pressure exactly when they are needed most.
• Container exclusion is a name-prefix (GITEA-ACTIONS-TASK) plus a RunningFor text heuristic — molecule containers owned by a live job are protected only by naming convention, not by an ownership claim.
• Cleanup logic is duplicated inline between docker-prune.service and the healthcheck script — the two paths already diverge (volume pruning exists in one tier only).
• There is no admission control: under disk pressure the runner keeps accepting new jobs while cleanup races in-flight work.
• The runner config never declares capacity, and nothing prevents installing a runner on a production host.

Approach

REQ-1: Define an ownership-lease label contract. Producers tag containers, images, volumes and networks with org.oblachno.lease-until (epoch seconds) and org.oblachno.owner (free-form run/job id). All cleanup paths must never remove an object whose lease-until is in the future; expired leases are reclaimable. Existing GITEA-ACTIONS-TASK name-prefix and status=exited guards are retained for unlabeled objects.

REQ-2: Introduce a single shared cleanup script (runner-cleanup.sh, templated next to the healthcheck script) invoked with --tier routine|pressure|critical. It replaces all inline prune logic in docker-prune.service and the healthcheck. Every prune is scoped: volumes and networks get label!=/until= filters at every tier (no unfiltered volume or network prune remains); images matching gitea_runner_keep_images are never removed, so warm base layers survive critical pressure.

REQ-3: Watermark-tiered behavior. routine (timer) prunes aged resources only. pressure (disk >= warn) prunes unowned resources older than 1h. critical (disk >= critical) drops age limits but still honors leases, keep-images, and never removes running or GITEA-ACTIONS-TASK containers.

REQ-4: Admission control under disk pressure. When disk is >= critical and no GITEA-ACTIONS-TASK container is running, the healthcheck writes a marker file and stops gitea-runner.service (the runner simply stops fetching new jobs). The service is restarted and the marker cleared by a later healthcheck once disk drops below the warn threshold. In-flight jobs are never killed by the admission path. Controlled by gitea_runner_disk_admission_enabled.

REQ-5: Declare physical-host capacity explicitly: runner.capacity: {{ gitea_runner_capacity }} in the act_runner config (default 1, matching upstream default).

REQ-6: Production-host exclusion. The role fails early when the target host carries the production marker file /etc/oblachno/production-host or when gitea_runner_on_production_host is true, unless gitea_runner_allow_production_host overrides. Infra-side provisioning of the marker is a follow-up task.

Historical spec for the colliding task ID: GRM-173-skills-historical.

Test Plan

• template-content molecule scenario: assert the prune service calls runner-cleanup.sh, assert lease filters and keep-images logic render in the cleanup script, assert capacity: renders in the runner config.
• default molecule scenario: assert the cleanup script is installed and executable.
• bash -n syntax check on rendered templates during development.
• make lint-all, make pytest-cov, fast molecule for the changed role.

Deploy Plan

• Merge to master via auto-merge; post-merge publishes the package and auto-creates the infra dependency-bump PR. Runner hosts pick up the change on the next grm install/update run — no manual host action.
• Producers emitting lease labels (molecule distribution, CI jobs) are a separate devx-side change; until then the guards degrade gracefully to the existing name-prefix/age behavior.

Rollback Plan

• Revert the merge commit; re-run grm install to redeploy the previous prune/healthcheck units. No persistent state or data migration — the marker file under gitea_runner_data_dir is removed by the previous template's absence (or harmless if left behind).

Acceptance Criteria

• REQ-1: org.oblachno.lease-until/org.oblachno.owner labels are honored by every cleanup path; valid leases are never removed, expired leases are.
• REQ-2: single shared runner-cleanup.sh used by prune service and healthcheck; no unfiltered system prune --volumes, volume prune, or network prune remains; gitea_runner_keep_images never removed.
• REQ-3: three tiers behave as specified (routine/pressure/critical).
• REQ-4: critical pressure with zero in-flight job containers stops admission via marker + service stop; recovery resumes automatically.
• REQ-5: runner.capacity rendered in config.yaml.
• REQ-6: role fails on production-marked hosts unless explicitly allowed.