GRM-173: Runner resource leases and scoped disk cleanup

Problem

Runner hosts protect in-flight work only via name-prefix and age heuristics:

The healthcheck critical tier runs unfiltered docker system prune -af --volumes and docker volume prune -f — a job's momentarily unused volume/network can be wiped mid-run, and warm base images are destroyed exactly when needed most.
Molecule containers owned by a live job are protected only by the GITEA-ACTIONS-TASK naming convention, not by an ownership claim.
Cleanup logic is duplicated between docker-prune.service and the healthcheck.
No admission control: under disk pressure the runner keeps accepting jobs while cleanup races in-flight work.
capacity is never declared; nothing prevents installing on a production host.

Approach

REQ-1: Define an ownership-lease label contract. Producers tag containers, images, volumes and networks with org.oblachno.lease-until (epoch seconds) and org.oblachno.owner (free-form run/job id). All cleanup paths must never remove an object whose lease-until is in the future; expired leases are reclaimable. Existing GITEA-ACTIONS-TASK name-prefix and status=exited guards are retained for unlabeled objects.

REQ-2: Introduce a single shared cleanup script (runner-cleanup.sh, templated next to the healthcheck script) invoked with --tier routine|pressure|critical, replacing all inline prune logic in docker-prune.service and the healthcheck. Every prune is scoped (leases, until= where supported); images matching gitea_runner_keep_images are never removed, so warm base layers survive critical pressure.

REQ-3: Watermark-tiered behavior: routine (timer) prunes aged resources; pressure (disk >= warn) prunes unowned resources older than 1h; critical (disk >= critical) drops age limits but still honors leases, keep-images, and never removes running or GITEA-ACTIONS-TASK containers.

REQ-4: Admission control under disk pressure. When disk is >= critical and no GITEA-ACTIONS-TASK container is running, the healthcheck writes a marker file and stops gitea-runner.service (the runner stops fetching jobs). A later healthcheck restarts it once disk drops below warn. In-flight jobs are never killed. Controlled by gitea_runner_disk_admission_enabled.

REQ-5: Declare physical-host capacity explicitly: runner.capacity: {{ gitea_runner_capacity }} in the act_runner config (default 1 = upstream).

REQ-6: Production-host exclusion. The role fails early when the target carries the marker file /etc/oblachno/production-host or gitea_runner_on_production_host is true, unless gitea_runner_allow_production_host overrides. Infra-side marker provisioning is a follow-up task.

Historical spec for the colliding task ID: GRM-173-skills-historical.

Test Plan

template-content molecule scenario: prune service calls runner-cleanup.sh; lease filters, keep-images and capacity: render correctly.
default scenario: cleanup script installed and executable.
bash -n on rendered templates; make lint-all, make pytest-cov, fast molecule for the changed role.

Deploy Plan

Merge via auto-merge; post-merge publishes the package and auto-creates the infra dependency-bump PR. Runner hosts pick up the change on the next grm install/update run — no manual host action.
Producer-side lease emission (molecule/CI jobs) is a separate devx change; until then the guards degrade to the existing name-prefix/age behavior.

Rollback Plan

Revert the merge commit and re-run grm install to redeploy the previous prune/healthcheck units. No persistent state or migration.

Acceptance Criteria

REQ-1: org.oblachno.lease-until/org.oblachno.owner labels are honored by every cleanup path; valid leases are never removed, expired leases are.
REQ-2: single shared runner-cleanup.sh used by prune service and healthcheck; no unfiltered system prune --volumes, volume prune, or network prune remains; gitea_runner_keep_images never removed.
REQ-3: three tiers behave as specified (routine/pressure/critical).
REQ-4: critical pressure with zero in-flight job containers stops admission via marker + service stop; recovery resumes automatically.
REQ-5: runner.capacity rendered in config.yaml.
REQ-6: role fails on production-marked hosts unless explicitly allowed.

4.4 KiB Raw Blame History