# GRM-173: Runner resource leases and scoped disk cleanup

## Problem

Runner hosts protect in-flight work only via name-prefix and age heuristics:

- The healthcheck critical tier runs unfiltered `docker system prune -af --volumes`
  and `docker volume prune -f` — a job's momentarily unused volume/network can be
  wiped mid-run, and warm base images are destroyed exactly when needed most.
- Molecule containers owned by a live job are protected only by the
  `GITEA-ACTIONS-TASK` naming convention, not by an ownership claim.
- Cleanup logic is duplicated between `docker-prune.service` and the healthcheck.
- No admission control: under disk pressure the runner keeps accepting jobs
  while cleanup races in-flight work.
- `capacity` is never declared; nothing prevents installing on a production host.

## Approach

REQ-1: Define an ownership-lease label contract. Producers tag containers,
images, volumes and networks with `org.oblachno.lease-until` (epoch seconds)
and `org.oblachno.owner` (free-form run/job id). All cleanup paths must never
remove an object whose `lease-until` is in the future; expired leases are
reclaimable. Existing `GITEA-ACTIONS-TASK` name-prefix and `status=exited`
guards are retained for unlabeled objects.

REQ-2: Introduce a single shared cleanup script (`runner-cleanup.sh`,
templated next to the healthcheck script) invoked with
`--tier routine|pressure|critical`, replacing all inline prune logic in
`docker-prune.service` and the healthcheck. Every prune is scoped (leases,
`until=` where supported); images matching `gitea_runner_keep_images` are
never removed, so warm base layers survive critical pressure.

REQ-3: Watermark-tiered behavior: `routine` (timer) prunes aged resources;
`pressure` (disk >= warn) prunes unowned resources older than 1h; `critical`
(disk >= critical) drops age limits but still honors leases, keep-images, and
never removes running or `GITEA-ACTIONS-TASK` containers.

REQ-4: Admission control under disk pressure. When disk is >= critical and no
`GITEA-ACTIONS-TASK` container is running, the healthcheck writes a marker file
and stops `gitea-runner.service` (the runner stops fetching jobs). A later
healthcheck restarts it once disk drops below warn. In-flight jobs are never
killed. Controlled by `gitea_runner_disk_admission_enabled`.

REQ-5: Declare physical-host capacity explicitly: `runner.capacity:
{{ gitea_runner_capacity }}` in the act_runner config (default 1 = upstream).

REQ-6: Production-host exclusion. The role fails early when the target carries
the marker file `/etc/oblachno/production-host` or
`gitea_runner_on_production_host` is true, unless
`gitea_runner_allow_production_host` overrides. Infra-side marker provisioning
is a follow-up task.

Historical spec for the colliding task ID:
[GRM-173-skills-historical](GRM-173-skills-historical.md).

## Test Plan

- `template-content` molecule scenario: prune service calls `runner-cleanup.sh`;
  lease filters, keep-images and `capacity:` render correctly.
- `default` scenario: cleanup script installed and executable.
- `bash -n` on rendered templates; `make lint-all`, `make pytest-cov`, fast
  molecule for the changed role.

## Deploy Plan

- Merge via auto-merge; post-merge publishes the package and auto-creates the
  infra dependency-bump PR. Runner hosts pick up the change on the next
  `grm install`/update run — no manual host action.
- Producer-side lease emission (molecule/CI jobs) is a separate devx change;
  until then the guards degrade to the existing name-prefix/age behavior.

## Rollback Plan

- Revert the merge commit and re-run `grm install` to redeploy the previous
  prune/healthcheck units. No persistent state or migration.

## Acceptance Criteria

- [x] REQ-1: `org.oblachno.lease-until`/`org.oblachno.owner` labels are honored
  by every cleanup path; valid leases are never removed, expired leases are.
- [x] REQ-2: single shared `runner-cleanup.sh` used by prune service and
  healthcheck; no unfiltered `system prune --volumes`, `volume prune`, or
  `network prune` remains; `gitea_runner_keep_images` never removed.
- [x] REQ-3: three tiers behave as specified (routine/pressure/critical).
- [x] REQ-4: critical pressure with zero in-flight job containers stops
  admission via marker + service stop; recovery resumes automatically.
- [x] REQ-5: `runner.capacity` rendered in `config.yaml`.
- [x] REQ-6: role fails on production-marked hosts unless explicitly allowed.