# GRM-173: Runner resource leases and scoped disk cleanup

## Problem

Runner hosts protect in-flight work only via name-prefix and age heuristics:

- The healthcheck critical tier runs unfiltered `docker system prune -af --volumes`
  and `docker volume prune -f` — a job's freshly created but momentarily unused
  volume/network can be wiped mid-run, and warm base images are destroyed under
  pressure exactly when they are needed most.
- Container exclusion is a name-prefix (`GITEA-ACTIONS-TASK`) plus a
  `RunningFor` text heuristic — molecule containers owned by a live job are
  protected only by naming convention, not by an ownership claim.
- Cleanup logic is duplicated inline between `docker-prune.service` and the
  healthcheck script — the two paths already diverge (volume pruning exists in
  one tier only).
- There is no admission control: under disk pressure the runner keeps
  accepting new jobs while cleanup races in-flight work.
- The runner config never declares `capacity`, and nothing prevents
  installing a runner on a production host.

## Approach

REQ-1: Define an ownership-lease label contract. Producers tag containers,
images, volumes and networks with `org.oblachno.lease-until` (epoch seconds)
and `org.oblachno.owner` (free-form run/job id). All cleanup paths must never
remove an object whose `lease-until` is in the future; expired leases are
reclaimable. Existing `GITEA-ACTIONS-TASK` name-prefix and `status=exited`
guards are retained for unlabeled objects.

REQ-2: Introduce a single shared cleanup script
(`runner-cleanup.sh`, templated next to the healthcheck script) invoked with
`--tier routine|pressure|critical`. It replaces all inline prune logic in
`docker-prune.service` and the healthcheck. Every prune is scoped: volumes and
networks get `label!=`/`until=` filters at every tier (no unfiltered volume or
network prune remains); images matching `gitea_runner_keep_images` are never
removed, so warm base layers survive critical pressure.

REQ-3: Watermark-tiered behavior. `routine` (timer) prunes aged resources only.
`pressure` (disk >= warn) prunes unowned resources older than 1h. `critical`
(disk >= critical) drops age limits but still honors leases, keep-images, and
never removes running or `GITEA-ACTIONS-TASK` containers.

REQ-4: Admission control under disk pressure. When disk is >= critical and no
`GITEA-ACTIONS-TASK` container is running, the healthcheck writes a marker
file and stops `gitea-runner.service` (the runner simply stops fetching new
jobs). The service is restarted and the marker cleared by a later healthcheck
once disk drops below the warn threshold. In-flight jobs are never killed by
the admission path. Controlled by `gitea_runner_disk_admission_enabled`.

REQ-5: Declare physical-host capacity explicitly:
`runner.capacity: {{ gitea_runner_capacity }}` in the act_runner config
(default 1, matching upstream default).

REQ-6: Production-host exclusion. The role fails early when the target host
carries the production marker file `/etc/oblachno/production-host` or when
`gitea_runner_on_production_host` is true, unless
`gitea_runner_allow_production_host` overrides. Infra-side provisioning of the
marker is a follow-up task.

Historical spec for the colliding task ID:
[GRM-173-skills-historical](GRM-173-skills-historical.md).

## Test Plan

- `template-content` molecule scenario: assert the prune service calls
  `runner-cleanup.sh`, assert lease filters and keep-images logic render in
  the cleanup script, assert `capacity:` renders in the runner config.
- `default` molecule scenario: assert the cleanup script is installed and
  executable.
- `bash -n` syntax check on rendered templates during development.
- `make lint-all`, `make pytest-cov`, fast molecule for the changed role.

## Deploy Plan

- Merge to master via auto-merge; post-merge publishes the package and
  auto-creates the infra dependency-bump PR. Runner hosts pick up the change
  on the next `grm install`/update run — no manual host action.
- Producers emitting lease labels (molecule distribution, CI jobs) are a
  separate devx-side change; until then the guards degrade gracefully to the
  existing name-prefix/age behavior.

## Rollback Plan

- Revert the merge commit; re-run `grm install` to redeploy the previous
  prune/healthcheck units. No persistent state or data migration — the
  marker file under `gitea_runner_data_dir` is removed by the previous
  template's absence (or harmless if left behind).

## Acceptance Criteria

- [x] REQ-1: `org.oblachno.lease-until`/`org.oblachno.owner` labels are honored
  by every cleanup path; valid leases are never removed, expired leases are.
- [x] REQ-2: single shared `runner-cleanup.sh` used by prune service and
  healthcheck; no unfiltered `system prune --volumes`, `volume prune`, or
  `network prune` remains; `gitea_runner_keep_images` never removed.
- [x] REQ-3: three tiers behave as specified (routine/pressure/critical).
- [x] REQ-4: critical pressure with zero in-flight job containers stops
  admission via marker + service stop; recovery resumes automatically.
- [x] REQ-5: `runner.capacity` rendered in `config.yaml`.
- [x] REQ-6: role fails on production-marked hosts unless explicitly allowed.