89 lines
4.4 KiB
Markdown
89 lines
4.4 KiB
Markdown
# GRM-173: Runner resource leases and scoped disk cleanup
|
|
|
|
## Problem
|
|
|
|
Runner hosts protect in-flight work only via name-prefix and age heuristics:
|
|
|
|
- The healthcheck critical tier runs unfiltered `docker system prune -af --volumes`
|
|
and `docker volume prune -f` — a job's momentarily unused volume/network can be
|
|
wiped mid-run, and warm base images are destroyed exactly when needed most.
|
|
- Molecule containers owned by a live job are protected only by the
|
|
`GITEA-ACTIONS-TASK` naming convention, not by an ownership claim.
|
|
- Cleanup logic is duplicated between `docker-prune.service` and the healthcheck.
|
|
- No admission control: under disk pressure the runner keeps accepting jobs
|
|
while cleanup races in-flight work.
|
|
- `capacity` is never declared; nothing prevents installing on a production host.
|
|
|
|
## Approach
|
|
|
|
REQ-1: Define an ownership-lease label contract. Producers tag containers,
|
|
images, volumes and networks with `org.oblachno.lease-until` (epoch seconds)
|
|
and `org.oblachno.owner` (free-form run/job id). All cleanup paths must never
|
|
remove an object whose `lease-until` is in the future; expired leases are
|
|
reclaimable. Existing `GITEA-ACTIONS-TASK` name-prefix and `status=exited`
|
|
guards are retained for unlabeled objects.
|
|
|
|
REQ-2: Introduce a single shared cleanup script (`runner-cleanup.sh`,
|
|
templated next to the healthcheck script) invoked with
|
|
`--tier routine|pressure|critical`, replacing all inline prune logic in
|
|
`docker-prune.service` and the healthcheck. Every prune is scoped (leases,
|
|
`until=` where supported); images matching `gitea_runner_keep_images` are
|
|
never removed, so warm base layers survive critical pressure.
|
|
|
|
REQ-3: Watermark-tiered behavior: `routine` (timer) prunes aged resources;
|
|
`pressure` (disk >= warn) prunes unowned resources older than 1h; `critical`
|
|
(disk >= critical) drops age limits but still honors leases, keep-images, and
|
|
never removes running or `GITEA-ACTIONS-TASK` containers.
|
|
|
|
REQ-4: Admission control under disk pressure. When disk is >= critical and no
|
|
`GITEA-ACTIONS-TASK` container is running, the healthcheck writes a marker file
|
|
and stops `gitea-runner.service` (the runner stops fetching jobs). A later
|
|
healthcheck restarts it once disk drops below warn. In-flight jobs are never
|
|
killed. Controlled by `gitea_runner_disk_admission_enabled`.
|
|
|
|
REQ-5: Declare physical-host capacity explicitly: `runner.capacity:
|
|
{{ gitea_runner_capacity }}` in the act_runner config (default 1 = upstream).
|
|
|
|
REQ-6: Production-host exclusion. The role fails early when the target carries
|
|
the marker file `/etc/oblachno/production-host` or
|
|
`gitea_runner_on_production_host` is true, unless
|
|
`gitea_runner_allow_production_host` overrides. Infra-side marker provisioning
|
|
is a follow-up task.
|
|
|
|
Historical spec for the colliding task ID:
|
|
[GRM-173-skills-historical](GRM-173-skills-historical.md).
|
|
|
|
## Test Plan
|
|
|
|
- `template-content` molecule scenario: prune service calls `runner-cleanup.sh`;
|
|
lease filters, keep-images and `capacity:` render correctly.
|
|
- `default` scenario: cleanup script installed and executable.
|
|
- `bash -n` on rendered templates; `make lint-all`, `make pytest-cov`, fast
|
|
molecule for the changed role.
|
|
|
|
## Deploy Plan
|
|
|
|
- Merge via auto-merge; post-merge publishes the package and auto-creates the
|
|
infra dependency-bump PR. Runner hosts pick up the change on the next
|
|
`grm install`/update run — no manual host action.
|
|
- Producer-side lease emission (molecule/CI jobs) is a separate devx change;
|
|
until then the guards degrade to the existing name-prefix/age behavior.
|
|
|
|
## Rollback Plan
|
|
|
|
- Revert the merge commit and re-run `grm install` to redeploy the previous
|
|
prune/healthcheck units. No persistent state or migration.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [x] REQ-1: `org.oblachno.lease-until`/`org.oblachno.owner` labels are honored
|
|
by every cleanup path; valid leases are never removed, expired leases are.
|
|
- [x] REQ-2: single shared `runner-cleanup.sh` used by prune service and
|
|
healthcheck; no unfiltered `system prune --volumes`, `volume prune`, or
|
|
`network prune` remains; `gitea_runner_keep_images` never removed.
|
|
- [x] REQ-3: three tiers behave as specified (routine/pressure/critical).
|
|
- [x] REQ-4: critical pressure with zero in-flight job containers stops
|
|
admission via marker + service stop; recovery resumes automatically.
|
|
- [x] REQ-5: `runner.capacity` rendered in `config.yaml`.
|
|
- [x] REQ-6: role fails on production-marked hosts unless explicitly allowed.
|