Emil Simeonov
3c0696f3f2
Extend stopped-container protection to an explicit lease contract (org.oblachno.lease-until / org.oblachno.owner) honored by every cleanup path. Consolidate the duplicated inline prune logic from docker-prune and the healthcheck into a single tiered runner-cleanup.sh; remove the unfiltered `system prune -af --volumes` / `volume prune` paths that could wipe a job's volumes mid-run, and keep warm base images under pressure. At critical disk usage the healthcheck now stops admitting new work (stops gitea-runner.service once no CI job is in flight) and resumes it automatically after recovery. The runner config declares capacity, and the role refuses to install on production-marked hosts.
102 lines
5.1 KiB
Markdown
102 lines
5.1 KiB
Markdown
# GRM-173: Runner resource leases and scoped disk cleanup
|
|
|
|
## Problem
|
|
|
|
Runner hosts protect in-flight work only via name-prefix and age heuristics:
|
|
|
|
- The healthcheck critical tier runs unfiltered `docker system prune -af --volumes`
|
|
and `docker volume prune -f` — a job's freshly created but momentarily unused
|
|
volume/network can be wiped mid-run, and warm base images are destroyed under
|
|
pressure exactly when they are needed most.
|
|
- Container exclusion is a name-prefix (`GITEA-ACTIONS-TASK`) plus a
|
|
`RunningFor` text heuristic — molecule containers owned by a live job are
|
|
protected only by naming convention, not by an ownership claim.
|
|
- Cleanup logic is duplicated inline between `docker-prune.service` and the
|
|
healthcheck script — the two paths already diverge (volume pruning exists in
|
|
one tier only).
|
|
- There is no admission control: under disk pressure the runner keeps
|
|
accepting new jobs while cleanup races in-flight work.
|
|
- The runner config never declares `capacity`, and nothing prevents
|
|
installing a runner on a production host.
|
|
|
|
## Approach
|
|
|
|
REQ-1: Define an ownership-lease label contract. Producers tag containers,
|
|
images, volumes and networks with `org.oblachno.lease-until` (epoch seconds)
|
|
and `org.oblachno.owner` (free-form run/job id). All cleanup paths must never
|
|
remove an object whose `lease-until` is in the future; expired leases are
|
|
reclaimable. Existing `GITEA-ACTIONS-TASK` name-prefix and `status=exited`
|
|
guards are retained for unlabeled objects.
|
|
|
|
REQ-2: Introduce a single shared cleanup script
|
|
(`runner-cleanup.sh`, templated next to the healthcheck script) invoked with
|
|
`--tier routine|pressure|critical`. It replaces all inline prune logic in
|
|
`docker-prune.service` and the healthcheck. Every prune is scoped: volumes and
|
|
networks get `label!=`/`until=` filters at every tier (no unfiltered volume or
|
|
network prune remains); images matching `gitea_runner_keep_images` are never
|
|
removed, so warm base layers survive critical pressure.
|
|
|
|
REQ-3: Watermark-tiered behavior. `routine` (timer) prunes aged resources only.
|
|
`pressure` (disk >= warn) prunes unowned resources older than 1h. `critical`
|
|
(disk >= critical) drops age limits but still honors leases, keep-images, and
|
|
never removes running or `GITEA-ACTIONS-TASK` containers.
|
|
|
|
REQ-4: Admission control under disk pressure. When disk is >= critical and no
|
|
`GITEA-ACTIONS-TASK` container is running, the healthcheck writes a marker
|
|
file and stops `gitea-runner.service` (the runner simply stops fetching new
|
|
jobs). The service is restarted and the marker cleared by a later healthcheck
|
|
once disk drops below the warn threshold. In-flight jobs are never killed by
|
|
the admission path. Controlled by `gitea_runner_disk_admission_enabled`.
|
|
|
|
REQ-5: Declare physical-host capacity explicitly:
|
|
`runner.capacity: {{ gitea_runner_capacity }}` in the act_runner config
|
|
(default 1, matching upstream default).
|
|
|
|
REQ-6: Production-host exclusion. The role fails early when the target host
|
|
carries the production marker file `/etc/oblachno/production-host` or when
|
|
`gitea_runner_on_production_host` is true, unless
|
|
`gitea_runner_allow_production_host` overrides. Infra-side provisioning of the
|
|
marker is a follow-up task.
|
|
|
|
Historical spec for the colliding task ID:
|
|
[GRM-173-skills-historical](GRM-173-skills-historical.md).
|
|
|
|
## Test Plan
|
|
|
|
- `template-content` molecule scenario: assert the prune service calls
|
|
`runner-cleanup.sh`, assert lease filters and keep-images logic render in
|
|
the cleanup script, assert `capacity:` renders in the runner config.
|
|
- `default` molecule scenario: assert the cleanup script is installed and
|
|
executable.
|
|
- `bash -n` syntax check on rendered templates during development.
|
|
- `make lint-all`, `make pytest-cov`, fast molecule for the changed role.
|
|
|
|
## Deploy Plan
|
|
|
|
- Merge to master via auto-merge; post-merge publishes the package and
|
|
auto-creates the infra dependency-bump PR. Runner hosts pick up the change
|
|
on the next `grm install`/update run — no manual host action.
|
|
- Producers emitting lease labels (molecule distribution, CI jobs) are a
|
|
separate devx-side change; until then the guards degrade gracefully to the
|
|
existing name-prefix/age behavior.
|
|
|
|
## Rollback Plan
|
|
|
|
- Revert the merge commit; re-run `grm install` to redeploy the previous
|
|
prune/healthcheck units. No persistent state or data migration — the
|
|
marker file under `gitea_runner_data_dir` is removed by the previous
|
|
template's absence (or harmless if left behind).
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [x] REQ-1: `org.oblachno.lease-until`/`org.oblachno.owner` labels are honored
|
|
by every cleanup path; valid leases are never removed, expired leases are.
|
|
- [x] REQ-2: single shared `runner-cleanup.sh` used by prune service and
|
|
healthcheck; no unfiltered `system prune --volumes`, `volume prune`, or
|
|
`network prune` remains; `gitea_runner_keep_images` never removed.
|
|
- [x] REQ-3: three tiers behave as specified (routine/pressure/critical).
|
|
- [x] REQ-4: critical pressure with zero in-flight job containers stops
|
|
admission via marker + service stop; recovery resumes automatically.
|
|
- [x] REQ-5: `runner.capacity` rendered in `config.yaml`.
|
|
- [x] REQ-6: role fails on production-marked hosts unless explicitly allowed.
|