GRM-173: Runner resource leases and scoped disk cleanup #282

Merged
kireto merged 8 commits from GRM-173-runner-resource-leases into master 2026-09-22 17:07:50 +00:00
Member
No description provided.
kireto added 2 commits 2026-09-22 16:40:26 +00:00
Extend stopped-container protection to an explicit lease contract
(org.oblachno.lease-until / org.oblachno.owner) honored by every cleanup
path. Consolidate the duplicated inline prune logic from docker-prune
and the healthcheck into a single tiered runner-cleanup.sh; remove the
unfiltered `system prune -af --volumes` / `volume prune` paths that could
wipe a job's volumes mid-run, and keep warm base images under pressure.

At critical disk usage the healthcheck now stops admitting new work
(stops gitea-runner.service once no CI job is in flight) and resumes it
automatically after recovery. The runner config declares capacity, and
the role refuses to install on production-marked hosts.
test: fix cleanup-script assertion (tier flag lives in unit file)
CI / validate (pull_request) Successful in 1m7s
CI / molecule-tests (1) (pull_request) Successful in 4m47s
CI / molecule-tests (4) (pull_request) Successful in 6m28s
CI / molecule-tests (2) (pull_request) Successful in 6m38s
CI / molecule-tests (3) (pull_request) Successful in 9m6s
CI / auto-merge (pull_request) Failing after 9m27s
c3283eedc7
emil approved these changes 2026-09-22 16:42:28 +00:00
Dismissed
emil left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
emil added 1 commit 2026-09-22 16:48:47 +00:00
fix: honor real docker filter/formatter support in cleanup script
CI / validate (pull_request) Failing after 1m34s
CI / molecule-tests (1) (pull_request) Skipped
CI / molecule-tests (2) (pull_request) Skipped
CI / molecule-tests (3) (pull_request) Skipped
CI / molecule-tests (4) (pull_request) Skipped
CI / auto-merge (pull_request) Skipped
8456a3497c
`docker images` has no label formatter — check leases via image inspect
per candidate. `label!=` is invalid on volume/network ls but valid on
prune; `until=` is unsupported for volumes — drop it. Expired-lease
volumes/networks are reclaimed by an explicit inspect pass; anonymous
volumes only at routine/pressure, all unused volumes at critical.
emil dismissed emil's review 2026-09-22 16:48:47 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

Owner

Deep Review Summary

  • Files reviewed: 10 (spec, 4 templates, 2 task files, defaults, verify.yml, historical-spec rename)
  • Issues found: 3 (all auto-fixed during development, verified against a live Docker daemon)

Fixed during review pass:

  1. docker images has no .Label/.Labels formatter — the image lease check now uses docker image inspect per candidate (previous version would have silently pruned nothing).
  2. label!= is invalid on volume ls/network ls but valid on the prune endpoints; until= is unsupported for volume prune (dropped — silent-failure risk).
  3. Expired-lease volumes/networks needed an explicit reclaim pass (reclaim_expired_leases), since label!= protects all leased objects including expired ones.

Verified:

  • molecule template-content: converge + idempotence + verify green (lease labels, tiers, capacity, admission marker assertions)
  • bash -n + shellcheck on rendered templates
  • lint-all + pytest-cov green; PR size 10 files / 495 lines

Non-blocking notes:

  • A valid lease pins its object indefinitely — producers must set bounded lease-until values (consumer-side contract, devx follow-up).
  • Producer-side lease emission (molecule/CI jobs) is out of scope here; until then guards degrade to the existing name-prefix/age behavior.

Outcome: No blocking issues remain. Awaiting CI before ready-to-merge.

## Deep Review Summary - **Files reviewed:** 10 (spec, 4 templates, 2 task files, defaults, verify.yml, historical-spec rename) - **Issues found:** 3 (all auto-fixed during development, verified against a live Docker daemon) **Fixed during review pass:** 1. `docker images` has no `.Label`/`.Labels` formatter — the image lease check now uses `docker image inspect` per candidate (previous version would have silently pruned nothing). 2. `label!=` is invalid on `volume ls`/`network ls` but valid on the prune endpoints; `until=` is unsupported for `volume prune` (dropped — silent-failure risk). 3. Expired-lease volumes/networks needed an explicit reclaim pass (`reclaim_expired_leases`), since `label!=` protects all leased objects including expired ones. **Verified:** - molecule `template-content`: converge + idempotence + verify green (lease labels, tiers, capacity, admission marker assertions) - `bash -n` + shellcheck on rendered templates - lint-all + pytest-cov green; PR size 10 files / 495 lines **Non-blocking notes:** - A valid lease pins its object indefinitely — producers must set bounded `lease-until` values (consumer-side contract, devx follow-up). - Producer-side lease emission (molecule/CI jobs) is out of scope here; until then guards degrade to the existing name-prefix/age behavior. **Outcome:** No blocking issues remain. Awaiting CI before `ready-to-merge`.
emil added 1 commit 2026-09-22 16:56:26 +00:00
refactor: dedupe volume/network prune, trim for size gate
CI / validate (pull_request) Failing after 1m32s
CI / molecule-tests (1) (pull_request) Skipped
CI / molecule-tests (2) (pull_request) Skipped
CI / molecule-tests (3) (pull_request) Skipped
CI / molecule-tests (4) (pull_request) Skipped
CI / auto-merge (pull_request) Skipped
28a230e7b0
emil added 1 commit 2026-09-22 16:57:05 +00:00
style: trim comments for size gate
CI / validate (pull_request) Failing after 1m7s
CI / molecule-tests (1) (pull_request) Skipped
CI / molecule-tests (2) (pull_request) Skipped
CI / molecule-tests (3) (pull_request) Skipped
CI / molecule-tests (4) (pull_request) Skipped
CI / auto-merge (pull_request) Skipped
757ccea5b9
emil added 1 commit 2026-09-22 16:57:39 +00:00
docs: tighten spec wording
CI / validate (pull_request) Failing after 1m5s
CI / molecule-tests (1) (pull_request) Skipped
CI / molecule-tests (2) (pull_request) Skipped
CI / molecule-tests (3) (pull_request) Skipped
CI / molecule-tests (4) (pull_request) Skipped
CI / auto-merge (pull_request) Skipped
7dfdee8b31
emil added 1 commit 2026-09-22 16:58:29 +00:00
fix: unquoted conditional filter args; trim spec
CI / validate (pull_request) Failing after 1m9s
CI / molecule-tests (1) (pull_request) Skipped
CI / molecule-tests (2) (pull_request) Skipped
CI / molecule-tests (3) (pull_request) Skipped
CI / molecule-tests (4) (pull_request) Skipped
CI / auto-merge (pull_request) Skipped
beed820562
emil added 1 commit 2026-09-22 16:58:52 +00:00
docs: condense test plan
CI / validate (pull_request) Successful in 1m8s
CI / molecule-tests (1) (pull_request) Successful in 5m16s
CI / molecule-tests (3) (pull_request) Successful in 7m21s
CI / molecule-tests (2) (pull_request) Successful in 7m31s
CI / molecule-tests (4) (pull_request) Successful in 7m33s
CI / auto-merge (pull_request) Successful in 7m47s
e202acdcf9
emil approved these changes 2026-09-22 17:01:08 +00:00
emil left a comment
Owner

Auto-approved: all CI checks passed (validate, molecule-tests).

Auto-approved: all CI checks passed (validate, molecule-tests).
kireto merged commit 832847fe66 into master 2026-09-22 17:07:50 +00:00
kireto deleted branch GRM-173-runner-resource-leases 2026-09-22 17:07:51 +00:00
Sign in to join this conversation.