oblachno-oss/grm
3
0
Fork 0
Code Issues 33 Pull Requests Actions Packages Projects Releases 23 Wiki Activity

feat: scoped runner cleanup with ownership leases and disk admission

Browse Source 
Extend stopped-container protection to an explicit lease contract
(org.oblachno.lease-until / org.oblachno.owner) honored by every cleanup
path. Consolidate the duplicated inline prune logic from docker-prune
and the healthcheck into a single tiered runner-cleanup.sh; remove the
unfiltered `system prune -af --volumes` / `volume prune` paths that could
wipe a job's volumes mid-run, and keep warm base images under pressure.

At critical disk usage the healthcheck now stops admitting new work
(stops gitea-runner.service once no CI job is in flight) and resumes it
automatically after recovery. The runner config declares capacity, and
the role refuses to install on production-marked hosts.
This commit is contained in:
Emil Simeonov
2026-09-22 18:34:12 +02:00
parent 49646c38db
commit 3c0696f3f2
10 changed files with 385 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/GRM-173-skills-historical.md
+46
View File
@@ -0,0 +1,46 @@
# GRM-173: Add dependency-graph, deployment-coordination, and skill-creation skills
## Problem
Agents working across the oblachno ecosystem lack shared, written context
for three recurring struggles: (1) knowing which repo produces what and
the correct order for cross-repo changes, (2) coordinating grm releases
with the downstream infra dependency PR, and (3) creating and validating
new Devin skills consistently. Without these skills, agents repeatedly
make mistakes such as deploying infra before the grm dependency PR is
merged, or writing skills that fail the validator.
## Approach
Add three skill files under `.devin/skills/`. Two are shared skills
(`dependency-graph`, `skill-creation`) that must be identical across
repos; one is grm-specific (`deployment-coordination`). All three
follow the standard skill structure (H1 title, When to Invoke,
Prerequisites, core content) and reference real make targets, file
paths, and API endpoints.
REQ-1: Add `.devin/skills/dependency-graph/SKILL.md` — shared skill mapping the oblachno ecosystem (repos, produces/consumers, dependency chain, correct change order, state verification)
REQ-2: Add `.devin/skills/deployment-coordination/SKILL.md` — grm-specific skill covering release flow, downstream consumer, coordinating a grm change, and common mistakes
REQ-3: Add `.devin/skills/skill-creation/SKILL.md` — shared skill for creating, validating, and maintaining skills (structure, quality standards, scope rules, automated validation, checklist)
## Files Affected
- `.devin/skills/dependency-graph/SKILL.md` (new)
- `.devin/skills/deployment-coordination/SKILL.md` (new)
- `.devin/skills/skill-creation/SKILL.md` (new)
- `docs/specs/GRM-173.md` (new)
## Test Plan
- Verify all three SKILL.md files follow the required structure (H1, When to Invoke, Prerequisites)
- Verify referenced make targets and file paths are accurate
- Run `make pytest-cov` to confirm no test regressions (skills are docs-only, no code changes)
- Confirm shared skills (`dependency-graph`, `skill-creation`) are ready for cross-repo sync
## Deploy Plan
- Merge to master via auto-merge workflow
- No runtime changes; documentation-only (`.devin/**` is infrastructure path, no release triggered)
## Rollback Plan
- Revert the merge commit; skill files are removed, no functional impact
## Acceptance Criteria
- [x] REQ-1: `.devin/skills/dependency-graph/SKILL.md` exists with ecosystem map, dependency chain, correct change order, and state verification sections
- [x] REQ-2: `.devin/skills/deployment-coordination/SKILL.md` exists with release flow, downstream consumer table, coordination steps, and common mistakes
- [x] REQ-3: `.devin/skills/skill-creation/SKILL.md` exists with skill structure template, quality standards, scope rules, automated validation, and creation checklist
docs/specs/GRM-173.md
+87 -32
View File
@@ -1,46 +1,101 @@
# GRM-173: Add dependency-graph, deployment-coordination, and skill-creation skills
# GRM-173: Runner resource leases and scoped disk cleanup
## Problem
Agents working across the oblachno ecosystem lack shared, written context
for three recurring struggles: (1) knowing which repo produces what and
the correct order for cross-repo changes, (2) coordinating grm releases
with the downstream infra dependency PR, and (3) creating and validating
new Devin skills consistently. Without these skills, agents repeatedly
make mistakes such as deploying infra before the grm dependency PR is
merged, or writing skills that fail the validator.
Runner hosts protect in-flight work only via name-prefix and age heuristics:
- The healthcheck critical tier runs unfiltered `docker system prune -af --volumes`
and `docker volume prune -f` — a job's freshly created but momentarily unused
volume/network can be wiped mid-run, and warm base images are destroyed under
pressure exactly when they are needed most.
- Container exclusion is a name-prefix (`GITEA-ACTIONS-TASK`) plus a
`RunningFor` text heuristic — molecule containers owned by a live job are
protected only by naming convention, not by an ownership claim.
- Cleanup logic is duplicated inline between `docker-prune.service` and the
healthcheck script — the two paths already diverge (volume pruning exists in
one tier only).
- There is no admission control: under disk pressure the runner keeps
accepting new jobs while cleanup races in-flight work.
- The runner config never declares `capacity`, and nothing prevents
installing a runner on a production host.
## Approach
Add three skill files under `.devin/skills/`. Two are shared skills
(`dependency-graph`, `skill-creation`) that must be identical across
repos; one is grm-specific (`deployment-coordination`). All three
follow the standard skill structure (H1 title, When to Invoke,
Prerequisites, core content) and reference real make targets, file
paths, and API endpoints.
REQ-1: Add `.devin/skills/dependency-graph/SKILL.md` — shared skill mapping the oblachno ecosystem (repos, produces/consumers, dependency chain, correct change order, state verification)
REQ-2: Add `.devin/skills/deployment-coordination/SKILL.md` — grm-specific skill covering release flow, downstream consumer, coordinating a grm change, and common mistakes
REQ-3: Add `.devin/skills/skill-creation/SKILL.md` — shared skill for creating, validating, and maintaining skills (structure, quality standards, scope rules, automated validation, checklist)
REQ-1: Define an ownership-lease label contract. Producers tag containers,
images, volumes and networks with `org.oblachno.lease-until` (epoch seconds)
and `org.oblachno.owner` (free-form run/job id). All cleanup paths must never
remove an object whose `lease-until` is in the future; expired leases are
reclaimable. Existing `GITEA-ACTIONS-TASK` name-prefix and `status=exited`
guards are retained for unlabeled objects.
## Files Affected
- `.devin/skills/dependency-graph/SKILL.md` (new)
- `.devin/skills/deployment-coordination/SKILL.md` (new)
- `.devin/skills/skill-creation/SKILL.md` (new)
- `docs/specs/GRM-173.md` (new)
REQ-2: Introduce a single shared cleanup script
(`runner-cleanup.sh`, templated next to the healthcheck script) invoked with
`--tier routine|pressure|critical`. It replaces all inline prune logic in
`docker-prune.service` and the healthcheck. Every prune is scoped: volumes and
networks get `label!=`/`until=` filters at every tier (no unfiltered volume or
network prune remains); images matching `gitea_runner_keep_images` are never
removed, so warm base layers survive critical pressure.
REQ-3: Watermark-tiered behavior. `routine` (timer) prunes aged resources only.
`pressure` (disk >= warn) prunes unowned resources older than 1h. `critical`
(disk >= critical) drops age limits but still honors leases, keep-images, and
never removes running or `GITEA-ACTIONS-TASK` containers.
REQ-4: Admission control under disk pressure. When disk is >= critical and no
`GITEA-ACTIONS-TASK` container is running, the healthcheck writes a marker
file and stops `gitea-runner.service` (the runner simply stops fetching new
jobs). The service is restarted and the marker cleared by a later healthcheck
once disk drops below the warn threshold. In-flight jobs are never killed by
the admission path. Controlled by `gitea_runner_disk_admission_enabled`.
REQ-5: Declare physical-host capacity explicitly:
`runner.capacity: {{ gitea_runner_capacity }}` in the act_runner config
(default 1, matching upstream default).
REQ-6: Production-host exclusion. The role fails early when the target host
carries the production marker file `/etc/oblachno/production-host` or when
`gitea_runner_on_production_host` is true, unless
`gitea_runner_allow_production_host` overrides. Infra-side provisioning of the
marker is a follow-up task.
Historical spec for the colliding task ID:
[GRM-173-skills-historical](GRM-173-skills-historical.md).
## Test Plan
- Verify all three SKILL.md files follow the required structure (H1, When to Invoke, Prerequisites)
- Verify referenced make targets and file paths are accurate
- Run `make pytest-cov` to confirm no test regressions (skills are docs-only, no code changes)
- Confirm shared skills (`dependency-graph`, `skill-creation`) are ready for cross-repo sync
- `template-content` molecule scenario: assert the prune service calls
`runner-cleanup.sh`, assert lease filters and keep-images logic render in
the cleanup script, assert `capacity:` renders in the runner config.
- `default` molecule scenario: assert the cleanup script is installed and
executable.
- `bash -n` syntax check on rendered templates during development.
- `make lint-all`, `make pytest-cov`, fast molecule for the changed role.
## Deploy Plan
- Merge to master via auto-merge workflow
- No runtime changes; documentation-only (`.devin/**` is infrastructure path, no release triggered)
- Merge to master via auto-merge; post-merge publishes the package and
auto-creates the infra dependency-bump PR. Runner hosts pick up the change
on the next `grm install`/update run — no manual host action.
- Producers emitting lease labels (molecule distribution, CI jobs) are a
separate devx-side change; until then the guards degrade gracefully to the
existing name-prefix/age behavior.
## Rollback Plan
- Revert the merge commit; skill files are removed, no functional impact
- Revert the merge commit; re-run `grm install` to redeploy the previous
prune/healthcheck units. No persistent state or data migration — the
marker file under `gitea_runner_data_dir` is removed by the previous
template's absence (or harmless if left behind).
## Acceptance Criteria
- [x] REQ-1: `.devin/skills/dependency-graph/SKILL.md` exists with ecosystem map, dependency chain, correct change order, and state verification sections
- [x] REQ-2: `.devin/skills/deployment-coordination/SKILL.md` exists with release flow, downstream consumer table, coordination steps, and common mistakes
- [x] REQ-3: `.devin/skills/skill-creation/SKILL.md` exists with skill structure template, quality standards, scope rules, automated validation, and creation checklist
- [x] REQ-1: `org.oblachno.lease-until`/`org.oblachno.owner` labels are honored
by every cleanup path; valid leases are never removed, expired leases are.
- [x] REQ-2: single shared `runner-cleanup.sh` used by prune service and
healthcheck; no unfiltered `system prune --volumes`, `volume prune`, or
`network prune` remains; `gitea_runner_keep_images` never removed.
- [x] REQ-3: three tiers behave as specified (routine/pressure/critical).
- [x] REQ-4: critical pressure with zero in-flight job containers stops
admission via marker + service stop; recovery resumes automatically.
- [x] REQ-5: `runner.capacity` rendered in `config.yaml`.
- [x] REQ-6: role fails on production-marked hosts unless explicitly allowed.