GRM-173: feat: scoped runner cleanup with ownership leases and disk admission

docs/specs/GRM-173-skills-historical.md

@@ -0,0 +1,46 @@
+# GRM-173: Add dependency-graph, deployment-coordination, and skill-creation skills
+
+## Problem
+Agents working across the oblachno ecosystem lack shared, written context
+for three recurring struggles: (1) knowing which repo produces what and
+the correct order for cross-repo changes, (2) coordinating grm releases
+with the downstream infra dependency PR, and (3) creating and validating
+new Devin skills consistently. Without these skills, agents repeatedly
+make mistakes such as deploying infra before the grm dependency PR is
+merged, or writing skills that fail the validator.
+
+## Approach
+Add three skill files under `.devin/skills/`. Two are shared skills
+(`dependency-graph`, `skill-creation`) that must be identical across
+repos; one is grm-specific (`deployment-coordination`). All three
+follow the standard skill structure (H1 title, When to Invoke,
+Prerequisites, core content) and reference real make targets, file
+paths, and API endpoints.
+
+REQ-1: Add `.devin/skills/dependency-graph/SKILL.md` — shared skill mapping the oblachno ecosystem (repos, produces/consumers, dependency chain, correct change order, state verification)
+REQ-2: Add `.devin/skills/deployment-coordination/SKILL.md` — grm-specific skill covering release flow, downstream consumer, coordinating a grm change, and common mistakes
+REQ-3: Add `.devin/skills/skill-creation/SKILL.md` — shared skill for creating, validating, and maintaining skills (structure, quality standards, scope rules, automated validation, checklist)
+
+## Files Affected
+- `.devin/skills/dependency-graph/SKILL.md` (new)
+- `.devin/skills/deployment-coordination/SKILL.md` (new)
+- `.devin/skills/skill-creation/SKILL.md` (new)
+- `docs/specs/GRM-173.md` (new)
+
+## Test Plan
+- Verify all three SKILL.md files follow the required structure (H1, When to Invoke, Prerequisites)
+- Verify referenced make targets and file paths are accurate
+- Run `make pytest-cov` to confirm no test regressions (skills are docs-only, no code changes)
+- Confirm shared skills (`dependency-graph`, `skill-creation`) are ready for cross-repo sync
+
+## Deploy Plan
+- Merge to master via auto-merge workflow
+- No runtime changes; documentation-only (`.devin/**` is infrastructure path, no release triggered)
+
+## Rollback Plan
+- Revert the merge commit; skill files are removed, no functional impact
+
+## Acceptance Criteria
+- [x] REQ-1: `.devin/skills/dependency-graph/SKILL.md` exists with ecosystem map, dependency chain, correct change order, and state verification sections
+- [x] REQ-2: `.devin/skills/deployment-coordination/SKILL.md` exists with release flow, downstream consumer table, coordination steps, and common mistakes
+- [x] REQ-3: `.devin/skills/skill-creation/SKILL.md` exists with skill structure template, quality standards, scope rules, automated validation, and creation checklist

docs/specs/GRM-173.md

@@ -1,46 +1,88 @@
-# GRM-173: Add dependency-graph, deployment-coordination, and skill-creation skills
+# GRM-173: Runner resource leases and scoped disk cleanup
 
 ## Problem
-Agents working across the oblachno ecosystem lack shared, written context
-for three recurring struggles: (1) knowing which repo produces what and
-the correct order for cross-repo changes, (2) coordinating grm releases
-with the downstream infra dependency PR, and (3) creating and validating
-new Devin skills consistently. Without these skills, agents repeatedly
-make mistakes such as deploying infra before the grm dependency PR is
-merged, or writing skills that fail the validator.
+
+Runner hosts protect in-flight work only via name-prefix and age heuristics:
+
+- The healthcheck critical tier runs unfiltered `docker system prune -af --volumes`
+  and `docker volume prune -f` — a job's momentarily unused volume/network can be
+  wiped mid-run, and warm base images are destroyed exactly when needed most.
+- Molecule containers owned by a live job are protected only by the
+  `GITEA-ACTIONS-TASK` naming convention, not by an ownership claim.
+- Cleanup logic is duplicated between `docker-prune.service` and the healthcheck.
+- No admission control: under disk pressure the runner keeps accepting jobs
+  while cleanup races in-flight work.
+- `capacity` is never declared; nothing prevents installing on a production host.
 
 ## Approach
-Add three skill files under `.devin/skills/`. Two are shared skills
-(`dependency-graph`, `skill-creation`) that must be identical across
-repos; one is grm-specific (`deployment-coordination`). All three
-follow the standard skill structure (H1 title, When to Invoke,
-Prerequisites, core content) and reference real make targets, file
-paths, and API endpoints.
 
-REQ-1: Add `.devin/skills/dependency-graph/SKILL.md` — shared skill mapping the oblachno ecosystem (repos, produces/consumers, dependency chain, correct change order, state verification)
-REQ-2: Add `.devin/skills/deployment-coordination/SKILL.md` — grm-specific skill covering release flow, downstream consumer, coordinating a grm change, and common mistakes
-REQ-3: Add `.devin/skills/skill-creation/SKILL.md` — shared skill for creating, validating, and maintaining skills (structure, quality standards, scope rules, automated validation, checklist)
+REQ-1: Define an ownership-lease label contract. Producers tag containers,
+images, volumes and networks with `org.oblachno.lease-until` (epoch seconds)
+and `org.oblachno.owner` (free-form run/job id). All cleanup paths must never
+remove an object whose `lease-until` is in the future; expired leases are
+reclaimable. Existing `GITEA-ACTIONS-TASK` name-prefix and `status=exited`
+guards are retained for unlabeled objects.
 
-## Files Affected
-- `.devin/skills/dependency-graph/SKILL.md` (new)
-- `.devin/skills/deployment-coordination/SKILL.md` (new)
-- `.devin/skills/skill-creation/SKILL.md` (new)
-- `docs/specs/GRM-173.md` (new)
+REQ-2: Introduce a single shared cleanup script (`runner-cleanup.sh`,
+templated next to the healthcheck script) invoked with
+`--tier routine|pressure|critical`, replacing all inline prune logic in
+`docker-prune.service` and the healthcheck. Every prune is scoped (leases,
+`until=` where supported); images matching `gitea_runner_keep_images` are
+never removed, so warm base layers survive critical pressure.
+
+REQ-3: Watermark-tiered behavior: `routine` (timer) prunes aged resources;
+`pressure` (disk >= warn) prunes unowned resources older than 1h; `critical`
+(disk >= critical) drops age limits but still honors leases, keep-images, and
+never removes running or `GITEA-ACTIONS-TASK` containers.
+
+REQ-4: Admission control under disk pressure. When disk is >= critical and no
+`GITEA-ACTIONS-TASK` container is running, the healthcheck writes a marker file
+and stops `gitea-runner.service` (the runner stops fetching jobs). A later
+healthcheck restarts it once disk drops below warn. In-flight jobs are never
+killed. Controlled by `gitea_runner_disk_admission_enabled`.
+
+REQ-5: Declare physical-host capacity explicitly: `runner.capacity:
+{{ gitea_runner_capacity }}` in the act_runner config (default 1 = upstream).
+
+REQ-6: Production-host exclusion. The role fails early when the target carries
+the marker file `/etc/oblachno/production-host` or
+`gitea_runner_on_production_host` is true, unless
+`gitea_runner_allow_production_host` overrides. Infra-side marker provisioning
+is a follow-up task.
+
+Historical spec for the colliding task ID:
+[GRM-173-skills-historical](GRM-173-skills-historical.md).
 
 ## Test Plan
-- Verify all three SKILL.md files follow the required structure (H1, When to Invoke, Prerequisites)
-- Verify referenced make targets and file paths are accurate
-- Run `make pytest-cov` to confirm no test regressions (skills are docs-only, no code changes)
-- Confirm shared skills (`dependency-graph`, `skill-creation`) are ready for cross-repo sync
+
+- `template-content` molecule scenario: prune service calls `runner-cleanup.sh`;
+  lease filters, keep-images and `capacity:` render correctly.
+- `default` scenario: cleanup script installed and executable.
+- `bash -n` on rendered templates; `make lint-all`, `make pytest-cov`, fast
+  molecule for the changed role.
 
 ## Deploy Plan
-- Merge to master via auto-merge workflow
-- No runtime changes; documentation-only (`.devin/**` is infrastructure path, no release triggered)
+
+- Merge via auto-merge; post-merge publishes the package and auto-creates the
+  infra dependency-bump PR. Runner hosts pick up the change on the next
+  `grm install`/update run — no manual host action.
+- Producer-side lease emission (molecule/CI jobs) is a separate devx change;
+  until then the guards degrade to the existing name-prefix/age behavior.
 
 ## Rollback Plan
-- Revert the merge commit; skill files are removed, no functional impact
+
+- Revert the merge commit and re-run `grm install` to redeploy the previous
+  prune/healthcheck units. No persistent state or migration.
 
 ## Acceptance Criteria
-- [x] REQ-1: `.devin/skills/dependency-graph/SKILL.md` exists with ecosystem map, dependency chain, correct change order, and state verification sections
-- [x] REQ-2: `.devin/skills/deployment-coordination/SKILL.md` exists with release flow, downstream consumer table, coordination steps, and common mistakes
-- [x] REQ-3: `.devin/skills/skill-creation/SKILL.md` exists with skill structure template, quality standards, scope rules, automated validation, and creation checklist
+
+- [x] REQ-1: `org.oblachno.lease-until`/`org.oblachno.owner` labels are honored
+  by every cleanup path; valid leases are never removed, expired leases are.
+- [x] REQ-2: single shared `runner-cleanup.sh` used by prune service and
+  healthcheck; no unfiltered `system prune --volumes`, `volume prune`, or
+  `network prune` remains; `gitea_runner_keep_images` never removed.
+- [x] REQ-3: three tiers behave as specified (routine/pressure/critical).
+- [x] REQ-4: critical pressure with zero in-flight job containers stops
+  admission via marker + service stop; recovery resumes automatically.
+- [x] REQ-5: `runner.capacity` rendered in `config.yaml`.
+- [x] REQ-6: role fails on production-marked hosts unless explicitly allowed.