Public Access
143 lines
5.1 KiB
Markdown
143 lines
5.1 KiB
Markdown
# skill-creation
|
|
|
|
How to create, validate, and maintain Devin skills. Skills must be
|
|
clear, succinct, and actionable — no AI slop.
|
|
|
|
## When to Invoke
|
|
|
|
Invoke this skill when:
|
|
- Creating a new skill
|
|
- Amending an existing skill
|
|
- Evaluating whether a skill is needed
|
|
- Reviewing a PR that adds or modifies skills
|
|
|
|
## Prerequisites
|
|
|
|
- Skill directory: `.devin/skills/<skill-name>/SKILL.md`
|
|
- Validator: `tests/test_skills.py` (infra) or reference to it
|
|
- Tests: `tests/unit/test_skills_validation.py` (infra)
|
|
|
|
## When to Create a Skill
|
|
|
|
Create a skill when:
|
|
- An agent struggles with a task repeatedly (branch hygiene, PR order)
|
|
- A workflow has non-obvious ordering constraints (deployment coordination)
|
|
- A task requires specific tool usage over raw commands (CI monitoring)
|
|
- Multiple agents need shared context (dependency graph)
|
|
|
|
Do NOT create a skill for:
|
|
- One-off tasks (use a spec instead)
|
|
- Tasks already covered by AGENTS.md
|
|
- Tasks that are obvious from the Makefile or README
|
|
- Tasks that change frequently (skills should be stable)
|
|
|
|
## Skill Structure
|
|
|
|
Every skill MUST have:
|
|
|
|
```markdown
|
|
# <skill-name>
|
|
|
|
One-line description of what the skill does.
|
|
|
|
## When to Invoke
|
|
|
|
2-4 bullet points describing when to use this skill.
|
|
|
|
## Prerequisites
|
|
|
|
What must exist before using the skill (venv, .env, tools).
|
|
|
|
## <Core Content>
|
|
|
|
The actual guidance. Keep it actionable.
|
|
|
|
## Verification (if applicable)
|
|
|
|
How to verify the skill's guidance works.
|
|
|
|
## Common Mistakes (if applicable)
|
|
|
|
What agents get wrong without this skill.
|
|
```
|
|
|
|
## Quality Standards
|
|
|
|
### Do
|
|
- **Be specific.** Reference exact make targets, file paths, commands.
|
|
- **Be concise.** Each section should be scannable in under 30 seconds.
|
|
- **Be actionable.** Every paragraph should tell the agent what to DO.
|
|
- **Use tables** for command reference, mappings, and comparisons.
|
|
- **Use code blocks** for commands the agent should run.
|
|
- **Link to other skills** when related (e.g., "See `dependency-graph` skill").
|
|
|
|
### Don't
|
|
- **No preamble.** Don't start with "This skill helps agents..." — just state what it does.
|
|
- **No filler.** Don't repeat information from AGENTS.md or other skills.
|
|
- **No vague advice.** "Be careful with branches" is useless. "Run `git branch --show-current` before every commit" is useful.
|
|
- **No AI slop.** Don't write "In this comprehensive guide, we will explore..." — just give the guidance.
|
|
- **No redundant sections.** If "Common Mistakes" would repeat "When to Invoke", skip it.
|
|
- **No marketing.** Don't describe the skill as "powerful" or "comprehensive".
|
|
|
|
## Scope Rules
|
|
|
|
- **One skill per concern.** Don't mix branch hygiene with CI monitoring.
|
|
- **Project-specific, not generic.** Skills reference this repo's make targets, file paths, and conventions — not abstract advice.
|
|
- **Shared skills must be identical across repos.** Use `SHARED_SKILLS` in the validator to enforce this.
|
|
- **Per-repo skills must reflect that repo's reality.** Don't copy infra-specific targets to sso-bridge.
|
|
|
|
## Automated Validation
|
|
|
|
Every skill must pass the validator (`tests/test_skills.py`). The validator checks:
|
|
|
|
1. **Structure** — H1 title, "When to Invoke" section, "Prerequisites" section
|
|
2. **Commands** — referenced `make <target>` commands exist in Makefile or devx.mak
|
|
3. **Paths** — referenced file paths exist in the repo
|
|
4. **Shared skills** — identical content across repos (SHA-256 comparison)
|
|
5. **No drift** — no references to nonexistent commands or files
|
|
|
|
Run the validator:
|
|
```bash
|
|
python3 tests/test_skills.py --repo infra --repo sso-bridge
|
|
```
|
|
|
|
## Effectiveness Evaluation
|
|
|
|
### Static Checks (automated, CI)
|
|
|
|
The validator runs in CI as part of `make pytest-cov`. A failing skill
|
|
test blocks the PR. This catches:
|
|
- Missing sections
|
|
- Invalid commands
|
|
- Broken file references
|
|
- Cross-repo drift
|
|
|
|
### Runtime Metrics (manual, periodic)
|
|
|
|
Track these signals to evaluate skill effectiveness:
|
|
- **Skill invocation frequency** — how often agents invoke the skill
|
|
- **Success rate when invoked** — did the skill prevent the mistake it targets?
|
|
- **Feedback issues** — agents create Gitea issues with `feedback` label when a skill is unclear or wrong
|
|
- **Mistake recurrence** — if agents still make the mistake the skill targets, the skill needs improvement
|
|
|
|
### Retrospective Review
|
|
|
|
Periodically (monthly or after major incidents) review skills:
|
|
1. List all skills and their last-modified dates
|
|
2. Check for feedback issues tagged `skill-improvement`
|
|
3. Verify referenced commands still exist (run validator)
|
|
4. Remove skills that are no longer relevant
|
|
5. Update skills where mistakes still recur
|
|
6. Document lessons in this skill's "Common Mistakes" section
|
|
|
|
## Creating a New Skill — Checklist
|
|
|
|
- [ ] Identify the repeated struggle or non-obvious workflow
|
|
- [ ] Check no existing skill covers it
|
|
- [ ] Write the skill following the structure above
|
|
- [ ] Run `python3 tests/test_skills.py` — must pass
|
|
- [ ] Run `make pytest-cov` — must pass with 100% coverage
|
|
- [ ] If shared across repos, copy identical content to each repo
|
|
- [ ] Add the skill to `SHARED_SKILLS` in the validator if shared
|
|
- [ ] Create PR, verify CI passes, merge
|