DEVX-136: feat: add fix_pr_title module and update_pr API method
Post-merge / detect-and-configure (push) Successful in 12s
Post-merge / release-and-maintain (push) Successful in 1m0s

This commit was merged in pull request #203.
This commit is contained in:
2026-07-13 23:55:11 +00:00
parent 68f0872134
commit ddfbdec956
32 changed files with 2761 additions and 322 deletions
@@ -40,20 +40,30 @@ invocation in any repo with devx installed automatically runs the
static analysis. No extra Makefile target or CI step needed.
The plugin (`devx.tools.check_test_isolation`) statically analyzes
test files during `pytest_collection_finish` and emits
`UserWarning` for violations:
test files during `pytest_collection_finish` and **fails the test run**
on any hard violation:
- **unpatched-subprocess**: `subprocess.run/call/Popen/check_call/check_output`
called in a test function without `@patch`
called in a test function without `@patch` or `with patch(...)`
- **unpatched-sleep**: `time.sleep` called without `@patch`
- **unpatched-helper**: known subprocess-spawning helpers
(`update_doc_versions`, `run_cmd`, `run_tests`) called without
`@patch` (and without patching their internal dependencies)
- **excessive-iterations**: `for _ in range(N)` where N > 100
- **heavy-module-import**: `httpx`, `ansible`, etc. imported at module
level in test files, slowing collection for all tests
- **reload-without-cleanup**: `importlib.reload()` called an odd number
of times, leaving module state modified
The plugin recognizes transitive safety: if `run_cmd` is patched,
`run_tests` (which calls `run_cmd`) is safe. This is tracked via
`HELPER_INTERNAL_CALLS`.
Transitive-subprocess findings (via call-graph analysis) are reported
as **advisories** — the static analysis can't predict early exits or
runtime branch conditions, so the runtime audit is authoritative.
The plugin also wraps `subprocess.run` at runtime to catch real
subprocess calls that leak through transitive call paths (for example
`CliRunner.invoke(main)``main()``update_doc_versions()`
`subprocess.run()`). If a test spawns a real subprocess without
`@patch`, the test fails.
A standalone CLI (`python -m devx.tools.check_test_isolation`) is also
provided for CI gates and pre-commit hooks where pytest isn't run.
@@ -85,16 +95,20 @@ is even created.
- **Automatic enforcement**: The pytest plugin runs on every `pytest`
invocation across devx, grm, and infra — no per-repo configuration
needed. New tests with unpatched subprocess calls emit warnings
immediately.
needed. New tests with unpatched subprocess calls fail immediately.
- **Shift-left**: Translation gaps and test isolation violations are
caught locally (pre-commit / `make lint`) instead of in CI.
- **Fast feedback**: Static analysis adds <0.1s to test runs — no
runtime overhead.
- **No false positives**: The transitive dependency tracking
(`HELPER_INTERNAL_CALLS`) correctly recognizes that patching
`run_cmd` makes `run_tests` safe, and patching `subprocess.run`
makes all helpers safe.
- **Fast feedback**: Static analysis adds <0.1s to test runs; runtime
subprocess audit adds negligible overhead (wrapper checks a
thread-local flag).
- **Transitive detection**: The call-graph BFS traces
`CliRunner.invoke(main)``main()``update_doc_versions()`
`subprocess.run()`, catching indirect subprocess leaks that direct
analysis misses. The runtime audit provides authoritative enforcement.
- **No false positives**: The call graph correctly recognizes that
patching `run_cmd` makes `run_tests` (which calls `run_cmd`) safe,
and class methods are excluded to avoid false positives when classes
like `TeaCLI` are patched.
### Negative
@@ -103,10 +117,12 @@ is even created.
definitions) appears uncovered. Mitigated by `-p no:devx_test_isolation`
in devx's own `pyproject.toml` `addopts` and `# pragma: no cover` on
plugin hook functions.
- **Static analysis limitations**: The plugin only sees direct calls
in test function bodies, not indirect calls through `main()` or
other wrappers. This is acceptable — the `check_test_speed` tool
catches the symptom (slow tests) for indirect cases.
- **Static analysis limitations**: The call-graph BFS can't predict
runtime branch conditions or early exits — a test that patches
`shutil.which` to return `None` may skip the subprocess path
entirely, but the static analysis still reports it. Transitive
findings are advisories (exit 0) for this reason; the runtime audit
is authoritative.
- **Translation burden**: Every new `_()` call in source requires
adding 6 language translations. This is by design (all supported
languages must be complete) but adds friction for quick prototypes.
@@ -122,21 +138,36 @@ in consumer repos.
### Disabling the Plugin
- `--no-test-isolation` flag: disables analysis for a single run
- `--no-test-isolation` flag: disables static analysis and runtime
subprocess audit for a single run
- `-p no:devx_test_isolation` in `addopts`: disables for a repo
(used in devx's own `pyproject.toml` for coverage reasons)
### Strict Mode
### Call-Graph Analysis
- `--strict-test-isolation` flag: promotes warnings to errors and
prints a summary to stderr
- `filterwarnings = ["error:Test isolation:UserWarning"]` in
`pyproject.toml`: same effect via pytest's warning filter system
The `CallGraph` class parses all `.py` files under `src/` and builds
a map of function → called functions. When a test calls
`CliRunner.invoke(target)`, a BFS traces the call graph from `target`
to find all reachable functions. Class methods are excluded from the
call graph to avoid false positives when classes are patched (for example
`@patch("...TeaCLI")` mocks all methods). The BFS respects `@patch`
decorators — if a function is patched, traversal stops at that node.
### Runtime Subprocess Audit
The `_SubprocessAudit` singleton wraps `subprocess.run`, `call`,
`check_call`, `check_output`, and `Popen` with thread-local
recording wrappers. During each non-integration test, the wrapper
records calls; if any are recorded (that is the test didn't `@patch`
subprocess), the test fails. The wrappers check a thread-local flag,
so inactive audits have zero overhead beyond the flag check.
### Known Subprocess Helpers
The `KNOWN_SUBPROCESS_HELPERS` dict maps function names to
descriptions. `HELPER_INTERNAL_CALLS` maps each helper to the
function names it internally calls, enabling transitive safety
checks. Both are defined in `check_test_isolation.py` and can be
extended as new subprocess-spawning helpers are added to devx.
checks for direct calls in test functions. The call-graph BFS
handles transitive detection for `CliRunner.invoke` targets. Both
are defined in `check_test_isolation.py` and can be extended as
new subprocess-spawning helpers are added to devx.