Public Access
DEVX-136: feat: add fix_pr_title module and update_pr API method
This commit was merged in pull request #203.
This commit is contained in:
@@ -40,20 +40,30 @@ invocation in any repo with devx installed automatically runs the
|
||||
static analysis. No extra Makefile target or CI step needed.
|
||||
|
||||
The plugin (`devx.tools.check_test_isolation`) statically analyzes
|
||||
test files during `pytest_collection_finish` and emits
|
||||
`UserWarning` for violations:
|
||||
test files during `pytest_collection_finish` and **fails the test run**
|
||||
on any hard violation:
|
||||
|
||||
- **unpatched-subprocess**: `subprocess.run/call/Popen/check_call/check_output`
|
||||
called in a test function without `@patch`
|
||||
called in a test function without `@patch` or `with patch(...)`
|
||||
- **unpatched-sleep**: `time.sleep` called without `@patch`
|
||||
- **unpatched-helper**: known subprocess-spawning helpers
|
||||
(`update_doc_versions`, `run_cmd`, `run_tests`) called without
|
||||
`@patch` (and without patching their internal dependencies)
|
||||
- **excessive-iterations**: `for _ in range(N)` where N > 100
|
||||
- **heavy-module-import**: `httpx`, `ansible`, etc. imported at module
|
||||
level in test files, slowing collection for all tests
|
||||
- **reload-without-cleanup**: `importlib.reload()` called an odd number
|
||||
of times, leaving module state modified
|
||||
|
||||
The plugin recognizes transitive safety: if `run_cmd` is patched,
|
||||
`run_tests` (which calls `run_cmd`) is safe. This is tracked via
|
||||
`HELPER_INTERNAL_CALLS`.
|
||||
Transitive-subprocess findings (via call-graph analysis) are reported
|
||||
as **advisories** — the static analysis can't predict early exits or
|
||||
runtime branch conditions, so the runtime audit is authoritative.
|
||||
|
||||
The plugin also wraps `subprocess.run` at runtime to catch real
|
||||
subprocess calls that leak through transitive call paths (for example
|
||||
`CliRunner.invoke(main)` → `main()` → `update_doc_versions()` →
|
||||
`subprocess.run()`). If a test spawns a real subprocess without
|
||||
`@patch`, the test fails.
|
||||
|
||||
A standalone CLI (`python -m devx.tools.check_test_isolation`) is also
|
||||
provided for CI gates and pre-commit hooks where pytest isn't run.
|
||||
@@ -85,16 +95,20 @@ is even created.
|
||||
|
||||
- **Automatic enforcement**: The pytest plugin runs on every `pytest`
|
||||
invocation across devx, grm, and infra — no per-repo configuration
|
||||
needed. New tests with unpatched subprocess calls emit warnings
|
||||
immediately.
|
||||
needed. New tests with unpatched subprocess calls fail immediately.
|
||||
- **Shift-left**: Translation gaps and test isolation violations are
|
||||
caught locally (pre-commit / `make lint`) instead of in CI.
|
||||
- **Fast feedback**: Static analysis adds <0.1s to test runs — no
|
||||
runtime overhead.
|
||||
- **No false positives**: The transitive dependency tracking
|
||||
(`HELPER_INTERNAL_CALLS`) correctly recognizes that patching
|
||||
`run_cmd` makes `run_tests` safe, and patching `subprocess.run`
|
||||
makes all helpers safe.
|
||||
- **Fast feedback**: Static analysis adds <0.1s to test runs; runtime
|
||||
subprocess audit adds negligible overhead (wrapper checks a
|
||||
thread-local flag).
|
||||
- **Transitive detection**: The call-graph BFS traces
|
||||
`CliRunner.invoke(main)` → `main()` → `update_doc_versions()` →
|
||||
`subprocess.run()`, catching indirect subprocess leaks that direct
|
||||
analysis misses. The runtime audit provides authoritative enforcement.
|
||||
- **No false positives**: The call graph correctly recognizes that
|
||||
patching `run_cmd` makes `run_tests` (which calls `run_cmd`) safe,
|
||||
and class methods are excluded to avoid false positives when classes
|
||||
like `TeaCLI` are patched.
|
||||
|
||||
### Negative
|
||||
|
||||
@@ -103,10 +117,12 @@ is even created.
|
||||
definitions) appears uncovered. Mitigated by `-p no:devx_test_isolation`
|
||||
in devx's own `pyproject.toml` `addopts` and `# pragma: no cover` on
|
||||
plugin hook functions.
|
||||
- **Static analysis limitations**: The plugin only sees direct calls
|
||||
in test function bodies, not indirect calls through `main()` or
|
||||
other wrappers. This is acceptable — the `check_test_speed` tool
|
||||
catches the symptom (slow tests) for indirect cases.
|
||||
- **Static analysis limitations**: The call-graph BFS can't predict
|
||||
runtime branch conditions or early exits — a test that patches
|
||||
`shutil.which` to return `None` may skip the subprocess path
|
||||
entirely, but the static analysis still reports it. Transitive
|
||||
findings are advisories (exit 0) for this reason; the runtime audit
|
||||
is authoritative.
|
||||
- **Translation burden**: Every new `_()` call in source requires
|
||||
adding 6 language translations. This is by design (all supported
|
||||
languages must be complete) but adds friction for quick prototypes.
|
||||
@@ -122,21 +138,36 @@ in consumer repos.
|
||||
|
||||
### Disabling the Plugin
|
||||
|
||||
- `--no-test-isolation` flag: disables analysis for a single run
|
||||
- `--no-test-isolation` flag: disables static analysis and runtime
|
||||
subprocess audit for a single run
|
||||
- `-p no:devx_test_isolation` in `addopts`: disables for a repo
|
||||
(used in devx's own `pyproject.toml` for coverage reasons)
|
||||
|
||||
### Strict Mode
|
||||
### Call-Graph Analysis
|
||||
|
||||
- `--strict-test-isolation` flag: promotes warnings to errors and
|
||||
prints a summary to stderr
|
||||
- `filterwarnings = ["error:Test isolation:UserWarning"]` in
|
||||
`pyproject.toml`: same effect via pytest's warning filter system
|
||||
The `CallGraph` class parses all `.py` files under `src/` and builds
|
||||
a map of function → called functions. When a test calls
|
||||
`CliRunner.invoke(target)`, a BFS traces the call graph from `target`
|
||||
to find all reachable functions. Class methods are excluded from the
|
||||
call graph to avoid false positives when classes are patched (for example
|
||||
`@patch("...TeaCLI")` mocks all methods). The BFS respects `@patch`
|
||||
decorators — if a function is patched, traversal stops at that node.
|
||||
|
||||
### Runtime Subprocess Audit
|
||||
|
||||
The `_SubprocessAudit` singleton wraps `subprocess.run`, `call`,
|
||||
`check_call`, `check_output`, and `Popen` with thread-local
|
||||
recording wrappers. During each non-integration test, the wrapper
|
||||
records calls; if any are recorded (that is the test didn't `@patch`
|
||||
subprocess), the test fails. The wrappers check a thread-local flag,
|
||||
so inactive audits have zero overhead beyond the flag check.
|
||||
|
||||
### Known Subprocess Helpers
|
||||
|
||||
The `KNOWN_SUBPROCESS_HELPERS` dict maps function names to
|
||||
descriptions. `HELPER_INTERNAL_CALLS` maps each helper to the
|
||||
function names it internally calls, enabling transitive safety
|
||||
checks. Both are defined in `check_test_isolation.py` and can be
|
||||
extended as new subprocess-spawning helpers are added to devx.
|
||||
checks for direct calls in test functions. The call-graph BFS
|
||||
handles transitive detection for `CliRunner.invoke` targets. Both
|
||||
are defined in `check_test_isolation.py` and can be extended as
|
||||
new subprocess-spawning helpers are added to devx.
|
||||
|
||||
Reference in New Issue
Block a user