DEVX-129: feat: test isolation pytest plugin, shift-left quality gates, dep upgrades

2026-07-13 00:57:28 +00:00
parent b923e47d81
commit 77c2f7e043
15 changed files with 1711 additions and 129 deletions
@@ -0,0 +1,142 @@
+# ADR-0001: Test Isolation Pytest Plugin and Shift-Left Quality Gates
+
+Date: 2026-07-13
+Status: Accepted
+
+## Context
+
+Unit tests in devx were slow (10s+) and getting slower. Investigation
+revealed two root causes:
+
+1. **Unpatched subprocess calls** — test functions calling
+   `subprocess.run`, `update_doc_versions`, or `run_cmd` without
+   `@patch` decorators, causing real subprocess execution during tests.
+2. **Excessive iterations** — statistical tests with 1000-iteration
+   loops that should use property-based testing or smaller samples.
+
+These issues were discovered manually by profiling with
+`pytest --durations=0`. There was no automated check to prevent
+regressions — new tests could introduce the same patterns and slow
+down the suite again.
+
+Additionally, translation completeness checks
+(`devx.ci.check_translations`) only ran in CI, not locally. Developers
+discovered missing translations at CI time, wasting round-trips.
+
+## Decision
+
+### 1. Test Isolation as a Pytest Plugin (pytest11 entry point)
+
+Implement the test isolation check as a **pytest plugin** registered
+via the `pytest11` entry point in `pyproject.toml`:
+
+```toml
+[project.entry-points.pytest11]
+devx_test_isolation = "devx.tools.check_test_isolation"
+```
+
+This makes the check **transparent and always-on** — every `pytest`
+invocation in any repo with devx installed automatically runs the
+static analysis. No extra Makefile target or CI step needed.
+
+The plugin (`devx.tools.check_test_isolation`) statically analyzes
+test files during `pytest_collection_finish` and emits
+`UserWarning` for violations:
+
+- **unpatched-subprocess**: `subprocess.run/call/Popen/check_call/check_output`
+  called in a test function without `@patch`
+- **unpatched-sleep**: `time.sleep` called without `@patch`
+- **unpatched-helper**: known subprocess-spawning helpers
+  (`update_doc_versions`, `run_cmd`, `run_tests`) called without
+  `@patch` (and without patching their internal dependencies)
+- **excessive-iterations**: `for _ in range(N)` where N > 100
+
+The plugin recognizes transitive safety: if `run_cmd` is patched,
+`run_tests` (which calls `run_cmd`) is safe. This is tracked via
+`HELPER_INTERNAL_CALLS`.
+
+A standalone CLI (`python -m devx.tools.check_test_isolation`) is also
+provided for CI gates and pre-commit hooks where pytest isn't run.
+
+### 2. Shift-Left Quality Gates in `make lint`
+
+Add `devx-check-translations` and `devx-check-test-isolation` to the
+`devx-lint` target in `devx.mak`. This means `make lint` now runs:
+
+- ruff check + format
+- pyright typecheck
+- bandit security scan
+- **translation completeness** (missing keys, dead keys, missing languages)
+- **test isolation** (unpatched subprocess, time.sleep, excessive loops)
+
+These were previously CI-only checks. Running them in `make lint`
+catches issues at the developer's machine, not in CI.
+
+### 3. Pre-commit Hook Coverage
+
+Update the pre-commit hook to run all three shift-left checks:
+test speed, translation completeness, and test isolation. This
+catches issues even earlier than `make lint` — before the commit
+is even created.
+
+## Consequences
+
+### Positive
+
+- **Automatic enforcement**: The pytest plugin runs on every `pytest`
+  invocation across devx, grm, and infra — no per-repo configuration
+  needed. New tests with unpatched subprocess calls emit warnings
+  immediately.
+- **Shift-left**: Translation gaps and test isolation violations are
+  caught locally (pre-commit / `make lint`) instead of in CI.
+- **Fast feedback**: Static analysis adds <0.1s to test runs — no
+  runtime overhead.
+- **No false positives**: The transitive dependency tracking
+  (`HELPER_INTERNAL_CALLS`) correctly recognizes that patching
+  `run_cmd` makes `run_tests` safe, and patching `subprocess.run`
+  makes all helpers safe.
+
+### Negative
+
+- **Coverage instrumentation gap**: The pytest plugin module is loaded
+  before coverage starts, so module-level code (decorators, class
+  definitions) appears uncovered. Mitigated by `-p no:devx_test_isolation`
+  in devx's own `pyproject.toml` `addopts` and `# pragma: no cover` on
+  plugin hook functions.
+- **Static analysis limitations**: The plugin only sees direct calls
+  in test function bodies, not indirect calls through `main()` or
+  other wrappers. This is acceptable — the `check_test_speed` tool
+  catches the symptom (slow tests) for indirect cases.
+- **Translation burden**: Every new `_()` call in source requires
+  adding 6 language translations. This is by design (all supported
+  languages must be complete) but adds friction for quick prototypes.
+
+## Implementation Details
+
+### Pytest Plugin Discovery
+
+The `pytest11` entry point is the standard mechanism for pytest
+plugins. When devx is installed (via pip), pytest auto-discovers
+the plugin. No `conftest.py` or `pytest_plugins` declaration needed
+in consumer repos.
+
+### Disabling the Plugin
+
+- `--no-test-isolation` flag: disables analysis for a single run
+- `-p no:devx_test_isolation` in `addopts`: disables for a repo
+  (used in devx's own `pyproject.toml` for coverage reasons)
+
+### Strict Mode
+
+- `--strict-test-isolation` flag: promotes warnings to errors and
+  prints a summary to stderr
+- `filterwarnings = ["error:Test isolation:UserWarning"]` in
+  `pyproject.toml`: same effect via pytest's warning filter system
+
+### Known Subprocess Helpers
+
+The `KNOWN_SUBPROCESS_HELPERS` dict maps function names to
+descriptions. `HELPER_INTERNAL_CALLS` maps each helper to the
+function names it internally calls, enabling transitive safety
+checks. Both are defined in `check_test_isolation.py` and can be
+extended as new subprocess-spawning helpers are added to devx.