devx/docs/decisions/0001-test-isolation-pytest-plugin-and-shift-left-quality-gates.md

ADR-0001: Test Isolation Pytest Plugin and Shift-Left Quality Gates

Date: 2026-07-13 Status: Accepted

Context

Unit tests in devx were slow (10s+) and getting slower. Investigation revealed two root causes:

1. Unpatched subprocess calls — test functions calling subprocess.run, update_doc_versions, or run_cmd without @patch decorators, causing real subprocess execution during tests.
2. Excessive iterations — statistical tests with 1000-iteration loops that should use property-based testing or smaller samples.

These issues were discovered manually by profiling with pytest --durations=0. There was no automated check to prevent regressions — new tests could introduce the same patterns and slow down the suite again.

Additionally, translation completeness checks (devx.ci.check_translations) only ran in CI, not locally. Developers discovered missing translations at CI time, wasting round-trips.

Decision

1. Test Isolation as a Pytest Plugin (pytest11 entry point)

Implement the test isolation check as a pytest plugin registered via the pytest11 entry point in pyproject.toml:

[project.entry-points.pytest11]
devx_test_isolation = "devx.tools.check_test_isolation"

This makes the check transparent and always-on — every pytest invocation in any repo with devx installed automatically runs the static analysis. No extra Makefile target or CI step needed.

The plugin (devx.tools.check_test_isolation) statically analyzes test files during pytest_collection_finish and fails the test run on any hard violation:

• unpatched-subprocess: subprocess.run/call/Popen/check_call/check_output called in a test function without @patch or with patch(...)
• unpatched-sleep: time.sleep called without @patch
• unpatched-helper: known subprocess-spawning helpers (update_doc_versions, run_cmd, run_tests) called without @patch (and without patching their internal dependencies)
• excessive-iterations: for _ in range(N) where N > 100
• heavy-module-import: httpx, ansible, etc. imported at module level in test files, slowing collection for all tests
• reload-without-cleanup: importlib.reload() called an odd number of times, leaving module state modified

Transitive-subprocess findings (via call-graph analysis) are reported as advisories — the static analysis can't predict early exits or runtime branch conditions, so the runtime audit is authoritative.

The plugin also wraps subprocess.run at runtime to catch real subprocess calls that leak through transitive call paths (for example CliRunner.invoke(main) → main() → update_doc_versions() → subprocess.run()). If a test spawns a real subprocess without @patch, the test fails.

A standalone CLI (python -m devx.tools.check_test_isolation) is also provided for CI gates and pre-commit hooks where pytest isn't run.

2. Shift-Left Quality Gates in make lint

Add devx-check-translations and devx-check-test-isolation to the devx-lint target in devx.mak. This means make lint now runs:

• ruff check + format
• pyright typecheck
• bandit security scan
• translation completeness (missing keys, dead keys, missing languages)
• test isolation (unpatched subprocess, time.sleep, excessive loops)

These were previously CI-only checks. Running them in make lint catches issues at the developer's machine, not in CI.

3. Pre-commit Hook Coverage

Update the pre-commit hook to run all three shift-left checks: test speed, translation completeness, and test isolation. This catches issues even earlier than make lint — before the commit is even created.

Consequences

Positive

• Automatic enforcement: The pytest plugin runs on every pytest invocation across devx, grm, and infra — no per-repo configuration needed. New tests with unpatched subprocess calls fail immediately.
• Shift-left: Translation gaps and test isolation violations are caught locally (pre-commit / make lint) instead of in CI.
• Fast feedback: Static analysis adds <0.1s to test runs; runtime subprocess audit adds negligible overhead (wrapper checks a thread-local flag).
• Transitive detection: The call-graph BFS traces CliRunner.invoke(main) → main() → update_doc_versions() → subprocess.run(), catching indirect subprocess leaks that direct analysis misses. The runtime audit provides authoritative enforcement.
• No false positives: The call graph correctly recognizes that patching run_cmd makes run_tests (which calls run_cmd) safe, and class methods are excluded to avoid false positives when classes like TeaCLI are patched.

Negative

• Coverage instrumentation gap: The pytest plugin module is loaded before coverage starts, so module-level code (decorators, class definitions) appears uncovered. Mitigated by -p no:devx_test_isolation in devx's own pyproject.toml addopts and # pragma: no cover on plugin hook functions.
• Static analysis limitations: The call-graph BFS can't predict runtime branch conditions or early exits — a test that patches shutil.which to return None may skip the subprocess path entirely, but the static analysis still reports it. Transitive findings are advisories (exit 0) for this reason; the runtime audit is authoritative.
• Translation burden: Every new _() call in source requires adding 6 language translations. This is by design (all supported languages must be complete) but adds friction for quick prototypes.

Implementation Details

Pytest Plugin Discovery

The pytest11 entry point is the standard mechanism for pytest plugins. When devx is installed (via pip), pytest auto-discovers the plugin. No conftest.py or pytest_plugins declaration needed in consumer repos.

Disabling the Plugin

• --no-test-isolation flag: disables static analysis and runtime subprocess audit for a single run
• -p no:devx_test_isolation in addopts: disables for a repo (used in devx's own pyproject.toml for coverage reasons)

Call-Graph Analysis

The CallGraph class parses all .py files under src/ and builds a map of function → called functions. When a test calls CliRunner.invoke(target), a BFS traces the call graph from target to find all reachable functions. Class methods are excluded from the call graph to avoid false positives when classes are patched (for example @patch("...TeaCLI") mocks all methods). The BFS respects @patch decorators — if a function is patched, traversal stops at that node.

Runtime Subprocess Audit

The _SubprocessAudit singleton wraps subprocess.run, call, check_call, check_output, and Popen with thread-local recording wrappers. During each non-integration test, the wrapper records calls; if any are recorded (that is the test didn't @patch subprocess), the test fails. The wrappers check a thread-local flag, so inactive audits have zero overhead beyond the flag check.

Known Subprocess Helpers

The KNOWN_SUBPROCESS_HELPERS dict maps function names to descriptions. HELPER_INTERNAL_CALLS maps each helper to the function names it internally calls, enabling transitive safety checks for direct calls in test functions. The call-graph BFS handles transitive detection for CliRunner.invoke targets. Both are defined in check_test_isolation.py and can be extended as new subprocess-spawning helpers are added to devx.