devx/docs/decisions/0001-test-isolation-pytest-plugin-and-shift-left-quality-gates.md

ADR-0001: Test Isolation Pytest Plugin and Shift-Left Quality Gates

Date: 2026-07-13 Status: Accepted

Context

Unit tests in devx were slow (10s+) and getting slower. Investigation revealed two root causes:

1. Unpatched subprocess calls — test functions calling subprocess.run, update_doc_versions, or run_cmd without @patch decorators, causing real subprocess execution during tests.
2. Excessive iterations — statistical tests with 1000-iteration loops that should use property-based testing or smaller samples.

These issues were discovered manually by profiling with pytest --durations=0. There was no automated check to prevent regressions — new tests could introduce the same patterns and slow down the suite again.

Additionally, translation completeness checks (devx.ci.check_translations) only ran in CI, not locally. Developers discovered missing translations at CI time, wasting round-trips.

Decision

1. Test Isolation as a Pytest Plugin (pytest11 entry point)

Implement the test isolation check as a pytest plugin registered via the pytest11 entry point in pyproject.toml:

[project.entry-points.pytest11]
devx_test_isolation = "devx.tools.check_test_isolation"

This makes the check transparent and always-on — every pytest invocation in any repo with devx installed automatically runs the static analysis. No extra Makefile target or CI step needed.

The plugin (devx.tools.check_test_isolation) statically analyzes test files during pytest_collection_finish and emits UserWarning for violations:

• unpatched-subprocess: subprocess.run/call/Popen/check_call/check_output called in a test function without @patch
• unpatched-sleep: time.sleep called without @patch
• unpatched-helper: known subprocess-spawning helpers (update_doc_versions, run_cmd, run_tests) called without @patch (and without patching their internal dependencies)
• excessive-iterations: for _ in range(N) where N > 100

The plugin recognizes transitive safety: if run_cmd is patched, run_tests (which calls run_cmd) is safe. This is tracked via HELPER_INTERNAL_CALLS.

A standalone CLI (python -m devx.tools.check_test_isolation) is also provided for CI gates and pre-commit hooks where pytest isn't run.

2. Shift-Left Quality Gates in make lint

Add devx-check-translations and devx-check-test-isolation to the devx-lint target in devx.mak. This means make lint now runs:

• ruff check + format
• pyright typecheck
• bandit security scan
• translation completeness (missing keys, dead keys, missing languages)
• test isolation (unpatched subprocess, time.sleep, excessive loops)

These were previously CI-only checks. Running them in make lint catches issues at the developer's machine, not in CI.

3. Pre-commit Hook Coverage

Update the pre-commit hook to run all three shift-left checks: test speed, translation completeness, and test isolation. This catches issues even earlier than make lint — before the commit is even created.

Consequences

Positive

• Automatic enforcement: The pytest plugin runs on every pytest invocation across devx, grm, and infra — no per-repo configuration needed. New tests with unpatched subprocess calls emit warnings immediately.
• Shift-left: Translation gaps and test isolation violations are caught locally (pre-commit / make lint) instead of in CI.
• Fast feedback: Static analysis adds <0.1s to test runs — no runtime overhead.
• No false positives: The transitive dependency tracking (HELPER_INTERNAL_CALLS) correctly recognizes that patching run_cmd makes run_tests safe, and patching subprocess.run makes all helpers safe.

Negative

• Coverage instrumentation gap: The pytest plugin module is loaded before coverage starts, so module-level code (decorators, class definitions) appears uncovered. Mitigated by -p no:devx_test_isolation in devx's own pyproject.toml addopts and # pragma: no cover on plugin hook functions.
• Static analysis limitations: The plugin only sees direct calls in test function bodies, not indirect calls through main() or other wrappers. This is acceptable — the check_test_speed tool catches the symptom (slow tests) for indirect cases.
• Translation burden: Every new _() call in source requires adding 6 language translations. This is by design (all supported languages must be complete) but adds friction for quick prototypes.

Implementation Details

Pytest Plugin Discovery

The pytest11 entry point is the standard mechanism for pytest plugins. When devx is installed (via pip), pytest auto-discovers the plugin. No conftest.py or pytest_plugins declaration needed in consumer repos.

Disabling the Plugin

• --no-test-isolation flag: disables analysis for a single run
• -p no:devx_test_isolation in addopts: disables for a repo (used in devx's own pyproject.toml for coverage reasons)

Strict Mode

• --strict-test-isolation flag: promotes warnings to errors and prints a summary to stderr
• filterwarnings = ["error:Test isolation:UserWarning"] in pyproject.toml: same effect via pytest's warning filter system

Known Subprocess Helpers

The KNOWN_SUBPROCESS_HELPERS dict maps function names to descriptions. HELPER_INTERNAL_CALLS maps each helper to the function names it internally calls, enabling transitive safety checks. Both are defined in check_test_isolation.py and can be extended as new subprocess-spawning helpers are added to devx.