# ADR-0001: Test Isolation Pytest Plugin and Shift-Left Quality Gates

Date: 2026-07-13
Status: Accepted

## Context

Unit tests in devx were slow (10s+) and getting slower. Investigation
revealed two root causes:

1. **Unpatched subprocess calls** — test functions calling
   `subprocess.run`, `update_doc_versions`, or `run_cmd` without
   `@patch` decorators, causing real subprocess execution during tests.
2. **Excessive iterations** — statistical tests with 1000-iteration
   loops that should use property-based testing or smaller samples.

These issues were discovered manually by profiling with
`pytest --durations=0`. There was no automated check to prevent
regressions — new tests could introduce the same patterns and slow
down the suite again.

Additionally, translation completeness checks
(`devx.ci.check_translations`) only ran in CI, not locally. Developers
discovered missing translations at CI time, wasting round-trips.

## Decision

### 1. Test Isolation as a Pytest Plugin (pytest11 entry point)

Implement the test isolation check as a **pytest plugin** registered
via the `pytest11` entry point in `pyproject.toml`:

```toml
[project.entry-points.pytest11]
devx_test_isolation = "devx.tools.check_test_isolation"
```

This makes the check **transparent and always-on** — every `pytest`
invocation in any repo with devx installed automatically runs the
static analysis. No extra Makefile target or CI step needed.

The plugin (`devx.tools.check_test_isolation`) statically analyzes
test files during `pytest_collection_finish` and emits
`UserWarning` for violations:

- **unpatched-subprocess**: `subprocess.run/call/Popen/check_call/check_output`
  called in a test function without `@patch`
- **unpatched-sleep**: `time.sleep` called without `@patch`
- **unpatched-helper**: known subprocess-spawning helpers
  (`update_doc_versions`, `run_cmd`, `run_tests`) called without
  `@patch` (and without patching their internal dependencies)
- **excessive-iterations**: `for _ in range(N)` where N > 100

The plugin recognizes transitive safety: if `run_cmd` is patched,
`run_tests` (which calls `run_cmd`) is safe. This is tracked via
`HELPER_INTERNAL_CALLS`.

A standalone CLI (`python -m devx.tools.check_test_isolation`) is also
provided for CI gates and pre-commit hooks where pytest isn't run.

### 2. Shift-Left Quality Gates in `make lint`

Add `devx-check-translations` and `devx-check-test-isolation` to the
`devx-lint` target in `devx.mak`. This means `make lint` now runs:

- ruff check + format
- pyright typecheck
- bandit security scan
- **translation completeness** (missing keys, dead keys, missing languages)
- **test isolation** (unpatched subprocess, time.sleep, excessive loops)

These were previously CI-only checks. Running them in `make lint`
catches issues at the developer's machine, not in CI.

### 3. Pre-commit Hook Coverage

Update the pre-commit hook to run all three shift-left checks:
test speed, translation completeness, and test isolation. This
catches issues even earlier than `make lint` — before the commit
is even created.

## Consequences

### Positive

- **Automatic enforcement**: The pytest plugin runs on every `pytest`
  invocation across devx, grm, and infra — no per-repo configuration
  needed. New tests with unpatched subprocess calls emit warnings
  immediately.
- **Shift-left**: Translation gaps and test isolation violations are
  caught locally (pre-commit / `make lint`) instead of in CI.
- **Fast feedback**: Static analysis adds <0.1s to test runs — no
  runtime overhead.
- **No false positives**: The transitive dependency tracking
  (`HELPER_INTERNAL_CALLS`) correctly recognizes that patching
  `run_cmd` makes `run_tests` safe, and patching `subprocess.run`
  makes all helpers safe.

### Negative

- **Coverage instrumentation gap**: The pytest plugin module is loaded
  before coverage starts, so module-level code (decorators, class
  definitions) appears uncovered. Mitigated by `-p no:devx_test_isolation`
  in devx's own `pyproject.toml` `addopts` and `# pragma: no cover` on
  plugin hook functions.
- **Static analysis limitations**: The plugin only sees direct calls
  in test function bodies, not indirect calls through `main()` or
  other wrappers. This is acceptable — the `check_test_speed` tool
  catches the symptom (slow tests) for indirect cases.
- **Translation burden**: Every new `_()` call in source requires
  adding 6 language translations. This is by design (all supported
  languages must be complete) but adds friction for quick prototypes.

## Implementation Details

### Pytest Plugin Discovery

The `pytest11` entry point is the standard mechanism for pytest
plugins. When devx is installed (via pip), pytest auto-discovers
the plugin. No `conftest.py` or `pytest_plugins` declaration needed
in consumer repos.

### Disabling the Plugin

- `--no-test-isolation` flag: disables analysis for a single run
- `-p no:devx_test_isolation` in `addopts`: disables for a repo
  (used in devx's own `pyproject.toml` for coverage reasons)

### Strict Mode

- `--strict-test-isolation` flag: promotes warnings to errors and
  prints a summary to stderr
- `filterwarnings = ["error:Test isolation:UserWarning"]` in
  `pyproject.toml`: same effect via pytest's warning filter system

### Known Subprocess Helpers

The `KNOWN_SUBPROCESS_HELPERS` dict maps function names to
descriptions. `HELPER_INTERNAL_CALLS` maps each helper to the
function names it internally calls, enabling transitive safety
checks. Both are defined in `check_test_isolation.py` and can be
extended as new subprocess-spawning helpers are added to devx.