Public Access
143 lines
5.5 KiB
Markdown
143 lines
5.5 KiB
Markdown
# ADR-0001: Test Isolation Pytest Plugin and Shift-Left Quality Gates
|
|
|
|
Date: 2026-07-13
|
|
Status: Accepted
|
|
|
|
## Context
|
|
|
|
Unit tests in devx were slow (10s+) and getting slower. Investigation
|
|
revealed two root causes:
|
|
|
|
1. **Unpatched subprocess calls** — test functions calling
|
|
`subprocess.run`, `update_doc_versions`, or `run_cmd` without
|
|
`@patch` decorators, causing real subprocess execution during tests.
|
|
2. **Excessive iterations** — statistical tests with 1000-iteration
|
|
loops that should use property-based testing or smaller samples.
|
|
|
|
These issues were discovered manually by profiling with
|
|
`pytest --durations=0`. There was no automated check to prevent
|
|
regressions — new tests could introduce the same patterns and slow
|
|
down the suite again.
|
|
|
|
Additionally, translation completeness checks
|
|
(`devx.ci.check_translations`) only ran in CI, not locally. Developers
|
|
discovered missing translations at CI time, wasting round-trips.
|
|
|
|
## Decision
|
|
|
|
### 1. Test Isolation as a Pytest Plugin (pytest11 entry point)
|
|
|
|
Implement the test isolation check as a **pytest plugin** registered
|
|
via the `pytest11` entry point in `pyproject.toml`:
|
|
|
|
```toml
|
|
[project.entry-points.pytest11]
|
|
devx_test_isolation = "devx.tools.check_test_isolation"
|
|
```
|
|
|
|
This makes the check **transparent and always-on** — every `pytest`
|
|
invocation in any repo with devx installed automatically runs the
|
|
static analysis. No extra Makefile target or CI step needed.
|
|
|
|
The plugin (`devx.tools.check_test_isolation`) statically analyzes
|
|
test files during `pytest_collection_finish` and emits
|
|
`UserWarning` for violations:
|
|
|
|
- **unpatched-subprocess**: `subprocess.run/call/Popen/check_call/check_output`
|
|
called in a test function without `@patch`
|
|
- **unpatched-sleep**: `time.sleep` called without `@patch`
|
|
- **unpatched-helper**: known subprocess-spawning helpers
|
|
(`update_doc_versions`, `run_cmd`, `run_tests`) called without
|
|
`@patch` (and without patching their internal dependencies)
|
|
- **excessive-iterations**: `for _ in range(N)` where N > 100
|
|
|
|
The plugin recognizes transitive safety: if `run_cmd` is patched,
|
|
`run_tests` (which calls `run_cmd`) is safe. This is tracked via
|
|
`HELPER_INTERNAL_CALLS`.
|
|
|
|
A standalone CLI (`python -m devx.tools.check_test_isolation`) is also
|
|
provided for CI gates and pre-commit hooks where pytest isn't run.
|
|
|
|
### 2. Shift-Left Quality Gates in `make lint`
|
|
|
|
Add `devx-check-translations` and `devx-check-test-isolation` to the
|
|
`devx-lint` target in `devx.mak`. This means `make lint` now runs:
|
|
|
|
- ruff check + format
|
|
- pyright typecheck
|
|
- bandit security scan
|
|
- **translation completeness** (missing keys, dead keys, missing languages)
|
|
- **test isolation** (unpatched subprocess, time.sleep, excessive loops)
|
|
|
|
These were previously CI-only checks. Running them in `make lint`
|
|
catches issues at the developer's machine, not in CI.
|
|
|
|
### 3. Pre-commit Hook Coverage
|
|
|
|
Update the pre-commit hook to run all three shift-left checks:
|
|
test speed, translation completeness, and test isolation. This
|
|
catches issues even earlier than `make lint` — before the commit
|
|
is even created.
|
|
|
|
## Consequences
|
|
|
|
### Positive
|
|
|
|
- **Automatic enforcement**: The pytest plugin runs on every `pytest`
|
|
invocation across devx, grm, and infra — no per-repo configuration
|
|
needed. New tests with unpatched subprocess calls emit warnings
|
|
immediately.
|
|
- **Shift-left**: Translation gaps and test isolation violations are
|
|
caught locally (pre-commit / `make lint`) instead of in CI.
|
|
- **Fast feedback**: Static analysis adds <0.1s to test runs — no
|
|
runtime overhead.
|
|
- **No false positives**: The transitive dependency tracking
|
|
(`HELPER_INTERNAL_CALLS`) correctly recognizes that patching
|
|
`run_cmd` makes `run_tests` safe, and patching `subprocess.run`
|
|
makes all helpers safe.
|
|
|
|
### Negative
|
|
|
|
- **Coverage instrumentation gap**: The pytest plugin module is loaded
|
|
before coverage starts, so module-level code (decorators, class
|
|
definitions) appears uncovered. Mitigated by `-p no:devx_test_isolation`
|
|
in devx's own `pyproject.toml` `addopts` and `# pragma: no cover` on
|
|
plugin hook functions.
|
|
- **Static analysis limitations**: The plugin only sees direct calls
|
|
in test function bodies, not indirect calls through `main()` or
|
|
other wrappers. This is acceptable — the `check_test_speed` tool
|
|
catches the symptom (slow tests) for indirect cases.
|
|
- **Translation burden**: Every new `_()` call in source requires
|
|
adding 6 language translations. This is by design (all supported
|
|
languages must be complete) but adds friction for quick prototypes.
|
|
|
|
## Implementation Details
|
|
|
|
### Pytest Plugin Discovery
|
|
|
|
The `pytest11` entry point is the standard mechanism for pytest
|
|
plugins. When devx is installed (via pip), pytest auto-discovers
|
|
the plugin. No `conftest.py` or `pytest_plugins` declaration needed
|
|
in consumer repos.
|
|
|
|
### Disabling the Plugin
|
|
|
|
- `--no-test-isolation` flag: disables analysis for a single run
|
|
- `-p no:devx_test_isolation` in `addopts`: disables for a repo
|
|
(used in devx's own `pyproject.toml` for coverage reasons)
|
|
|
|
### Strict Mode
|
|
|
|
- `--strict-test-isolation` flag: promotes warnings to errors and
|
|
prints a summary to stderr
|
|
- `filterwarnings = ["error:Test isolation:UserWarning"]` in
|
|
`pyproject.toml`: same effect via pytest's warning filter system
|
|
|
|
### Known Subprocess Helpers
|
|
|
|
The `KNOWN_SUBPROCESS_HELPERS` dict maps function names to
|
|
descriptions. `HELPER_INTERNAL_CALLS` maps each helper to the
|
|
function names it internally calls, enabling transitive safety
|
|
checks. Both are defined in `check_test_isolation.py` and can be
|
|
extended as new subprocess-spawning helpers are added to devx.
|