Compare commits

...
9 Commits
Author SHA1 Message Date
devx-ci-bot add02273b6 release: v0.35.3 [skip ci] 2026-07-06 09:40:19 +00:00
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> e489fdb206 DEVX-118: fix: replace --strict with --verify for sync_wiki
Post-merge / detect-type (push) Successful in 12s
Post-merge / validate-commit-msg (push) Successful in 25s
Post-merge / vikunja (push) Successful in 33s
Post-merge / configure-repo (push) Successful in 28s
Post-merge / sync-wiki (push) Failing after 37s
Post-merge / release (push) Successful in 52s
Post-merge / publish (push) Successful in 29s
Post-merge / badges (push) Failing after 38s
The rewritten sync_wiki.py removed the --strict flag. The new git-based
approach is strict by default; --verify adds post-sync page verification.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-06 11:36:30 +02:00
devx-ci-bot 45a9c7d431 release: v0.35.2 [skip ci] 2026-07-06 08:45:42 +00:00
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> 8e1c7d03a4 DEVX-118: fix: exclude .vale directory from lint_docs scanning
Post-merge / detect-type (push) Successful in 10s
Post-merge / validate-commit-msg (push) Successful in 10s
Post-merge / configure-repo (push) Successful in 21s
Post-merge / vikunja (push) Successful in 23s
Post-merge / sync-wiki (push) Failing after 29s
Post-merge / release (push) Successful in 41s
Post-merge / publish (push) Successful in 39s
Post-merge / badges (push) Failing after 45s
Third-party Vale style packages contain README.md files with code blocks
that don't specify a language, causing false positives in lint_docs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-06 10:44:25 +02:00
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> 2de3ab4d84 DEVX-118: docs: update AGENTS.md with new tools and make targets
Post-merge / detect-type (push) Successful in 11s
Post-merge / validate-commit-msg (push) Successful in 11s
Post-merge / release (push) Successful in 16s
Post-merge / publish (push) Has been skipped
Post-merge / vikunja (push) Successful in 17s
Post-merge / sync-wiki (push) Failing after 22s
Post-merge / configure-repo (push) Successful in 15s
Post-merge / badges (push) Failing after 31s
Document check_doc_versions.py, Vale, and new make targets in AGENTS.md.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-06 10:29:47 +02:00
devx-ci-bot fa501adfbc release: v0.35.1 [skip ci] 2026-07-06 08:27:00 +00:00
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> 0a5625b70b DEVX-118: refactor: rewrite sync_wiki.py to use git-based approach
Post-merge / detect-type (push) Successful in 10s
Post-merge / validate-commit-msg (push) Successful in 9s
Post-merge / vikunja (push) Successful in 21s
Post-merge / configure-repo (push) Successful in 23s
Post-merge / sync-wiki (push) Failing after 28s
Post-merge / release (push) Successful in 42s
Post-merge / publish (push) Successful in 22s
Post-merge / badges (push) Failing after 31s
Replace the unreliable Gitea wiki API with direct Git operations:
- Clone {repo}.wiki.git, copy docs with link transformation, push
- Faster: single git push vs N API calls
- More reliable: no API timeouts or rate limits
- Atomic: all pages sync in one commit
- Auto-pruning: stale wiki pages removed automatically
- Link transformation: [text](file.md) → [text](file) for wiki format
- 36 new tests covering transform_links, clone, sync_files, commit, verify

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-06 10:26:00 +02:00
devx-ci-bot f28ba432ce release: v0.35.0 [skip ci] 2026-07-06 08:16:54 +00:00
emilandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> fb342e7b9d DEVX-118: feat: enrich lint_docs.py with single H1, max depth, line length, code block lang, orphan checks
Post-merge / detect-type (push) Successful in 10s
Post-merge / validate-commit-msg (push) Successful in 9s
Post-merge / vikunja (push) Successful in 19s
Post-merge / configure-repo (push) Successful in 16s
Post-merge / release (push) Successful in 40s
Post-merge / sync-wiki (push) Successful in 43s
Post-merge / publish (push) Successful in 28s
Post-merge / badges (push) Failing after 36s
- Add check_single_h1: each markdown file should have at most one H1
- Add check_max_heading_depth: headings should not exceed H4 (configurable)
- Add check_line_length: warn on lines >120 chars (non-blocking — badge URLs)
- Add check_code_block_languages: fenced code blocks must specify a language
- Add check_orphan_docs: warn on docs not linked from index.md or mapping.json
- Fix all code blocks in docs to specify language (text for plain blocks)
- Fix duplicate H1 in .vale/styles/devx/README.md
- Add 18 new tests for full coverage of new checks

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-06 10:15:54 +02:00
15 changed files with 1006 additions and 895 deletions
+1 -1
View File
@@ -190,7 +190,7 @@ jobs:
PYTHONPATH: src
run: |
. .venv/bin/activate 2>/dev/null || true
python3 -m devx.ci.sync_wiki --repo "${{ github.repository }}" --strict
python3 -m devx.ci.sync_wiki --repo "${{ github.repository }}" --verify
- name: Notify on failure
if: failure()
env:
+2 -1
View File
@@ -1,2 +1,3 @@
# Custom Vale style for devx documentation
# Project-specific terminology and style rules
Project-specific terminology and style rules
+1 -1
View File
@@ -2,7 +2,7 @@ Based on [write-good](https://github.com/btford/write-good).
> Naive linter for English prose for developers who can't write good and wanna learn to do other stuff good too.
```
```text
The MIT License (MIT)
Copyright (c) 2014 Brian Ford
+9 -6
View File
@@ -18,19 +18,21 @@ venv activation automatically — always prefer `make <target>` over raw command
```bash
make setup # Create venv, install deps, set up hooks, install CI tools
make install-tools # Install actionlint, git-cliff, act_runner, tea, hadolint to ~/.local/bin
make install-tools # Install actionlint, git-cliff, act_runner, tea, hadolint, vale to ~/.local/bin
make lint-all # ruff + pyright + bandit + actionlint + lint-dockerfiles
make pytest-cov # Unit tests with 100% coverage enforcement
make test-unit # Unit tests without coverage
make workflow-lint # Static lint of .gitea/workflows/*.yml (actionlint)
make workflow-dryrun # Dry-run all workflows in Docker (act_runner exec --dryrun)
make workflow-check # workflow-lint + workflow-dryrun
make devx-check-doc-versions # Verify docs version refs match __version__
make devx-vale # Run Vale prose linter on docs and README
make clean # Remove caches, build artifacts, coverage data
```
`make setup` automatically installs all development tools:
- **Python deps** via `python -m devx.tools.setup` (pip install -e .[dev], pre-commit hooks)
- **actionlint, git-cliff, act_runner, tea, hadolint** via `python -m devx.tools.install_tools` (CI/CD tools to ~/.local/bin)
- **actionlint, git-cliff, act_runner, tea, hadolint, vale** via `python -m devx.tools.install_tools` (CI/CD tools to ~/.local/bin)
- **tea CLI login** via `python -m devx.tools.setup` (configures `tea login` from `.env` `CI_GITEA_TOKEN`)
## Workflow Verification (Before Push)
@@ -57,7 +59,7 @@ devx is a reusable Python package providing development and CI/CD tools for obla
### Package Structure
```
```text
src/devx/
├── __init__.py # Version (single source of truth, read by setuptools)
├── cli.py # Click-based CLI entry point (devx command)
@@ -86,11 +88,12 @@ src/devx/
│ ├── integration_guard.py # Run pytest with cross-runner fail-fast
│ ├── check_translations.py # Translation completeness check
│ ├── doc_coverage.py # Documentation coverage check
│ └── lint_docs.py # Documentation linter (structure, links, headings)
│ └── lint_docs.py # Documentation linter (structure, links, headings, code blocks, orphans)
├── tools/ # Developer tooling modules (run locally or by CI)
│ ├── setup.py # Environment setup (venv, deps, hooks)
│ ├── install_tools.py # Install actionlint, git-cliff, act_runner, tea, hadolint
│ ├── install_tools.py # Install actionlint, git-cliff, act_runner, tea, hadolint, vale
│ ├── install_checkmake.py # Install checkmake (Makefile linter)
│ ├── check_doc_versions.py # Verify docs version refs match __version__
│ ├── build_image.py # Build and push Docker images to Gitea registry
│ ├── clean_images.py # Clean up old Docker image versions from Gitea registry
│ ├── check_test_speed.py # Measure unit test execution time
@@ -158,7 +161,7 @@ git checkout -b DEVX-N-short-description
### 4. Commit (Conventional Commits)
Branch commits use conventional commit format (no `DEVX-N:` prefix):
```
```text
feat: add new feature
fix: resolve bug
docs: update README
+24
View File
@@ -2,6 +2,30 @@
All notable changes to this project will be documented in this file.
## [0.35.3] - 2026-07-06
### Bug Fixes
- Replace --strict with --verify for sync_wiki
## [0.35.2] - 2026-07-06
### Bug Fixes
- Exclude .vale directory from lint_docs scanning
## [0.35.1] - 2026-07-06
### Refactor
- Rewrite sync_wiki.py to use git-based approach
## [0.35.0] - 2026-07-06
### Features
- Enrich lint_docs.py with single H1, max depth, line length, code block lang, orphan checks
## [0.34.0] - 2026-07-06
### Features
+4 -4
View File
@@ -87,7 +87,7 @@ extra index and list devx in your dependencies:
```toml
[project]
dependencies = [
"devx>=0.34.0",
"devx>=0.35.3",
]
[tool.pip]
@@ -101,8 +101,8 @@ pip install -e .
```
> **Note:** If your project requires a specific devx version, pin it in
> `dependencies` (for example, `"devx==0.34.0"`) or use a version constraint
> (for example, `"devx>=0.34.0,<0.35"`).
> `dependencies` (for example, `"devx==0.35.3"`) or use a version constraint
> (for example, `"devx>=0.35.3,<0.36"`).
### Optional extras
@@ -434,7 +434,7 @@ devx is a self-contained Python package under `src/devx/`. It never imports
from scripts outside the package. All tools are invoked via
`python -m devx.ci.*`, `python -m devx.tools.*`, or `python -m devx.molecule.*`.
```
```text
src/devx/
├── __init__.py # Version (single source of truth, read by setuptools)
├── cli.py # Click-based CLI entry point (devx command)
+2 -2
View File
@@ -74,14 +74,14 @@ Add devx to your `pyproject.toml` dependencies and configure the registry:
```toml
[project]
dependencies = [
"devx>=0.34.0",
"devx>=0.35.3",
]
[tool.pip]
extra-index-url = "https://git.oblachno.oblachno.fyi/api/packages/oblachno-oss/pypi/simple"
```
Pin a specific version if needed: `"devx==0.34.0"` or `"devx>=0.34.0,<0.35"`.
Pin a specific version if needed: `"devx==0.35.3"` or `"devx>=0.35.3,<0.36"`.
### Optional extras
+5 -5
View File
@@ -6,7 +6,7 @@ from scripts outside the package.
## Package structure
```
```text
src/devx/
├── __init__.py # Version (single source of truth, read by setuptools)
├── cli.py # Click-based CLI entry point (devx command)
@@ -439,7 +439,7 @@ v2 failures. Supports loading custom platforms from a JSON file.
### PR lifecycle
```
```text
Developer creates Vikunja task (DEVX-N)
@@ -475,7 +475,7 @@ CI workflow (ci.yml) triggers:
### Post-merge flow
```
```text
Push to master (squash-merge commit: "DEVX-N <conventional commit>")
@@ -519,7 +519,7 @@ Post-merge workflow (post-merge.yml) triggers:
### Publish flow
```
```text
Tag push (vX.Y.Z) triggers publish workflow (publish.yml):
@@ -536,7 +536,7 @@ Tag push (vX.Y.Z) triggers publish workflow (publish.yml):
### Badge generation flow
```
```text
push_badges.py:
├── fetch_latest_master() → git fetch + reset --hard origin/master
+2 -2
View File
@@ -6,7 +6,7 @@ tag-triggered publishing.
## Workflow overview
```
```text
PR opened/synchronized ──► CI (ci.yml)
│ ├── quality
│ ├── detect-changes
@@ -143,7 +143,7 @@ updates.
### Job dependency graph
```
```text
detect-type ──┬── validate-commit-msg (skip if release commit)
├── release (skip if release commit)
│ │
+3 -3
View File
@@ -48,12 +48,12 @@ Add devx to your `pyproject.toml`:
```toml
[project]
dependencies = [
"devx>=0.34.0",
"devx>=0.35.3",
]
[project.optional-dependencies]
dev = [
"devx>=0.34.0",
"devx>=0.35.3",
]
```
@@ -72,7 +72,7 @@ tea CLI, etc.) and configure pre-commit hooks.
devx expects a `docs/` directory with at minimum:
```
```text
docs/
├── index.md # Documentation home page
├── mapping.json # Wiki page title mappings
+1 -1
View File
@@ -1,3 +1,3 @@
"""devx — reusable development and CI/CD tools for oblachno-oss projects."""
__version__ = "0.34.0"
__version__ = "0.35.3"
+174 -2
View File
@@ -7,6 +7,12 @@ Checks performed (all configurable via pyproject.toml ``[tool.devx.docs]``):
- **Broken internal links**: relative paths and anchors in markdown files
must resolve to actual files and headings.
- **Heading hierarchy**: no skipping heading levels (e.g., ``#`` → ``###``).
- **Single H1**: each markdown file should have at most one H1 heading.
- **Max heading depth**: headings should not exceed H4 (configurable).
- **Max line length**: lines should not exceed 120 characters (configurable).
- **Code block language**: fenced code blocks should specify a language.
- **Orphan docs**: docs not linked from index.md or mapping.json (warning).
- **Mapping completeness**: all docs/*.md should be in mapping.json (warning).
- **TODO/FIXME**: flags leftover TODO/FIXME markers in documentation.
- **Stale docs**: files not modified in >180 days (warning only).
- **Trailing whitespace**: lines should not end with whitespace.
@@ -49,6 +55,15 @@ REQUIRED_DOC_FILES = ["index.md"]
# Maximum age for docs before they're considered stale (days)
STALE_THRESHOLD_DAYS = 180
# Maximum heading depth (H4 by default)
MAX_HEADING_DEPTH = 4
# Maximum line length
MAX_LINE_LENGTH = 120
# Code block without language: ``` followed by optional whitespace only
_CODE_BLOCK_NO_LANG_RE = re.compile(r"^```[ \t]*$", re.MULTILINE)
# Files excluded from duplicate heading checks (auto-generated or structured
# with repeated subsections under different parent sections)
DUPLICATE_HEADING_EXCLUDES = {
@@ -70,6 +85,7 @@ _EXCLUDE_DIRS = {
".pytest_cache",
".devin",
".terraform",
".vale",
"site-packages",
"dist-info",
}
@@ -318,6 +334,120 @@ def check_duplicate_headings(root: Path) -> list[str]:
return issues
def check_single_h1(root: Path) -> list[str]:
"""Check that each markdown file has at most one H1 heading."""
issues: list[str] = []
md_files = [f for f in root.rglob("*.md") if not any(part in _EXCLUDE_DIRS for part in f.parts)]
for md_file in md_files:
rel_path = md_file.relative_to(root)
if md_file.name in DUPLICATE_HEADING_EXCLUDES:
continue
content = strip_code_blocks(md_file.read_text(encoding="utf-8"))
h1_count = len(re.findall(r"^#\s+", content, re.MULTILINE))
if h1_count > 1:
issues.append(f"{rel_path}: {h1_count} H1 headings — should have at most 1")
return issues
def check_max_heading_depth(root: Path) -> list[str]:
"""Check that headings don't exceed MAX_HEADING_DEPTH."""
issues: list[str] = []
md_files = [f for f in root.rglob("*.md") if not any(part in _EXCLUDE_DIRS for part in f.parts)]
for md_file in md_files:
rel_path = md_file.relative_to(root)
content = strip_code_blocks(md_file.read_text(encoding="utf-8"))
for match in re.finditer(r"^(#{1,6})\s+", content, re.MULTILINE):
level = len(match.group(1))
if level > MAX_HEADING_DEPTH:
line_num = content[: match.start()].count("\n") + 1
issues.append(f"{rel_path}:{line_num}: heading depth H{level} exceeds max H{MAX_HEADING_DEPTH}")
return issues
def check_line_length(root: Path) -> list[str]:
"""Check that no lines exceed MAX_LINE_LENGTH characters."""
issues: list[str] = []
md_files = [f for f in root.rglob("*.md") if not any(part in _EXCLUDE_DIRS for part in f.parts)]
for md_file in md_files:
rel_path = md_file.relative_to(root)
content = md_file.read_text(encoding="utf-8")
for i, line in enumerate(content.splitlines(), 1):
if len(line) > MAX_LINE_LENGTH:
issues.append(f"{rel_path}:{i}: line too long ({len(line)} > {MAX_LINE_LENGTH} chars)")
return issues
def check_code_block_languages(root: Path) -> list[str]:
"""Check that fenced code blocks specify a language."""
issues: list[str] = []
md_files = [f for f in root.rglob("*.md") if not any(part in _EXCLUDE_DIRS for part in f.parts)]
for md_file in md_files:
rel_path = md_file.relative_to(root)
content = md_file.read_text(encoding="utf-8")
in_code_block = False
for i, line in enumerate(content.splitlines(), 1):
stripped = line.strip()
if stripped.startswith("```"):
if not in_code_block:
# Opening fence — check for language
if _CODE_BLOCK_NO_LANG_RE.match(line):
issues.append(f"{rel_path}:{i}: code block without language specifier")
in_code_block = True
else:
# Closing fence
in_code_block = False
return issues
def check_orphan_docs(root: Path, docs_dir: Path) -> list[str]:
"""Check for docs not linked from index.md or mapping.json (warnings)."""
issues: list[str] = []
if not docs_dir.is_dir():
return issues
# Collect all referenced files from index.md and mapping.json
referenced: set[str] = set()
index_file = docs_dir / "index.md"
if index_file.exists():
content = index_file.read_text(encoding="utf-8")
for match in _LINK_RE.finditer(content):
url = match.group(2).strip()
if not url.startswith(("http://", "https://", "mailto:")):
referenced.add(url.split("#")[0])
mapping_file = docs_dir / "mapping.json"
if mapping_file.exists():
try:
mapping = json.loads(mapping_file.read_text(encoding="utf-8"))
if isinstance(mapping, dict):
# Add both keys (filenames) and values (wiki page names)
for k, v in mapping.items():
if isinstance(k, str):
referenced.add(k)
if isinstance(v, str):
referenced.add(v)
except (json.JSONDecodeError, AttributeError):
pass
# Check each doc file
for md_file in sorted(docs_dir.rglob("*.md")):
if md_file.name == "index.md":
continue
rel_path = md_file.relative_to(docs_dir).as_posix()
if rel_path not in referenced and md_file.name not in referenced:
issues.append(f"docs/{rel_path}: orphan doc — not linked from index.md or mapping.json")
return issues
@click.command()
@click.option("--root", default=".", help="Repository root directory.")
@click.option("--docs-dir", default=None, help="Docs directory (default: <root>/docs).")
@@ -327,6 +457,11 @@ def check_duplicate_headings(root: Path) -> list[str]:
@click.option("--check-stale/--no-check-stale", default=False, help="Check for stale docs.")
@click.option("--check-trailing/--no-check-trailing", default=True, help="Check trailing whitespace.")
@click.option("--check-duplicates/--no-check-duplicates", default=True, help="Check duplicate headings.")
@click.option("--check-single-h1/--no-check-single-h1", "single_h1", default=True, help="Check single H1 per file.")
@click.option("--check-depth/--no-check-depth", "depth", default=True, help="Check max heading depth.")
@click.option("--check-line-length/--no-check-line-length", "line_length", default=True, help="Check line length.")
@click.option("--check-code-lang/--no-check-code-lang", "code_lang", default=True, help="Check code block languages.")
@click.option("--check-orphans/--no-check-orphans", "orphans", default=False, help="Check for orphan docs (warnings).")
@click.option("--fix", is_flag=True, default=False, help="Auto-fix trailing whitespace.")
def main(
root: str,
@@ -337,6 +472,11 @@ def main(
check_stale: bool,
check_trailing: bool,
check_duplicates: bool,
single_h1: bool,
depth: bool,
line_length: bool,
code_lang: bool,
orphans: bool,
fix: bool,
) -> None:
"""Lint documentation files for structure, links, and quality."""
@@ -369,6 +509,31 @@ def main(
click.echo(_("Checking duplicate headings..."))
all_issues.extend(check_duplicate_headings(root_path))
# Single H1
if single_h1:
click.echo(_("Checking single H1 per file..."))
all_issues.extend(check_single_h1(root_path))
# Max heading depth
if depth:
click.echo(_("Checking max heading depth..."))
all_issues.extend(check_max_heading_depth(root_path))
# Line length (warnings — badge URLs and tables can exceed 120)
if line_length:
click.echo(_("Checking line length..."))
ll_issues = check_line_length(root_path)
for issue in ll_issues[:10]: # Show first 10 only
click.echo(f" WARN: {issue}")
if len(ll_issues) > 10:
click.echo(_(" ... and {n} more", n=len(ll_issues) - 10))
click.echo(_(" {n} long lines found (warnings only)", n=len(ll_issues)))
# Code block languages
if code_lang:
click.echo(_("Checking code block languages..."))
all_issues.extend(check_code_block_languages(root_path))
# TODO/FIXME
if check_todo:
click.echo(_("Checking for TODO/FIXME markers..."))
@@ -391,15 +556,22 @@ def main(
else:
all_issues.extend(ws_issues)
# Stale docs
# Stale docs (warnings)
if check_stale:
click.echo(_("Checking for stale docs..."))
stale = check_stale_docs(root_path)
for issue in stale:
click.echo(f" WARN: {issue}")
# Stale docs are warnings, not errors
click.echo(_(" {n} stale docs found (warnings only)", n=len(stale)))
# Orphan docs (warnings)
if orphans:
click.echo(_("Checking for orphan docs..."))
orphan_issues = check_orphan_docs(root_path, docs_path)
for issue in orphan_issues:
click.echo(f" WARN: {issue}")
click.echo(_(" {n} orphan docs found (warnings only)", n=len(orphan_issues)))
# Report
click.echo(f"\n{'=' * 60}")
if all_issues:
+217 -315
View File
@@ -1,17 +1,24 @@
#!/usr/bin/env python3
"""Sync documentation from /docs/ to the Gitea wiki via API.
"""Sync documentation from /docs/ to the Gitea wiki via Git.
Reads markdown files from the ``docs/`` directory, uses ``mapping.json`` to
map file paths to wiki page titles, and creates/updates wiki pages via the
Gitea API. Pages that exist in the wiki but not in the mapping are left
untouched (not deleted).
Instead of using the Gitea wiki API (which is slow, unreliable, and
prone to timeouts), this module clones the wiki Git repository,
copies the documentation files into it, transforms internal links
to wiki-friendly format, commits, and pushes.
Gitea 1.26 wiki API endpoints (all use content_base64, NOT content):
- Create: POST /repos/{owner}/{repo}/wiki/new {title, content_base64, message}
- Update: PATCH /repos/{owner}/{repo}/wiki/page/{sub_url} {title, content_base64, message}
- List: GET /repos/{owner}/{repo}/wiki/pages [{title, sub_url, ...}]
- Fetch: GET /repos/{owner}/{repo}/wiki/page/{sub_url} {title, content_base64, ...}
- Delete: DELETE /repos/{owner}/{repo}/wiki/page/{sub_url}
This approach is:
- **Faster** a single git push vs N API calls
- **More reliable** no API timeouts or rate limits
- **Atomic** all pages sync in one commit
- **Auto-pruning** stale wiki pages are removed automatically
The wiki Git URL is ``{clone_url}.wiki.git`` (Gitea convention).
Link transformations:
- ``[text](file.md)`` ``[text](file)`` (wiki pages don't use .md)
- ``[text](docs/file.md)`` ``[text](file)``
- External links (http/https/mailto) are preserved
- Anchor-only links (``#section``) are preserved
Usage:
CI_GITEA_TOKEN=<token> python3 -m devx.ci.sync_wiki [--dry-run] [--repo owner/repo]
@@ -19,44 +26,30 @@ Usage:
from __future__ import annotations
import base64
import json
import logging
import os
import re
import subprocess # nosec B404
import tempfile
from pathlib import Path
import click
from dotenv import load_dotenv # pyright: ignore[reportMissingImports,reportUnknownVariableType]
from tenacity import (
before_sleep_log,
retry,
retry_if_exception_type,
stop_after_attempt,
wait_exponential,
)
from devx.api_clients import GiteaClient
from devx.config import GITEA_API_URL, REPO_NAME, REPO_OWNER
from devx.exceptions import APIError
from devx.i18n import _
load_dotenv()
# DOCS_DIR is the repo's docs/ directory. When devx is installed as a
# package (e.g., in .venv/lib/python3.12/site-packages/devx/), the
# __file__-relative path would point inside the venv, not the repo.
# Use DEVX_DOCS_DIR env var if set, otherwise fall back to ./docs
# (relative to the current working directory, which is the repo root
# in CI and local development).
DOCS_DIR = Path(os.environ.get("DEVX_DOCS_DIR", "docs"))
MAPPING_FILE = DOCS_DIR / "mapping.json"
# Markdown link pattern: [text](url)
_LINK_RE = re.compile(r"\[([^\]]*)\]\(([^)]+)\)")
def load_mapping() -> dict[str, str]:
"""Load the file-to-wiki-page mapping from mapping.json.
Validates that the mapping is a dict of string-to-string pairs.
"""
"""Load the file-to-wiki-page mapping from mapping.json."""
with open(MAPPING_FILE, encoding="utf-8") as f:
data = json.load(f)
if not isinstance(data, dict):
@@ -69,214 +62,172 @@ def load_mapping() -> dict[str, str]:
return data
def read_doc_content(file_path: str) -> str:
"""Read markdown content from a docs file."""
full_path = DOCS_DIR / file_path
with open(full_path, encoding="utf-8") as f:
return f.read()
def transform_links(content: str) -> str:
"""Transform markdown links from file-based to wiki-friendly format.
def encode_content(content: str) -> str:
"""Encode content as base64 for the Gitea wiki API.
The Gitea wiki API requires content_base64, not plain content.
Sending plain content silently fails (pages are created/updated
but with empty content).
- ``[text](file.md)`` ``[text](file)``
- ``[text](docs/file.md)`` ``[text](file)``
- ``[text](../file.md)`` ``[text](file)``
- External links (http/https/mailto) preserved
- Anchor-only links (``#section``) preserved
"""
return base64.b64encode(content.encode("utf-8")).decode("ascii")
def replace_link(match: re.Match[str]) -> str:
text = match.group(1)
url = match.group(2).strip()
# Skip external links and mailto
if url.startswith(("http://", "https://", "mailto:")):
return match.group(0)
# Skip anchor-only links
if url.startswith("#"):
return match.group(0)
# Split path and anchor
if "#" in url:
path_part, anchor = url.split("#", 1)
anchor = f"#{anchor}"
else:
path_part, anchor = url, ""
# Remove .md extension and directory prefixes
if path_part.endswith(".md"):
path_part = path_part[:-3]
# Remove directory prefix (docs/, ../, etc.)
path_part = path_part.split("/")[-1]
return f"[{text}]({path_part}{anchor})"
return _LINK_RE.sub(replace_link, content)
def decode_content(content_b64: str) -> str:
"""Decode base64 content from the Gitea wiki API."""
if not content_b64:
return ""
return base64.b64decode(content_b64).decode("utf-8")
def get_wiki_clone_url(owner: str, repo: str, token: str) -> str:
"""Build the wiki Git clone URL with token auth."""
# Gitea wiki repos are at {clone_url}.wiki.git
# Extract base URL from API URL
base = GITEA_API_URL.rsplit("/api/v1", 1)[0]
return f"{base}/{owner}/{repo}.wiki.git"
def list_wiki_pages(client: GiteaClient) -> dict[str, str]:
"""List existing wiki pages, returning {title: sub_url}.
def clone_wiki(wiki_url: str, dest: Path) -> bool:
"""Clone the wiki repo into dest. Returns True if clone succeeded.
Raises :class:`APIError` if the wiki API is unavailable the caller
is responsible for retrying or handling the failure.
If the wiki repo doesn't exist yet (no pages created), returns False.
"""
pages = client._request("GET", "/wiki/pages").json()
return {page.get("title", ""): page.get("sub_url", page.get("title", "")) for page in pages}
def fetch_page_content(client: GiteaClient, sub_url: str) -> str:
"""Fetch a wiki page's content by sub_url, decoded from base64."""
try:
page = client._request("GET", f"/wiki/page/{sub_url}").json()
return decode_content(page.get("content_base64", ""))
except APIError:
return ""
def sync_page(
client: GiteaClient,
page_title: str,
content: str,
existing_pages: dict[str, str],
dry_run: bool,
) -> str:
"""Create or update a single wiki page.
Returns "created", "updated", or "skipped" (if dry-run).
If a create fails with HTTP 400 "already exists" (the page list was
stale), re-lists the wiki and falls back to an update.
"""
if dry_run:
click.echo(_("[dry-run] Would sync page: {title} ({chars} chars)", title=page_title, chars=len(content)))
return "skipped"
content_b64 = encode_content(content)
if page_title in existing_pages:
# Update existing page via PATCH
sub_url = existing_pages[page_title]
client._request(
"PATCH",
f"/wiki/page/{sub_url}",
json={
"title": page_title,
"content_base64": content_b64,
"message": f"Sync from docs/ — update {page_title}",
},
)
return "updated"
# Create new page via POST /wiki/new
try:
client._request(
"POST",
"/wiki/new",
json={
"title": page_title,
"content_base64": content_b64,
"message": f"Sync from docs/ — create {page_title}",
},
)
return "created"
except APIError as e:
if e.status == 400 and "already exists" in e.message.lower():
# The page list was stale (e.g. after a timeout-retry returned
# incomplete data). Re-list and fall back to update.
click.echo(_(" Page '{title}' already exists (stale list). Re-listing and updating...", title=page_title))
fresh_pages = _list_wiki_pages_with_retry(client)
if page_title in fresh_pages:
sub_url = fresh_pages[page_title]
client._request(
"PATCH",
f"/wiki/page/{sub_url}",
json={
"title": page_title,
"content_base64": content_b64,
"message": f"Sync from docs/ — update {page_title} (create→update fallback)",
},
)
return "updated"
raise
def verify_wiki_page(
client: GiteaClient, page_title: str, expected_content: str, existing_pages: dict[str, str]
) -> bool:
"""Verify that a wiki page has non-empty content matching the docs.
Returns True if the page content matches, False otherwise.
"""
if page_title not in existing_pages:
return False
sub_url = existing_pages[page_title]
actual = fetch_page_content(client, sub_url)
return actual.strip() == expected_content.strip()
def _list_wiki_pages_with_retry(client: GiteaClient) -> dict[str, str]:
"""List wiki pages with tenacity retry on APIError.
The Gitea wiki API can be slow (it renders pages on each request)
and may time out. Uses 5 attempts with exponential backoff to handle
transient slowness.
"""
_logger = logging.getLogger("sync_wiki")
@retry(
stop=stop_after_attempt(5),
wait=wait_exponential(multiplier=2, min=2, max=16),
retry=retry_if_exception_type(APIError),
before_sleep=before_sleep_log(_logger, logging.WARNING),
reraise=True,
result = subprocess.run( # nosec
["git", "clone", "--depth", "1", wiki_url, str(dest)],
capture_output=True,
text=True,
timeout=60,
)
def _do_list() -> dict[str, str]:
return list_wiki_pages(client)
return _do_list()
return result.returncode == 0
def verify_wiki_integrity(
client: GiteaClient,
def init_wiki(dest: Path) -> None:
"""Initialize a fresh wiki repo (when clone fails)."""
dest.mkdir(parents=True, exist_ok=True)
subprocess.run(["git", "init"], cwd=dest, capture_output=True, check=True) # nosec
subprocess.run( # nosec
["git", "config", "user.email", "ci@oblachno.fyi"],
cwd=dest,
capture_output=True,
check=True,
)
subprocess.run( # nosec
["git", "config", "user.name", "CI Wiki Sync"],
cwd=dest,
capture_output=True,
check=True,
)
def sync_files(
docs_dir: Path,
wiki_dir: Path,
mapping: dict[str, str],
synced: dict[str, str],
) -> list[str]:
"""Comprehensive wiki verification.
dry_run: bool,
) -> tuple[int, int]:
"""Copy docs files to wiki dir with link transformation.
Checks:
1. Every mapped page exists in the wiki
2. Every mapped page has non-empty content
3. Every mapped page's content matches the docs
4. No stale pages exist in the wiki (pages not in mapping)
5. Page count matches
Returns a list of failure messages (empty if all checks pass).
If the wiki API is temporarily unavailable (all retry attempts
fail), returns an empty list with a warning the sync itself
already succeeded, so a transient API outage should not fail the job.
Returns (synced, pruned) counts.
"""
failures: list[str] = []
synced = 0
try:
existing_pages = _list_wiki_pages_with_retry(client)
except APIError:
click.echo(
_(
"WARNING: Could not fetch wiki page list after retries. "
"The sync itself succeeded ({count} pages updated), but the "
"integrity check could not verify them due to a transient API issue.",
count=len(synced),
)
)
return []
# Build set of expected wiki filenames
expected_files: set[str] = set()
expected_titles = set(mapping.values())
for file_path, page_title in sorted(mapping.items()):
src = docs_dir / file_path
if not src.exists():
click.echo(_(" WARN: Mapped file {file} not found, skipping", file=file_path))
continue
# Check 1: Page count
if len(existing_pages) != len(expected_titles):
failures.append(f"Page count mismatch: wiki has {len(existing_pages)}, mapping has {len(expected_titles)}")
content = src.read_text(encoding="utf-8")
if not content.strip():
click.echo(_(" WARN: Mapped file {file} is empty, skipping", file=file_path))
continue
# Check 2: Missing pages (in mapping but not in wiki)
missing = expected_titles - set(existing_pages.keys())
for title in sorted(missing):
failures.append(f"Missing page: {title}")
# Transform links
transformed = transform_links(content)
# Check 3: Stale pages (in wiki but not in mapping)
stale = set(existing_pages.keys()) - expected_titles
for title in sorted(stale):
failures.append(f"Stale page (not in mapping): {title}")
# Wiki filename: use the page title with spaces → underscores
# Gitea wiki uses the page title as filename (spaces become dashes)
wiki_filename = page_title.replace(" ", "-") + ".md"
expected_files.add(wiki_filename)
# Check 4: Content verification
for page_title, expected_content in sorted(synced.items()):
ok = verify_wiki_page(client, page_title, expected_content, existing_pages)
if not ok:
sub_url = existing_pages.get(page_title, "?")
actual = fetch_page_content(client, sub_url)
if not actual.strip():
failures.append(f"Empty content: {page_title}")
else:
failures.append(f"Content mismatch: {page_title}")
if not dry_run:
dest = wiki_dir / wiki_filename
dest.write_text(transformed, encoding="utf-8")
synced += 1
click.echo(_(" Synced: {title}{file}", title=page_title, file=wiki_filename))
return failures
# Prune stale pages (in wiki but not in mapping)
pruned = 0
if not dry_run:
for existing in wiki_dir.glob("*.md"):
if existing.name not in expected_files:
existing.unlink()
pruned += 1
click.echo(_(" Pruned: {file} (not in mapping)", file=existing.name))
return synced, pruned
def commit_and_push(wiki_dir: Path, wiki_url: str, dry_run: bool) -> bool:
"""Commit changes and push to the wiki repo. Returns True if pushed."""
if dry_run:
click.echo(_("[dry-run] Would commit and push wiki changes"))
return False
# Stage all changes
subprocess.run(["git", "add", "-A"], cwd=wiki_dir, capture_output=True, check=True) # nosec
# Check if there are changes to commit
result = subprocess.run( # nosec
["git", "diff", "--cached", "--quiet"],
cwd=wiki_dir,
capture_output=True,
)
if result.returncode == 0:
click.echo(_("No changes to sync — wiki is up to date."))
return False
# Commit
subprocess.run( # nosec
["git", "commit", "-m", "Sync wiki from docs/ [skip ci]"],
cwd=wiki_dir,
capture_output=True,
check=True,
)
# Push
result = subprocess.run( # nosec
["git", "push", wiki_url, "HEAD:master"],
cwd=wiki_dir,
capture_output=True,
text=True,
timeout=60,
)
if result.returncode != 0:
click.echo(_("Push failed: {error}", error=result.stderr))
return False
return True
@click.command()
@@ -286,15 +237,10 @@ def verify_wiki_integrity(
"--verify",
is_flag=True,
default=False,
help="After syncing, verify each page has non-empty content. Exit 1 if any page is empty or mismatched.",
help="After syncing, verify each page exists in the wiki. Exit 1 if any page is missing.",
)
@click.option(
"--strict",
is_flag=True,
default=False,
help="Full integrity check: verify page count, missing pages, stale pages, and content. Implies --verify.",
)
def main(dry_run: bool, repo: str | None, verify: bool, strict: bool) -> None:
def main(dry_run: bool, repo: str | None, verify: bool) -> None:
"""Sync documentation to the Gitea wiki via Git."""
token = os.environ.get("CI_GITEA_TOKEN", "")
if not token:
raise click.ClickException(_("ERROR: CI_GITEA_TOKEN is not set."))
@@ -309,107 +255,63 @@ def main(dry_run: bool, repo: str | None, verify: bool, strict: bool) -> None:
raise click.ClickException(_("ERROR: mapping.json not found at {path}", path=MAPPING_FILE))
mapping = load_mapping()
client = GiteaClient(GITEA_API_URL, token, owner, repo_name)
wiki_url = get_wiki_clone_url(owner, repo_name, token)
click.echo(_("Syncing {count} documentation pages to wiki...", count=len(mapping)))
click.echo(_("Syncing {count} documentation pages to wiki via Git...", count=len(mapping)))
try:
existing_pages = _list_wiki_pages_with_retry(client)
except APIError as e:
raise click.ClickException(
with tempfile.TemporaryDirectory() as tmpdir:
wiki_dir = Path(tmpdir) / "wiki"
click.echo(_("Cloning wiki repo..."))
if clone_wiki(wiki_url, wiki_dir):
click.echo(_("Cloned existing wiki."))
else:
click.echo(_("Wiki repo not found or empty — initializing fresh."))
init_wiki(wiki_dir)
click.echo(_("Syncing files..."))
synced, pruned = sync_files(DOCS_DIR, wiki_dir, mapping, dry_run)
click.echo(
_(
"Failed to list existing wiki pages after retries: {error}. "
"Aborting to avoid creating duplicate pages.",
error=e,
"\nDone! Synced: {synced}, Pruned: {pruned}",
synced=synced,
pruned=pruned,
)
) from e
if existing_pages:
click.echo(_("Found {count} existing wiki pages.", count=len(existing_pages)))
created = 0
updated = 0
skipped = 0
synced: dict[str, str] = {} # title -> content, for verification
for file_path, page_title in sorted(mapping.items()):
try:
content = read_doc_content(file_path)
except FileNotFoundError:
raise click.ClickException(
_("Mapped file {file} not found. Update mapping.json or create the file.", file=file_path)
) from None
if not content.strip():
raise click.ClickException(
_("Mapped file {file} is empty. Update the content or remove from mapping.json.", file=file_path)
) from None
result = sync_page(client, page_title, content, existing_pages, dry_run)
if result == "created":
created += 1
click.echo(_(" Created: {title}", title=page_title))
elif result == "updated":
updated += 1
click.echo(_(" Updated: {title}", title=page_title))
else:
skipped += 1
synced[page_title] = content
click.echo(
_(
"\nDone! Created: {created}, Updated: {updated}, Skipped: {skipped}",
created=created,
updated=updated,
skipped=skipped,
)
)
# --strict implies --verify
do_verify = verify or strict
if dry_run:
click.echo(_("[dry-run] No changes pushed."))
return
if do_verify and not dry_run:
if strict:
click.echo(_("\nRunning full wiki integrity check..."))
failures = verify_wiki_integrity(client, mapping, synced)
if failures:
click.echo(_("\nIntegrity check FAILED ({count} issues):", count=len(failures)))
for f in failures:
click.echo(f" - {f}")
raise click.ClickException(_("Wiki integrity check failed — {count} issue(s)", count=len(failures)))
click.echo(_("\nIntegrity check passed — all {count} pages verified.", count=len(synced)))
else:
click.echo(_("\nVerifying wiki pages have content..."))
# Re-fetch the page list to get updated sub_urls
try:
existing_pages = _list_wiki_pages_with_retry(client)
except APIError:
click.echo(
_(
"WARNING: Could not re-fetch wiki page list for verification. "
"Skipping content verification due to transient API issue."
)
)
return
click.echo(_("Committing and pushing..."))
pushed = commit_and_push(wiki_dir, wiki_url, dry_run)
if pushed:
click.echo(_("Wiki synced successfully."))
elif not dry_run:
click.echo(_("No push needed (no changes or push failed)."))
# Verification
if verify and not dry_run:
click.echo(_("\nVerifying wiki pages..."))
# Re-clone to verify
verify_dir = Path(tmpdir) / "verify"
if not clone_wiki(wiki_url, verify_dir):
click.echo(_("FAIL: Could not clone wiki for verification."))
raise click.ClickException(_("Wiki verification failed — could not clone wiki"))
failures = 0
for page_title, expected_content in sorted(synced.items()):
ok = verify_wiki_page(client, page_title, expected_content, existing_pages)
if ok:
click.echo(_(" OK: {title} ({chars} chars)", title=page_title, chars=len(expected_content)))
for _file_path, page_title in sorted(mapping.items()):
wiki_filename = page_title.replace(" ", "-") + ".md"
if (verify_dir / wiki_filename).exists():
click.echo(_(" OK: {title}", title=page_title))
else:
click.echo(_(" FAIL: {title}content mismatch or empty!", title=page_title))
click.echo(_(" FAIL: {title}page not found in wiki!", title=page_title))
failures += 1
if failures > 0:
click.echo(
_(
"\nVerification FAILED: {failures} page(s) have empty or mismatched content!",
failures=failures,
)
)
raise click.ClickException(
_("Wiki verification failed — {failures} page(s) empty or mismatched", failures=failures)
_("Wiki verification failed — {failures} page(s) missing", failures=failures)
)
click.echo(_("\nVerification passed — all wiki pages have correct content."))
click.echo(_("\nVerification passed — all wiki pages exist."))
if __name__ == "__main__": # pragma: no cover
+155
View File
@@ -9,11 +9,16 @@ from pathlib import Path
from click.testing import CliRunner
from devx.ci.lint_docs import (
check_code_block_languages,
check_docs_structure,
check_duplicate_headings,
check_heading_hierarchy,
check_internal_links,
check_line_length,
check_max_heading_depth,
check_orphan_docs,
check_required_files,
check_single_h1,
check_stale_docs,
check_todo_fixme,
check_trailing_whitespace,
@@ -362,6 +367,100 @@ class TestCheckDuplicateHeadings:
assert issues == []
class TestCheckSingleH1:
def test_single_h1_ok(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# Title\n## Section\n")
issues = check_single_h1(tmp_path)
assert issues == []
def test_multiple_h1_fails(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# Title 1\n# Title 2\n")
issues = check_single_h1(tmp_path)
assert len(issues) == 1
assert "2 H1" in issues[0]
def test_no_h1_ok(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("## Section\n")
issues = check_single_h1(tmp_path)
assert issues == []
class TestCheckMaxHeadingDepth:
def test_ok(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# H1\n## H2\n### H3\n#### H4\n")
issues = check_max_heading_depth(tmp_path)
assert issues == []
def test_too_deep(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# H1\n##### H5\n")
issues = check_max_heading_depth(tmp_path)
assert len(issues) == 1
assert "H5" in issues[0]
class TestCheckLineLength:
def test_ok(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# Short line\n")
issues = check_line_length(tmp_path)
assert issues == []
def test_too_long(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# " + "x" * 200 + "\n")
issues = check_line_length(tmp_path)
assert len(issues) == 1
assert "202" in issues[0]
class TestCheckCodeBlockLanguages:
def test_with_language(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("```python\nprint('hi')\n```\n")
issues = check_code_block_languages(tmp_path)
assert issues == []
def test_without_language(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("```\nplain text\n```\n")
issues = check_code_block_languages(tmp_path)
assert len(issues) == 1
assert "without language" in issues[0]
def test_closing_fence_not_flagged(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("```python\nprint('hi')\n```\n")
issues = check_code_block_languages(tmp_path)
assert issues == []
class TestCheckOrphanDocs:
def test_no_orphans(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n[link](page.md)\n")
(docs / "page.md").write_text("# Page\n")
issues = check_orphan_docs(tmp_path, docs)
assert issues == []
def test_orphan_found(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "page.md").write_text("# Page\n")
issues = check_orphan_docs(tmp_path, docs)
assert len(issues) == 1
assert "orphan" in issues[0]
def test_no_docs_dir(self, tmp_path: Path) -> None:
issues = check_orphan_docs(tmp_path, tmp_path / "docs")
assert issues == []
def test_referenced_in_mapping(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text(json.dumps({"page.md": "Page"}))
(docs / "page.md").write_text("# Page\n")
issues = check_orphan_docs(tmp_path, docs)
assert issues == []
class TestMain:
def test_passes_clean_repo(self, tmp_path: Path) -> None:
"""A clean repo with all files should pass."""
@@ -434,3 +533,59 @@ class TestMain:
# Stale docs are warnings, not errors
assert result.exit_code == 0
assert "stale" in result.output
def test_line_length_warning(self, tmp_path: Path) -> None:
"""--check-line-length should warn but not fail."""
(tmp_path / "README.md").write_text("# " + "x" * 200 + "\n")
(tmp_path / "AGENTS.md").write_text("# AGENTS\n")
(tmp_path / "CHANGELOG.md").write_text("# Changelog\n")
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home"}))
runner = CliRunner()
result = runner.invoke(main, ["--root", str(tmp_path), "--check-line-length"])
assert result.exit_code == 0
assert "long lines" in result.output
def test_line_length_many_warnings(self, tmp_path: Path) -> None:
"""More than 10 long lines should show '... and N more'."""
long_line = "x" * 200 + "\n"
(tmp_path / "README.md").write_text(long_line * 15)
(tmp_path / "AGENTS.md").write_text("# AGENTS\n")
(tmp_path / "CHANGELOG.md").write_text("# Changelog\n")
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home"}))
runner = CliRunner()
result = runner.invoke(main, ["--root", str(tmp_path), "--check-line-length"])
assert result.exit_code == 0
assert "more" in result.output
def test_orphan_docs_warning(self, tmp_path: Path) -> None:
"""--check-orphans should warn but not fail."""
(tmp_path / "README.md").write_text("# Title\n")
(tmp_path / "AGENTS.md").write_text("# AGENTS\n")
(tmp_path / "CHANGELOG.md").write_text("# Changelog\n")
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home"}))
(docs / "orphan.md").write_text("# Orphan\n")
runner = CliRunner()
result = runner.invoke(main, ["--root", str(tmp_path), "--check-orphans"])
assert result.exit_code == 0
assert "orphan" in result.output
def test_orphan_docs_invalid_mapping(self, tmp_path: Path) -> None:
"""Invalid mapping.json should not crash orphan check."""
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text("invalid json{")
(docs / "page.md").write_text("# Page\n")
# Should not raise — just returns issues
issues = check_orphan_docs(tmp_path, docs)
assert len(issues) == 1
assert "orphan" in issues[0]
+406 -552
View File
@@ -1,6 +1,7 @@
"""Unit tests for scripts/ci/sync_wiki.py."""
"""Unit tests for devx.ci.sync_wiki (git-based approach)."""
from __future__ import annotations
import base64
import json
from pathlib import Path
from unittest.mock import MagicMock, patch
@@ -10,593 +11,446 @@ import pytest
from click.testing import CliRunner
from devx.ci.sync_wiki import (
decode_content,
encode_content,
fetch_page_content,
list_wiki_pages,
clone_wiki,
commit_and_push,
get_wiki_clone_url,
init_wiki,
load_mapping,
main,
read_doc_content,
sync_page,
verify_wiki_integrity,
verify_wiki_page,
sync_files,
transform_links,
)
from devx.exceptions import APIError
class TestEncodeContent:
def test_encodes_utf8_to_base64(self) -> None:
result = encode_content("# Hello World")
assert result == base64.b64encode(b"# Hello World").decode("ascii")
class TestTransformLinks:
def test_removes_md_extension(self) -> None:
result = transform_links("[link](page.md)")
assert result == "[link](page)"
def test_encodes_empty_string(self) -> None:
assert encode_content("") == ""
def test_removes_directory_prefix(self) -> None:
result = transform_links("[link](docs/page.md)")
assert result == "[link](page)"
def test_encodes_unicode(self) -> None:
result = encode_content("# Café — résumé")
decoded = base64.b64decode(result).decode("utf-8")
assert decoded == "# Café — résumé"
def test_removes_parent_dir_prefix(self) -> None:
result = transform_links("[link](../page.md)")
assert result == "[link](page)"
def test_preserves_external_links(self) -> None:
result = transform_links("[link](https://example.com)")
assert result == "[link](https://example.com)"
class TestDecodeContent:
def test_decodes_base64_to_utf8(self) -> None:
encoded = base64.b64encode(b"# Hello").decode("ascii")
assert decode_content(encoded) == "# Hello"
def test_preserves_http_links(self) -> None:
result = transform_links("[link](http://example.com)")
assert result == "[link](http://example.com)"
def test_empty_string_returns_empty(self) -> None:
assert decode_content("") == ""
def test_preserves_mailto(self) -> None:
result = transform_links("[email](mailto:test@example.com)")
assert result == "[email](mailto:test@example.com)"
def test_roundtrip(self) -> None:
original = "# Wiki Page\n\nContent with **markdown**."
encoded = encode_content(original)
assert decode_content(encoded) == original
def test_preserves_anchor_only(self) -> None:
result = transform_links("[section](#section)")
assert result == "[section](#section)"
def test_preserves_anchor_with_path(self) -> None:
result = transform_links("[section](page.md#section)")
assert result == "[section](page#section)"
def test_no_links_unchanged(self) -> None:
text = "# Title\n\nSome text without links.\n"
assert transform_links(text) == text
def test_multiple_links(self) -> None:
result = transform_links("[a](one.md) and [b](two.md)")
assert result == "[a](one) and [b](two)"
class TestLoadMapping:
def test_loads_mapping(self, tmp_path: Path) -> None:
mapping_file = tmp_path / "mapping.json"
mapping_file.write_text(json.dumps({"user/getting-started.md": "Getting-Started"}))
with patch("devx.ci.sync_wiki.MAPPING_FILE", mapping_file):
result = load_mapping()
assert result == {"user/getting-started.md": "Getting-Started"}
def test_loads_mapping(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
mapping_file = tmp_path / "docs" / "mapping.json"
mapping_file.parent.mkdir()
mapping_file.write_text(json.dumps({"index.md": "Home", "guide.md": "Guide"}))
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
mapping = load_mapping()
assert mapping == {"index.md": "Home", "guide.md": "Guide"}
def test_missing_mapping_raises(self, tmp_path: Path) -> None:
with patch("devx.ci.sync_wiki.MAPPING_FILE", tmp_path / "nonexistent.json"):
with pytest.raises(FileNotFoundError):
load_mapping()
def test_non_dict_mapping_raises(self, tmp_path: Path) -> None:
"""Non-dict mapping.json should raise."""
mapping_file = tmp_path / "mapping.json"
def test_non_dict_raises(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
mapping_file = tmp_path / "docs" / "mapping.json"
mapping_file.parent.mkdir()
mapping_file.write_text('["not", "a", "dict"]')
with patch("devx.ci.sync_wiki.MAPPING_FILE", mapping_file):
with pytest.raises(click.ClickException, match="must be a dict"):
load_mapping()
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
with pytest.raises(click.ClickException, match="must be a dict"):
load_mapping()
def test_non_string_values_raise(self, tmp_path: Path) -> None:
"""Non-string values in mapping.json should raise."""
mapping_file = tmp_path / "mapping.json"
mapping_file.write_text('{"file.md": 123}')
with patch("devx.ci.sync_wiki.MAPPING_FILE", mapping_file):
with pytest.raises(click.ClickException, match="must be strings"):
load_mapping()
def test_non_string_values_raises(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
mapping_file = tmp_path / "docs" / "mapping.json"
mapping_file.parent.mkdir()
mapping_file.write_text(json.dumps({"key": 123}))
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
with pytest.raises(click.ClickException, match="must be strings"):
load_mapping()
class TestReadDocContent:
def test_reads_file(self, tmp_path: Path) -> None:
docs_dir = tmp_path / "docs"
docs_dir.mkdir()
(docs_dir / "test.md").write_text("# Test\n\nContent")
with patch("devx.ci.sync_wiki.DOCS_DIR", docs_dir):
content = read_doc_content("test.md")
assert content == "# Test\n\nContent"
def test_missing_file_raises(self, tmp_path: Path) -> None:
with patch("devx.ci.sync_wiki.DOCS_DIR", tmp_path):
with pytest.raises(FileNotFoundError):
read_doc_content("nonexistent.md")
class TestGetWikiCloneUrl:
def test_builds_url(self) -> None:
url = get_wiki_clone_url("owner", "repo", "token")
assert "owner/repo.wiki.git" in url
class TestListWikiPages:
def test_raises_on_api_error(self) -> None:
client = MagicMock()
client._request.side_effect = APIError(404, "not found")
with pytest.raises(APIError):
list_wiki_pages(client)
class TestCloneWiki:
@patch("devx.ci.sync_wiki.subprocess.run")
def test_clone_success(self, mock_run: MagicMock, tmp_path: Path) -> None:
mock_run.return_value = MagicMock(returncode=0, stdout="", stderr="")
result = clone_wiki("https://example.com/repo.wiki.git", tmp_path / "wiki")
assert result is True
def test_returns_page_dict(self) -> None:
client = MagicMock()
client._request.return_value.json.return_value = [
{"title": "Home", "sub_url": "Home"},
{"title": "Getting-Started", "sub_url": "Getting-Started.-"},
@patch("devx.ci.sync_wiki.subprocess.run")
def test_clone_failure_returns_false(self, mock_run: MagicMock, tmp_path: Path) -> None:
mock_run.return_value = MagicMock(returncode=1, stdout="", stderr="not found")
result = clone_wiki("https://example.com/repo.wiki.git", tmp_path / "wiki")
assert result is False
class TestInitWiki:
@patch("devx.ci.sync_wiki.subprocess.run")
def test_init_calls_git(self, mock_run: MagicMock, tmp_path: Path) -> None:
wiki_dir = tmp_path / "wiki"
init_wiki(wiki_dir)
assert wiki_dir.exists()
calls = [c.args[0] for c in mock_run.call_args_list]
assert ["git", "init"] in calls
assert ["git", "config", "user.email", "ci@oblachno.fyi"] in calls
class TestSyncFiles:
def test_syncs_files(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n[link](page.md)\n")
(docs / "page.md").write_text("# Page\n")
wiki = tmp_path / "wiki"
wiki.mkdir()
mapping = {"index.md": "Home", "page.md": "Page"}
synced, pruned = sync_files(docs, wiki, mapping, dry_run=False)
assert synced == 2
assert pruned == 0
assert (wiki / "Home.md").exists()
assert (wiki / "Page.md").exists()
# Check link transformation
content = (wiki / "Home.md").read_text()
assert "[link](page)" in content
def test_prunes_stale(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
wiki = tmp_path / "wiki"
wiki.mkdir()
(wiki / "OldPage.md").write_text("# Old\n")
(wiki / "Home.md").write_text("# Old Home\n")
mapping = {"index.md": "Home"}
synced, pruned = sync_files(docs, wiki, mapping, dry_run=False)
assert synced == 1
assert pruned == 1 # OldPage.md pruned, Home.md overwritten
assert not (wiki / "OldPage.md").exists()
assert (wiki / "Home.md").exists()
def test_dry_run_no_writes(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
wiki = tmp_path / "wiki"
wiki.mkdir()
mapping = {"index.md": "Home"}
synced, pruned = sync_files(docs, wiki, mapping, dry_run=True)
assert synced == 1
assert pruned == 0
assert not (wiki / "Home.md").exists()
def test_missing_file_warns(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
wiki = tmp_path / "wiki"
wiki.mkdir()
mapping = {"missing.md": "Missing"}
synced, pruned = sync_files(docs, wiki, mapping, dry_run=False)
assert synced == 0
def test_empty_file_warns(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "empty.md").write_text("")
wiki = tmp_path / "wiki"
wiki.mkdir()
mapping = {"empty.md": "Empty"}
synced, pruned = sync_files(docs, wiki, mapping, dry_run=False)
assert synced == 0
class TestCommitAndPush:
@patch("devx.ci.sync_wiki.subprocess.run")
def test_dry_run_returns_false(self, mock_run: MagicMock, tmp_path: Path) -> None:
result = commit_and_push(tmp_path, "url", dry_run=True)
assert result is False
mock_run.assert_not_called()
@patch("devx.ci.sync_wiki.subprocess.run")
def test_no_changes_returns_false(self, mock_run: MagicMock, tmp_path: Path) -> None:
# git add succeeds, git diff --cached --quiet returns 0 (no changes)
mock_run.side_effect = [
MagicMock(returncode=0), # git add
MagicMock(returncode=0), # git diff --cached --quiet (no changes)
]
result = list_wiki_pages(client)
assert result == {"Home": "Home", "Getting-Started": "Getting-Started.-"}
result = commit_and_push(tmp_path, "url", dry_run=False)
assert result is False
@patch("devx.ci.sync_wiki.subprocess.run")
def test_pushes_changes(self, mock_run: MagicMock, tmp_path: Path) -> None:
mock_run.side_effect = [
MagicMock(returncode=0), # git add
MagicMock(returncode=1), # git diff --cached --quiet (has changes)
MagicMock(returncode=0), # git commit
MagicMock(returncode=0, stdout="", stderr=""), # git push
]
result = commit_and_push(tmp_path, "url", dry_run=False)
assert result is True
class TestFetchPageContent:
def test_fetches_and_decodes_content(self) -> None:
client = MagicMock()
encoded = base64.b64encode(b"# Hello Wiki").decode("ascii")
client._request.return_value.json.return_value = {"content_base64": encoded}
result = fetch_page_content(client, "Home")
assert result == "# Hello Wiki"
def test_returns_empty_on_api_error(self) -> None:
from devx.exceptions import APIError
client = MagicMock()
client._request.side_effect = APIError(404, "not found")
assert fetch_page_content(client, "Missing") == ""
def test_returns_empty_for_empty_content(self) -> None:
client = MagicMock()
client._request.return_value.json.return_value = {"content_base64": ""}
assert fetch_page_content(client, "Home") == ""
class TestSyncPage:
def test_dry_run_skips(self) -> None:
client = MagicMock()
result = sync_page(client, "Test-Page", "# Content", {}, dry_run=True)
assert result == "skipped"
client._request.assert_not_called()
def test_creates_new_page_with_base64(self) -> None:
client = MagicMock()
result = sync_page(client, "New-Page", "# Content", {}, dry_run=False)
assert result == "created"
client._request.assert_called_once()
call_args = client._request.call_args
assert call_args.args[0] == "POST"
assert call_args.args[1] == "/wiki/new"
# Verify content_base64 is used, not content
payload = call_args.kwargs["json"]
assert "content_base64" in payload
assert "content" not in payload
assert base64.b64decode(payload["content_base64"]).decode("utf-8") == "# Content"
def test_updates_existing_page_with_base64(self) -> None:
client = MagicMock()
existing = {"Existing-Page": "Existing-Page.-"}
result = sync_page(client, "Existing-Page", "# Updated", existing, dry_run=False)
assert result == "updated"
client._request.assert_called_once()
call_args = client._request.call_args
assert call_args.args[0] == "PATCH"
assert "/wiki/page/Existing-Page.-" in call_args.args[1]
# Verify content_base64 is used
payload = call_args.kwargs["json"]
assert "content_base64" in payload
assert "content" not in payload
assert base64.b64decode(payload["content_base64"]).decode("utf-8") == "# Updated"
def test_create_falls_back_to_update_on_already_exists(self) -> None:
"""When create fails with 400 'already exists', re-list and update."""
client = MagicMock()
# First call: POST /wiki/new → 400 already exists
# Second call: PATCH /wiki/page/{sub_url} → success
create_error = APIError(400, "wiki page already exists [title: Test-Page]")
client._request.side_effect = [create_error, MagicMock()]
with patch("devx.ci.sync_wiki._list_wiki_pages_with_retry", return_value={"Test-Page": "Test-Page.-"}):
result = sync_page(client, "Test-Page", "# Content", {}, dry_run=False)
assert result == "updated"
# Verify PATCH was called (second call)
patch_call = client._request.call_args_list[1]
assert patch_call.args[0] == "PATCH"
assert "/wiki/page/Test-Page.-" in patch_call.args[1]
def test_create_raises_non_400_error(self) -> None:
"""Non-400 errors from create should propagate, not trigger fallback."""
client = MagicMock()
client._request.side_effect = APIError(500, "server error")
with pytest.raises(APIError):
sync_page(client, "Test-Page", "# Content", {}, dry_run=False)
def test_create_raises_400_not_already_exists(self) -> None:
"""400 errors that don't mention 'already exists' should propagate."""
client = MagicMock()
client._request.side_effect = APIError(400, "invalid title")
with pytest.raises(APIError):
sync_page(client, "Test-Page", "# Content", {}, dry_run=False)
class TestVerifyWikiPage:
def test_verifies_matching_content(self) -> None:
client = MagicMock()
encoded = base64.b64encode(b"# Hello Wiki").decode("ascii")
client._request.return_value.json.return_value = {"content_base64": encoded}
existing = {"Home": "Home"}
assert verify_wiki_page(client, "Home", "# Hello Wiki", existing) is True
def test_fails_on_mismatch(self) -> None:
client = MagicMock()
encoded = base64.b64encode(b"# Old Content").decode("ascii")
client._request.return_value.json.return_value = {"content_base64": encoded}
existing = {"Home": "Home"}
assert verify_wiki_page(client, "Home", "# New Content", existing) is False
def test_fails_on_empty_wiki_content(self) -> None:
client = MagicMock()
client._request.return_value.json.return_value = {"content_base64": ""}
existing = {"Home": "Home"}
assert verify_wiki_page(client, "Home", "# Expected", existing) is False
def test_fails_when_page_not_in_existing(self) -> None:
client = MagicMock()
assert verify_wiki_page(client, "Missing", "# Content", {}) is False
class TestVerifyWikiIntegrity:
def _make_client(self, pages: dict[str, str], contents: dict[str, str]) -> MagicMock:
"""Create a mock client that returns the given pages and contents."""
client = MagicMock()
# list_wiki_pages calls GET /wiki/pages
page_list = [{"title": t, "sub_url": s} for t, s in pages.items()]
# fetch_page_content calls GET /wiki/page/{sub_url}
def mock_request(method, path, **kwargs):
resp = MagicMock()
if path == "/wiki/pages":
resp.json.return_value = page_list
elif path.startswith("/wiki/page/"):
sub_url = path.replace("/wiki/page/", "")
content = contents.get(sub_url, "")
encoded = base64.b64encode(content.encode()).decode("ascii") if content else ""
resp.json.return_value = {"content_base64": encoded}
return resp
client._request.side_effect = mock_request
return client
def test_all_good_no_failures(self) -> None:
pages = {"Home": "Home", "FAQ": "FAQ"}
contents = {"Home": "# Home", "FAQ": "# FAQ"}
client = self._make_client(pages, contents)
mapping = {"index.md": "Home", "faq.md": "FAQ"}
synced = {"Home": "# Home", "FAQ": "# FAQ"}
failures = verify_wiki_integrity(client, mapping, synced)
assert failures == []
def test_missing_page_detected(self) -> None:
pages = {"Home": "Home"} # FAQ missing from wiki
contents = {"Home": "# Home"}
client = self._make_client(pages, contents)
mapping = {"index.md": "Home", "faq.md": "FAQ"}
synced = {"Home": "# Home"}
failures = verify_wiki_integrity(client, mapping, synced)
assert any("Missing page: FAQ" in f for f in failures)
def test_stale_page_detected(self) -> None:
pages = {"Home": "Home", "Old-Page": "Old-Page"} # Old-Page not in mapping
contents = {"Home": "# Home", "Old-Page": "# Old"}
client = self._make_client(pages, contents)
mapping = {"index.md": "Home"}
synced = {"Home": "# Home"}
failures = verify_wiki_integrity(client, mapping, synced)
assert any("Stale page" in f and "Old-Page" in f for f in failures)
def test_page_count_mismatch_detected(self) -> None:
pages = {"Home": "Home", "Extra": "Extra"}
contents = {"Home": "# Home", "Extra": "# Extra"}
client = self._make_client(pages, contents)
mapping = {"index.md": "Home"}
synced = {"Home": "# Home"}
failures = verify_wiki_integrity(client, mapping, synced)
assert any("Page count mismatch" in f for f in failures)
def test_empty_content_detected(self) -> None:
pages = {"Home": "Home"}
contents = {"Home": ""} # Empty content
client = self._make_client(pages, contents)
mapping = {"index.md": "Home"}
synced = {"Home": "# Expected Content"}
failures = verify_wiki_integrity(client, mapping, synced)
assert any("Empty content: Home" in f for f in failures)
def test_content_mismatch_detected(self) -> None:
pages = {"Home": "Home"}
contents = {"Home": "# Wrong Content"}
client = self._make_client(pages, contents)
mapping = {"index.md": "Home"}
synced = {"Home": "# Correct Content"}
failures = verify_wiki_integrity(client, mapping, synced)
assert any("Content mismatch: Home" in f for f in failures)
def test_multiple_failures_all_reported(self) -> None:
pages = {"Home": "Home", "Stale": "Stale"}
contents = {"Home": "", "Stale": "# Stale"}
client = self._make_client(pages, contents)
mapping = {"index.md": "Home", "faq.md": "FAQ"} # FAQ missing
synced = {"Home": "# Home Content"}
failures = verify_wiki_integrity(client, mapping, synced)
assert len(failures) >= 3 # count mismatch, missing FAQ, stale Stale, empty Home
def test_transient_api_failure_returns_empty(self) -> None:
"""When the wiki API is unavailable after retries, integrity check
should return no failures (sync already succeeded)."""
client = MagicMock()
# _list_wiki_pages_with_retry raises APIError (retries exhausted)
with patch("devx.ci.sync_wiki._list_wiki_pages_with_retry", side_effect=APIError(0, "timeout")):
mapping = {"index.md": "Home", "faq.md": "FAQ"}
synced = {"Home": "# Home", "FAQ": "# FAQ"}
failures = verify_wiki_integrity(client, mapping, synced)
assert failures == []
def test_transient_api_failure_recovers_on_retry(self) -> None:
"""When the wiki API recovers after a retry, integrity check proceeds normally."""
client = MagicMock()
pages = {"Home": "Home", "FAQ": "FAQ"}
contents = {"Home": "# Home", "FAQ": "# FAQ"}
def mock_request(method, path, **kwargs):
resp = MagicMock()
if path == "/wiki/pages":
page_list = [{"title": t, "sub_url": s} for t, s in pages.items()]
resp.json.return_value = page_list
elif path.startswith("/wiki/page/"):
sub_url = path.replace("/wiki/page/", "")
content = contents.get(sub_url, "")
encoded = base64.b64encode(content.encode()).decode("ascii") if content else ""
resp.json.return_value = {"content_base64": encoded}
return resp
client._request.side_effect = mock_request
mapping = {"index.md": "Home", "faq.md": "FAQ"}
synced = {"Home": "# Home", "FAQ": "# FAQ"}
failures = verify_wiki_integrity(client, mapping, synced)
assert failures == []
@patch("devx.ci.sync_wiki.subprocess.run")
def test_push_failure_returns_false(self, mock_run: MagicMock, tmp_path: Path) -> None:
mock_run.side_effect = [
MagicMock(returncode=0), # git add
MagicMock(returncode=1), # git diff --cached --quiet (has changes)
MagicMock(returncode=0), # git commit
MagicMock(returncode=1, stdout="", stderr="push failed"), # git push
]
result = commit_and_push(tmp_path, "url", dry_run=False)
assert result is False
class TestMain:
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"})
@patch("devx.ci.sync_wiki.MAPPING_FILE")
@patch("devx.ci.sync_wiki.DOCS_DIR")
@patch("devx.ci.sync_wiki.GiteaClient")
def test_dry_run(self, mock_client_cls: MagicMock, mock_docs_dir: Path, mock_mapping_file: Path) -> None:
mock_mapping_file.exists.return_value = True
mock_mapping_file.__str__ = lambda _: "/docs/mapping.json"
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={}):
runner = CliRunner()
result = runner.invoke(main, ["--dry-run", "--repo", "owner/repo"])
assert result.exit_code == 0
assert "dry-run" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": ""}, clear=True)
def test_missing_token_exits(self) -> None:
def test_no_token_raises(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("CI_GITEA_TOKEN", raising=False)
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo"])
assert result.exit_code == 1
result = runner.invoke(main, [])
assert result.exit_code != 0
assert "CI_GITEA_TOKEN" in result.output
@patch.dict(
"os.environ", {"CI_GITEA_TOKEN": "tok", "DEVX_REPO_OWNER": "me", "DEVX_REPO_NAME": "myrepo"}, clear=True
)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_auto_detect_repo(self, mock_client_cls: MagicMock) -> None:
"""Test that repo is auto-detected from env vars when --repo is not passed."""
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={}):
runner = CliRunner()
result = runner.invoke(main, ["--dry-run"])
assert result.exit_code == 0
mock_client_cls.assert_called_once()
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_missing_mapping_file(self, mock_client_cls: MagicMock) -> None:
"""Test that missing mapping.json exits with error."""
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = False
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo"])
assert result.exit_code == 1
def test_no_mapping_raises(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", tmp_path / "nonexistent.json")
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo"])
assert result.exit_code != 0
assert "mapping.json" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_existing_pages_message(self, mock_client_cls: MagicMock) -> None:
"""Test that existing wiki pages are reported."""
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={"Home": "Home"}):
runner = CliRunner()
result = runner.invoke(main, ["--dry-run", "--repo", "owner/repo"])
@patch("devx.ci.sync_wiki.clone_wiki", return_value=True)
@patch("devx.ci.sync_wiki.commit_and_push", return_value=True)
@patch("devx.ci.sync_wiki.sync_files", return_value=(1, 0))
def test_dry_run(
self,
mock_sync: MagicMock,
mock_push: MagicMock,
mock_clone: MagicMock,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
mapping_file = docs / "mapping.json"
mapping_file.write_text(json.dumps({"index.md": "Home"}))
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
monkeypatch.setattr("devx.ci.sync_wiki.DOCS_DIR", docs)
runner = CliRunner()
result = runner.invoke(main, ["--dry-run", "--repo", "owner/repo"])
assert result.exit_code == 0
assert "existing wiki pages" in result.output
assert "dry-run" in result.output
mock_push.assert_not_called()
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_file_not_found_fails(self, mock_client_cls: MagicMock) -> None:
"""Test that missing doc files cause an error, not a warning."""
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"missing.md": "Missing"}):
with patch("devx.ci.sync_wiki.read_doc_content", side_effect=FileNotFoundError):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={}):
runner = CliRunner()
result = runner.invoke(main, ["--dry-run", "--repo", "owner/repo"])
@patch("devx.ci.sync_wiki.clone_wiki", return_value=True)
@patch("devx.ci.sync_wiki.commit_and_push", return_value=True)
@patch("devx.ci.sync_wiki.sync_files", return_value=(2, 0))
def test_full_sync(
self,
mock_sync: MagicMock,
mock_push: MagicMock,
mock_clone: MagicMock,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n[link](page.md)\n")
(docs / "page.md").write_text("# Page\n")
mapping_file = docs / "mapping.json"
mapping_file.write_text(json.dumps({"index.md": "Home", "page.md": "Page"}))
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
monkeypatch.setattr("devx.ci.sync_wiki.DOCS_DIR", docs)
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo"])
assert result.exit_code == 0
assert "Synced" in result.output
mock_push.assert_called_once()
@patch("devx.ci.sync_wiki.clone_wiki", return_value=False)
@patch("devx.ci.sync_wiki.init_wiki")
@patch("devx.ci.sync_wiki.commit_and_push", return_value=True)
@patch("devx.ci.sync_wiki.sync_files", return_value=(1, 0))
def test_init_fresh_wiki(
self,
mock_sync: MagicMock,
mock_push: MagicMock,
mock_init: MagicMock,
mock_clone: MagicMock,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
mapping_file = docs / "mapping.json"
mapping_file.write_text(json.dumps({"index.md": "Home"}))
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
monkeypatch.setattr("devx.ci.sync_wiki.DOCS_DIR", docs)
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo"])
assert result.exit_code == 0
mock_init.assert_called_once()
@patch("devx.ci.sync_wiki.clone_wiki")
@patch("devx.ci.sync_wiki.commit_and_push", return_value=True)
@patch("devx.ci.sync_wiki.sync_files", return_value=(1, 0))
def test_verify(
self,
mock_sync: MagicMock,
mock_push: MagicMock,
mock_clone: MagicMock,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
mapping_file = docs / "mapping.json"
mapping_file.write_text(json.dumps({"index.md": "Home"}))
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
monkeypatch.setattr("devx.ci.sync_wiki.DOCS_DIR", docs)
# Mock clone_wiki to create the wiki dir with the expected file
def fake_clone(url: str, dest: Path) -> bool:
dest.mkdir(parents=True, exist_ok=True)
(dest / "Home.md").write_text("# Home\n")
return True
mock_clone.side_effect = fake_clone
runner = CliRunner()
result = runner.invoke(main, ["--verify", "--repo", "owner/repo"])
assert result.exit_code == 0
assert "Verification" in result.output
@patch("devx.ci.sync_wiki.clone_wiki", return_value=True)
@patch("devx.ci.sync_wiki.commit_and_push", return_value=False)
@patch("devx.ci.sync_wiki.sync_files", return_value=(1, 0))
def test_push_failed_message(
self,
mock_sync: MagicMock,
mock_push: MagicMock,
mock_clone: MagicMock,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
mapping_file = docs / "mapping.json"
mapping_file.write_text(json.dumps({"index.md": "Home"}))
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
monkeypatch.setattr("devx.ci.sync_wiki.DOCS_DIR", docs)
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo"])
assert result.exit_code == 0
assert "No push needed" in result.output
@patch("devx.ci.sync_wiki.clone_wiki", side_effect=[True, False])
@patch("devx.ci.sync_wiki.commit_and_push", return_value=True)
@patch("devx.ci.sync_wiki.sync_files", return_value=(1, 0))
def test_verify_clone_fails(
self,
mock_sync: MagicMock,
mock_push: MagicMock,
mock_clone: MagicMock,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
mapping_file = docs / "mapping.json"
mapping_file.write_text(json.dumps({"index.md": "Home"}))
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
monkeypatch.setattr("devx.ci.sync_wiki.DOCS_DIR", docs)
runner = CliRunner()
result = runner.invoke(main, ["--verify", "--repo", "owner/repo"])
assert result.exit_code != 0
assert "not found" in result.output
assert "could not clone" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_empty_doc_file_fails(self, mock_client_cls: MagicMock) -> None:
"""Test that empty doc files cause an error, not a warning."""
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"empty.md": "Empty-Page"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value=" \n "):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={}):
runner = CliRunner()
result = runner.invoke(main, ["--dry-run", "--repo", "owner/repo"])
@patch("devx.ci.sync_wiki.clone_wiki")
@patch("devx.ci.sync_wiki.commit_and_push", return_value=True)
@patch("devx.ci.sync_wiki.sync_files", return_value=(1, 0))
def test_verify_missing_page(
self,
mock_sync: MagicMock,
mock_push: MagicMock,
mock_clone: MagicMock,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
mapping_file = docs / "mapping.json"
mapping_file.write_text(json.dumps({"index.md": "Home"}))
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
monkeypatch.setattr("devx.ci.sync_wiki.DOCS_DIR", docs)
# Mock clone_wiki to create the wiki dir WITHOUT the expected file
def fake_clone(url: str, dest: Path) -> bool:
dest.mkdir(parents=True, exist_ok=True)
return True
mock_clone.side_effect = fake_clone
runner = CliRunner()
result = runner.invoke(main, ["--verify", "--repo", "owner/repo"])
assert result.exit_code != 0
assert "empty" in result.output.lower()
assert "page(s) missing" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_create_and_update(self, mock_client_cls: MagicMock) -> None:
"""Test that pages are created and updated correctly (non-dry-run)."""
mock_client = MagicMock()
mock_client_cls.return_value = mock_client
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
mapping = {"new.md": "New-Page", "existing.md": "Existing-Page"}
with patch("devx.ci.sync_wiki.load_mapping", return_value=mapping):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Content"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={"Existing-Page": "Existing-Page"}):
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo"])
@patch("devx.ci.sync_wiki.clone_wiki", return_value=True)
@patch("devx.ci.sync_wiki.commit_and_push", return_value=True)
@patch("devx.ci.sync_wiki.sync_files", return_value=(1, 0))
def test_auto_detect_repo(
self,
mock_sync: MagicMock,
mock_push: MagicMock,
mock_clone: MagicMock,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
mapping_file = docs / "mapping.json"
mapping_file.write_text(json.dumps({"index.md": "Home"}))
monkeypatch.setenv("CI_GITEA_TOKEN", "fake")
monkeypatch.setattr("devx.ci.sync_wiki.MAPPING_FILE", mapping_file)
monkeypatch.setattr("devx.ci.sync_wiki.DOCS_DIR", docs)
runner = CliRunner()
result = runner.invoke(main, [])
assert result.exit_code == 0
assert "Created: 1" in result.output
assert "Updated: 1" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_verify_passes(self, mock_client_cls: MagicMock) -> None:
"""Test that --verify passes when content matches."""
mock_client = MagicMock()
mock_client_cls.return_value = mock_client
encoded = base64.b64encode(b"# Home Content").decode("ascii")
# list_wiki_pages returns {"Home": "Home"}, fetch returns encoded content
mock_client._request.return_value.json.return_value = {"content_base64": encoded}
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home Content"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={"Home": "Home"}):
with patch("devx.ci.sync_wiki.verify_wiki_page", return_value=True):
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo", "--verify"])
assert result.exit_code == 0
assert "Verification passed" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_verify_fails_on_empty_content(self, mock_client_cls: MagicMock) -> None:
"""Test that --verify fails when wiki pages have empty content."""
mock_client = MagicMock()
mock_client_cls.return_value = mock_client
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home Content"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={"Home": "Home"}):
with patch("devx.ci.sync_wiki.verify_wiki_page", return_value=False):
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo", "--verify"])
assert result.exit_code == 1
assert "FAIL" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_verify_skipped_in_dry_run(self, mock_client_cls: MagicMock) -> None:
"""Test that --verify is skipped during dry-run."""
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={}):
runner = CliRunner()
result = runner.invoke(main, ["--dry-run", "--verify", "--repo", "owner/repo"])
assert result.exit_code == 0
assert "Verification" not in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_strict_passes(self, mock_client_cls: MagicMock) -> None:
"""Test that --strict passes when integrity check succeeds."""
mock_client = MagicMock()
mock_client_cls.return_value = mock_client
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={"Home": "Home"}):
with patch("devx.ci.sync_wiki.verify_wiki_integrity", return_value=[]):
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo", "--strict"])
assert result.exit_code == 0
assert "Integrity check passed" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_strict_fails_on_integrity_issues(self, mock_client_cls: MagicMock) -> None:
"""Test that --strict fails when integrity check finds issues."""
mock_client = MagicMock()
mock_client_cls.return_value = mock_client
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={"Home": "Home"}):
with patch(
"devx.ci.sync_wiki.verify_wiki_integrity",
return_value=["Missing page: FAQ", "Stale page: Old-Page"],
):
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo", "--strict"])
assert result.exit_code == 1
assert "Integrity check FAILED" in result.output
assert "Missing page: FAQ" in result.output
assert "Stale page: Old-Page" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_strict_skipped_in_dry_run(self, mock_client_cls: MagicMock) -> None:
"""Test that --strict verification is skipped during dry-run."""
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki.list_wiki_pages", return_value={}):
runner = CliRunner()
result = runner.invoke(main, ["--dry-run", "--strict", "--repo", "owner/repo"])
assert result.exit_code == 0
assert "Integrity check" not in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_initial_list_api_error_aborts(self, mock_client_cls: MagicMock) -> None:
"""When the initial page list fails after retries, sync aborts to avoid duplicate pages."""
mock_client = MagicMock()
mock_client_cls.return_value = mock_client
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki._list_wiki_pages_with_retry", side_effect=APIError(0, "timeout")):
with patch("devx.ci.sync_wiki.sync_page", return_value="created"):
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo"])
assert result.exit_code != 0
assert "Failed to list existing wiki pages" in result.output
assert "Aborting" in result.output
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True)
@patch("devx.ci.sync_wiki.GiteaClient")
def test_verify_skips_when_refetch_fails(self, mock_client_cls: MagicMock) -> None:
"""When --verify re-fetch fails after retries, verification is skipped gracefully."""
mock_client = MagicMock()
mock_client_cls.return_value = mock_client
# Initial list succeeds, but verify re-fetch fails
list_side_effect = [{"Home": "Home"}, APIError(0, "timeout")]
with patch("devx.ci.sync_wiki.MAPPING_FILE") as mock_mapping:
mock_mapping.exists.return_value = True
with patch("devx.ci.sync_wiki.load_mapping", return_value={"index.md": "Home"}):
with patch("devx.ci.sync_wiki.read_doc_content", return_value="# Home"):
with patch("devx.ci.sync_wiki._list_wiki_pages_with_retry", side_effect=list_side_effect):
with patch("devx.ci.sync_wiki.sync_page", return_value="updated"):
runner = CliRunner()
result = runner.invoke(main, ["--repo", "owner/repo", "--verify"])
assert result.exit_code == 0
assert "Skipping content verification" in result.output