Skip to content

fix(mutmut): make scripts/ test-loader module names match mutmut's path-derived keys - #264

Merged
cdeust merged 3 commits into
mainfrom
fix-mutation-scripts-attribution-262
Jul 30, 2026
Merged

fix(mutmut): make scripts/ test-loader module names match mutmut's path-derived keys#264
cdeust merged 3 commits into
mainfrom
fix-mutation-scripts-attribution-262

Conversation

@cdeust

@cdeust cdeust commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes #262: mutation testing could not attribute mutants for any
scripts/ module because tests_py/scripts/*.py tests loaded their
subjects under synthetic or bare module names
(_cortex_launcher_deps, generate_pip_constraints, bare
import wiki_backfill_ids, ...), while mutmut keys mutant trampolines
on the path-derived dotted name (scripts.launcher_deps,
scripts.generate_pip_constraints, ...) it computes purely from each
mutated file's path relative to the repo root. Every scoped run aborted
early: "Stopping early... tests recorded trampoline hits but none match
any mutant key."

Root cause traced by reading mutmut's own source
(mutmut/mutation/trampoline.py's record_trampoline_hit(f"{orig_func.__module__}...")
vs mutmut/utils/format_utils.py::get_mutant_name, which derives the key
from the file path alone). __module__ is bound to whatever name
importlib.util.spec_from_file_location/module_from_spec gave the
loaded module — a test-only artifact completely disconnected from the
path-derived key mutmut expects.

Precedent already existed in this repo: PR #236 fixed the identical
defect for test_check_doc_claims.py alone and filed the survivor
backlog it surfaced as #235, rather than folding a much larger
test-writing effort into the attribution-fix PR. This PR applies the
same fix to every other affected file (test_check_doc_claims.py
excluded — already fixed, and owned by the concurrent #235 worktree),
and follows the same split for the backlog it surfaces: filed as #263.

Changes (3 commits)

  1. fix(mutmut) — rename every affected test's module-loader handle
    to the dotted scripts.<file> name mutmut expects (14 files); widen
    [tool.mutmut] also_copy (uv.lock, requirements/, pyproject.toml)
    so generate_pip_constraints.py's test — which reads all three
    directly — mutates cleanly instead of every mutant raising
    FileNotFoundError.
  2. test — new regression test
    (test_launcher_deps_stays_bare_importable_by_launcher_py) proving
    AC 2: launcher_deps.py stays loadable by a bare import at real
    runtime (a genuine python3 scripts/launcher.py subprocess, not a
    test-loader path). Verified it actually catches a regression
    (temporarily broke the import, watched it fail, reverted).
  3. docs — resync the advertised test count across README.md,
    CONTRIBUTING.md, CLAUDE.md, docs/ASSURANCE-CASE.md,
    .bestpractices.json, and the regenerated assets/badge-tests.svg.
    Rebased onto main after it independently merged json_native.py: 14 surviving mutants under the repository's own committed mutmut scope #250/pyright gate resolves a different tree-sitter-language-pack than the documented dev install #253/normalize_date_to_iso silently re-anchors an unresolvable timezone abbreviation to UTC #252
    (baseline moved 6439 -> 6549 while this branch was in flight);
    resolved the six suite-size files' conflicts on their merits —
    main's structure, count recomputed from a live collection on the
    rebased tree (6550 = main's 6549 + this PR's 1 new test), not a
    delta applied to either side's stale number.

Verification (acceptance criteria from #262)

  1. Real mutant tallies now produced for both AC-named modules (previously
    an early abort with zero usable result):
    scripts/mutation_check.sh tests_py/scripts/test_launcher_deps.py scripts/launcher_deps.py
      -> 213 mutants, 128 killed, 85 survived, 0 no-tests
    
    scripts/mutation_check.sh tests_py/scripts/test_generate_pip_constraints.py scripts/generate_pip_constraints.py
      -> 243 mutants, 95 killed, 65 survived, 83 no-tests
    
  2. launcher_deps.py bare-importability asserted, not assumed (new
    regression test above).
  3. also_copy widened with uv.lock/requirements//pyproject.toml.
  4. Survivors triaged per §12.4: filed as Mutation-test survivors in scripts/launcher_deps.py and scripts/generate_pip_constraints.py (85 + 65), now that #262 makes them measurable #263 (85 + 65, matching the
    #236 -> #235 precedent) — this PR fixes the attribution seam, not
    the pre-existing coverage gaps it now makes measurable for the first
    time.

Local, on the rebased tree (head 16d296c), quoted for reviewer sanity:
pytest tests_py/ -> 6550 passed, 117 subtests passed in 161s.
pytest tests_py/scripts/ -> 301 passed, 117 subtests passed.
ruff check + ruff format --check on every touched .py file: clean.
scripts/check_doc_claims.py --test-count 6550 -> OK.
scripts/generate_repo_badges.py --check --test-count 6550 -> OK (5
checked). scripts/check_marketplace_pins.py -> OK.

Completion Ledger

Path introduced by this diff Asserting test / evidence
Renamed loader in 14 tests_py/scripts/*.py files tests_py/scripts/ full run: 301 passed before AND after the rename commit (0 test-code behavior change)
Widened [tool.mutmut] also_copy generate_pip_constraints.py scoped mutation run produces a real 95/65/83 tally instead of FileNotFoundError aborts
New test_launcher_deps_stays_bare_importable_by_launcher_py Self-verifying: shown to fail against a deliberately broken launcher.py import, then pass against the real one
Doc/badge count resync (post-rebase: 6550) check_doc_claims.py --test-count 6550 + generate_repo_badges.py --check --test-count 6550 both green; full suite collects exactly 6550

Boy-scout (§14)

  • ruff format flagged one line-length issue introduced by my own edit in
    test_setup_py_backend_skip.py — fixed in the same commit (formatter
    auto-fix, verified ruff format --check clean afterward).
  • Rebase reconciliation (this session): resolved 6 conflicting doc/badge
    files by recomputing the count from a live pytest --collect-only -q
    on the rebased tree rather than taking either side's stale number, and
    corrected the reconciliation commit's own message/date to describe
    what actually happened (was previously drafted against the pre-rebase
    6440 number before the conflict was known).
  • No other defects seen in touched material this session.

Deferred (§14.3 — outside this PR's blast radius, filed with acceptance criteria)

🤖 Generated with Claude Code

cdeust and others added 3 commits July 30, 2026 01:21
…th-derived keys

Rename (Fowler, Rename Function/Variable) every tests_py/scripts/*.py
module handle that loaded its scripts/ subject under a synthetic or bare
name (e.g. "_cortex_launcher_deps", "generate_pip_constraints", bare
`import wiki_backfill_ids`) to the dotted path mutmut derives from the
file's location ("scripts.launcher_deps", "scripts.generate_pip_constraints",
"scripts.wiki_backfill_ids", ...).

Root cause (read from mutmut's own source, mutmut/mutation/trampoline.py +
__main__.py::_check_test_to_mutant_associations): the trampoline records
`orig_func.__module__` at test-run time, which is whatever name
spec_from_file_location/module_from_spec gave the loaded module. Mutant
keys are computed independently, purely from the mutated file's path
relative to repo root ("scripts/launcher_deps.py" -> "scripts.launcher_deps").
A synthetic test-loader name never matches, so mutmut aborted every scoped
run before scoring anything ("Stopping early... none match any mutant key").

Excludes tests_py/scripts/test_check_doc_claims.py, already fixed by #236
(precedent for this exact fix) and owned by the concurrent #235 worktree.

Also widens [tool.mutmut] also_copy (uv.lock, requirements/, pyproject.toml)
so scripts/generate_pip_constraints.py's test — which reads all three
directly — mutates cleanly instead of every mutant raising
FileNotFoundError inside mutants/ (both changes are needed together for a
trustworthy tally on that module).

Verified: real mutant tallies now produced for both AC-named modules
(previously an early abort with zero usable result):
  scripts/launcher_deps.py:            213 mutants, 128 killed, 85 survived
  scripts/generate_pip_constraints.py: 243 mutants,  95 killed, 65 survived, 83 no-tests

Survivor backlog filed as #263 (same split precedent as #236 -> #235):
this PR fixes the attribution SEAM, not the pre-existing coverage gaps it
now makes visible.

tests_py/scripts/: 291 passed before and after (0 test-code behavior
change — only the module handle string changed).

Closes #262 (attribution mechanism fix; see #263 for the surfaced backlog)

Co-Authored-By: Claude <noreply@anthropic.com>
Issue #262 AC 2: "launcher_deps.py remains loadable by path at runtime
(the launcher's pre-dependency bootstrap must not regress) — assert it,
don't assume it." The prior commit renamed this file's OWN test-loader
handle from a synthetic name to "scripts.launcher_deps" (so mutmut can
attribute mutants); that rename does not touch launcher_deps.py or
launcher.py themselves, but the assertion was still missing.

test_launcher_deps_stays_bare_importable_by_launcher_py spawns a real
`python3 scripts/launcher.py` subprocess (no pytest sys.path, no test
loader involved) and asserts it reaches the plain "Usage: ..." message
rather than a traceback — proving launcher.py's own bare
`import launcher_deps` (module-level, executed before main() runs) still
resolves.

Verified the test actually catches a regression: temporarily renamed
launcher.py's import target to a nonexistent module, watched this test
fail with the ModuleNotFoundError surfaced in stderr, then reverted.

tests_py/scripts/test_launcher_deps.py: 33 passed (32 before + this one).

Co-Authored-By: Claude <noreply@anthropic.com>
test_launcher_deps_stays_bare_importable_by_launcher_py (two commits
prior) adds one collected test on top of main's count. Rebased onto
main (which had independently merged #250/#253/#252 in the interim,
advancing the baseline from 6439 to 6549); resolved the six suite-size
files' conflicts on their merits (structure from main, count
recomputed) rather than taking either side's stale number, per
coding-standards.md's rule that the advertised count is a measured
absolute, not a delta arithmetic on a side's claim.

Verified against a live collection on the rebased tree:
  pytest --collect-only -q                                    -> 6550 tests collected
  python scripts/check_doc_claims.py --test-count 6550         -> doc claims OK
  python scripts/generate_repo_badges.py --check --test-count 6550 -> badges OK (5 checked)

Co-Authored-By: Claude <noreply@anthropic.com>
@cdeust
cdeust force-pushed the fix-mutation-scripts-attribution-262 branch from 7b926e0 to 16d296c Compare July 29, 2026 23:26
@cdeust
cdeust merged commit 7d2842d into main Jul 30, 2026
19 checks passed
@cdeust
cdeust deleted the fix-mutation-scripts-attribution-262 branch July 30, 2026 00:52
cdeust added a commit that referenced this pull request Jul 30, 2026
…nt_sets.py mutation-testable (#262) (#280)

* fix(mutmut): dot the last scripts/ test-loader gap and make pip_constraint_sets.py mutation-testable (#262)

test_typecheck_env_parity.py loaded scripts/pip_constraint_sets.py under
the bare name "pip_constraint_sets" instead of the path-derived
"scripts.pip_constraint_sets" mutmut keys mutant trampolines on — #264
fixed this for 14 sibling scripts/ test files but missed this one, so a
scoped mutation run on it aborted with "Stopping early ... none match
any mutant key" (found by code-reviewer post-merge on #264).

Dotting the loader alone was not sufficient: ConstraintSet's three
methods (path/command/header_lines) carried its entire logic, and
mutmut's mutation generator categorically excludes the body of any
@dataclass-decorated class (confirmed empirically: 0 mutants for the
whole file with the methods in place, vs 58 after extracting them to
free functions constraint_path/constraint_command/constraint_header_lines
with ConstraintSet left as pure data). Every call site across
generate_pip_constraints.py and its two test files is updated to match;
the existing 301-test tests_py/scripts/ suite passes unchanged, proving
the refactor is behavior-preserving.

A dedicated tests_py/scripts/test_pip_constraint_sets.py directly
exercises the three extracted functions (neither existing test file
called them for real: everything went through compose()/render(),
which every test either mocks or short-circuits before reaching). The
scoped mutation run (tests_py/scripts/test_typecheck_env_parity.py +
test_pip_constraint_sets.py against scripts/pip_constraint_sets.py) now
produces real attribution: 58 mutants, 58 killed, 0 survived, 0 no-tests.

Boy-scout finds in the same sweep (grepped every spec_from_file_location
and sys.path-plus-bare-import in tests_py/, 21 call sites verified):
- tests_py/scripts/test_generate_repo_badges.py bare-imported
  scripts/badge_render.py (`import badge_render` after a sys.path
  insert) instead of loading it under "scripts.badge_render" — same
  defect class, now fixed with its own dotted load, independent of
  generate_repo_badges.py's own bare import of the same file.
- Two ruff E501 violations introduced by this diff's own line-length
  growth, fixed in the same commit (§14).
- generate_repo_badges.py's RepoBadge class has the identical
  @dataclass-exclusion gap (confirmed: 234 mutants total, 0 attributed
  to RepoBadge.path()/.render()) — out of this PR's blast radius (a
  separate root-cause fix of comparable size; mixing it in here would
  violate §15.1's "don't mix the enabling refactor with the fix");
  reported with evidence rather than filed, per this task's no-new-
  issues instruction.

Resynced the 6 suite-count sites + badge SVG to the measured absolute
(6560, +10 for test_pip_constraint_sets.py) via generate_repo_badges.py
and manual doc edits; both check_doc_claims.py --test-count 6560 and
generate_repo_badges.py --check --test-count 6560 pass. Full suite:
6560 passed, 117 subtests passed (167.83s).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u

* fix(mutmut): add .github to [tool.mutmut] also_copy for scripts/pip_constraint_sets.py scoped runs

tests_py/scripts/test_typecheck_env_parity.py reads .github/workflows/ci.yml
directly (cross-checking CONTRIBUTING.md's documented type-check command
against it). Running scripts/mutation_check.sh with this test in the
selection failed collection under every mutant with FileNotFoundError
(mutants/.github/workflows/ci.yml missing) instead of scoring the suite --
found while producing the real mutant tally for the previous commit's
pip_constraint_sets.py fix. .github wasn't in also_copy for any prior
scoped run (json_native.py, launcher_deps.py, generate_pip_constraints.py
all read files already listed), so this is additive and does not change
behavior for any other scoped run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mutation testing cannot attribute mutants for any scripts/ module (synthetic module names in tests_py/scripts)

1 participant