Luigit
repositories / dotfiles

dotfiles

bugabingas dorkfiles

owned by admin

pi/agent/skills/testing/references/mutation-testing.md

Raw
Rendered preview

Mutation testing

Also: mutation analysis. Mutants are changed program variants; mutation fuzzing is different.

Coverage asks “did it run?” Mutation asks “did selected tests detect selected changes?” Neither establishes overall correctness.

Workflow

  1. Establish a passing relevant baseline and a protection question.
  2. Scope production files/functions/lines/mutants using supported selectors.
  3. Select relevant tests through existing per-test coverage mapping or small test groups. Mutant scope ≠ test selection; omitting a legitimate detecting test can mislead.
  4. Reuse valid caches/history.
  5. Inspect useful survivors, improve behavioral protection, rerun affected scope. Improve high-level tests when appropriate, not one test per mutant.

Survivor ≠ bug. Distinguish surviving, uncovered, equivalent, invalid, timed-out, and infrastructure outcomes using runner definitions. Do not encode mutation operators as requirements or chase a perfect score.

Cache discipline

  • Check tracked inputs: goldens/snapshots, fixtures, dependencies, environment, config, helpers, and test deletions may invalidate history.
  • Relevant untracked input changed → force fresh affected-scope analysis through supported automation. Unchanged production source alone does not justify reusing a kill.
  • Report fresh | reused | partial | stale evidence and selected scope. Combined reports may include historical results outside the new run.
  • No supported optimization → bounded run or defer with cost/limits disclosed. No automatic infrastructure repair, new gates, broad campaign, erased survivors, or weakened thresholds/operators.

Sources

# Mutation testing

Also: mutation analysis.
Mutants are changed program variants; mutation fuzzing is different.

Coverage asks “did it run?”
Mutation asks “did selected tests detect selected changes?”
Neither establishes overall correctness.

## Workflow

1. Establish a passing relevant baseline and a protection question.
2. Scope production files/functions/lines/mutants using supported selectors.
3. Select relevant tests through existing per-test coverage mapping or small test groups.
   Mutant scope ≠ test selection; omitting a legitimate detecting test can mislead.
4. Reuse valid caches/history.
5. Inspect useful survivors, improve behavioral protection, rerun affected scope.
   Improve high-level tests when appropriate, not one test per mutant.

Survivor ≠ bug.
Distinguish surviving, uncovered, equivalent, invalid, timed-out, and infrastructure outcomes using runner definitions.
Do not encode mutation operators as requirements or chase a perfect score.

## Cache discipline

- Check tracked inputs: goldens/snapshots, fixtures, dependencies, environment, config, helpers, and test deletions may invalidate history.
- Relevant untracked input changed → force fresh affected-scope analysis through supported automation.
  Unchanged production source alone does not justify reusing a kill.
- Report fresh | reused | partial | stale evidence and selected scope.
  Combined reports may include historical results outside the new run.
- No supported optimization → bounded run or defer with cost/limits disclosed.
  No automatic infrastructure repair, new gates, broad campaign, erased survivors, or weakened thresholds/operators.

## Sources

- [Stryker: incremental runs and invalidation limits](https://stryker-mutator.io/docs/stryker-js/incremental/)
- [Stryker: coverage-based test selection](https://stryker-mutator.io/docs/stryker-js/configuration/#coverageanalysis-string)
- [PIT: history and dependency-tracking limits](https://pitest.org/quickstart/incremental_analysis/)