# Keeping Mutation Test Suites Consistent and Relevant with Long-Standing Mutants

arXiv: https://arxiv.org/abs/2212.11762 · HTML: https://mlsanity.com/paper/2212.11762/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2022.

## Built on by (most cited first, 1 of 1)

- [Mutation-Guided LLM-based Test Generation at Meta](https://mlsanity.com/paper/2501.12862/) (2025)

## Abstract (arXiv)

Mutation testing has been demonstrated to be one of the most powerful fault-revealing tools in the tester's tool kit. Much previous work implicitly assumed it to be sufficient to re-compute mutant suites per release. Sadly, this makes mutation results inconsistent; mutant scores from each release cannot be directly compared, making it harder to measure test improvement. Furthermore, regular code change means that a mutant suite's relevance will naturally degrade over time. We measure this degradation in relevance for 143,500 mutants in 4 non-trivial systems finding that, on overage, 52% degrade. We introduce a mutant brittleness measure and use it to audit software systems and their mutation suites. We also demonstrate how consistent-by-construction long-standing mutant suites can be identified with a 10x improvement in mutant relevance over an arbitrary test suite. Our results indicate that the research community should avoid the re-computation of mutant suites and focus, instead, on long-standing mutants, thereby improving the consistency and relevance of mutation testing.