# Practical Mutation Testing at Scale

arXiv: https://arxiv.org/abs/2102.11378 · HTML: https://mlsanity.com/paper/2102.11378/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2021.

## Built on by (most cited first, 3 of 3)

- [Mutation-Guided LLM-based Test Generation at Meta](https://mlsanity.com/paper/2501.12862/) (2025)
- [Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs](https://mlsanity.com/paper/2401.05940/) (2024)
- [Quality-Driven Selective Mutation for Deep Learning](https://mlsanity.com/paper/2604.22640/) (2026)

## Abstract (arXiv)

Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem -- both in terms of human and computational effort. This has hindered the adoption of mutation testing as an industry standard. For example, Google has a codebase of two billion lines of code and more than 500,000,000 tests are executed on a daily basis. The traditional approach to mutation testing does not scale to such an environment. To address these challenges, this paper presents a scalable approach to mutation testing based on the following main ideas: (1) Mutation testing is done incrementally, mutating only changed code during code review, rather than the entire code base; (2) Mutants are filtered, removing mutants that are likely to be irrelevant to developers, and limiting the number of mutants per line and per code review process; (3) Mutants are selected based on the historical performance of mutation operators, further eliminating irrelevant mutants and improving mutant quality. Evaluation in a code-review-based setting with more than 24,000 developers on more than 1,000 projects shows that the proposed approach produces orders of magnitude fewer mutants and that context-based mutant filtering and selection improve mutant quality and actionability. Overall, the proposed approach represents a mutation testing framework that seamlessly integrates into the software development workflow and is applicable up to large-scale industrial settings.