# Will My Tests Tell Me If I Break This Code?

arXiv: https://arxiv.org/abs/1611.07163 · HTML: https://mlsanity.com/paper/1611.07163/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2016.

## Built on by (most cited first, 2 of 2)

- [Reinforcement Learning Integrated Agentic RAG for Software Test Cases Authoring](https://mlsanity.com/paper/2512.06060/) (2025)
- [Hard-Gate Candidacy in a Deployed Validator Suite](https://mlsanity.com/paper/2609.39037/) (2026)

## Abstract (arXiv)

Automated tests play an important role in software evolution because they can rapidly detect faults introduced during changes. In practice, code-coverage metrics are often used as criteria to evaluate the effectiveness of test suites with focus on regression faults. However, code coverage only expresses which portion of a system has been executed by tests, but not how effective the tests actually are in detecting regression faults. Our goal was to evaluate the validity of code coverage as a measure for test effectiveness. To do so, we conducted an empirical study in which we applied an extreme mutation testing approach to analyze the tests of open-source projects written in Java. We assessed the ratio of pseudo-tested methods (those tested in a way such that faults would not be detected) to all covered methods and judged their impact on the software project. The results show that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system). Therefore, we conclude that the coverage metric is only a valid effectiveness indicator for unit tests.