# Mutation-Guided Unit Test Generation with a Large Language Model

arXiv: https://arxiv.org/abs/2506.02954 · HTML: https://mlsanity.com/paper/2506.02954/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2025.

## Built on by (most cited first, 7 of 7)

- [SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?](https://mlsanity.com/paper/2605.22175/) (2026)
- [RM -RF: Reward Model for Run-Free Unit Test Evaluation](https://mlsanity.com/paper/2601.13097/) (2026)
- [PBT-Bench: Benchmarking AI Agents on Property-Based Testing](https://mlsanity.com/paper/2605.15229/) (2026)
- [FASE: Fast Adaptive Semantic Entropy for Code Quality](https://mlsanity.com/paper/2606.09800/) (2026)
- [SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generation](https://mlsanity.com/paper/2607.08983/) (2026)
- [Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility](https://mlsanity.com/paper/2605.06856/) (2026)
- [Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models](https://mlsanity.com/paper/2609.35841/) (2026)

## Abstract (arXiv)

Unit tests play a vital role in uncovering potential faults in software. While tools like EvoSuite focus on maximizing code coverage, recent advances in large language models (LLMs) have shifted attention toward LLM-based test generation. However, code coverage metrics -- such as line and branch coverage -- remain overly emphasized in reported research, despite being weak indicators of a test suite's fault-detection capability. In contrast, mutation score offers a more reliable and stringent measure, as demonstrated in our findings where some test suites achieve 100% coverage but only 4% mutation score. Although a few studies consider mutation score, the effectiveness of LLMs in killing mutants remains underexplored. In this paper, we propose MUTGEN, a mutation-guided, LLM-based test generation approach that incorporates mutation feedback directly into the prompt. Evaluated on 204 subjects from two benchmarks, MUTGEN significantly outperforms both EvoSuite and vanilla prompt-based strategies in terms of mutation score. Furthermore, MUTGEN introduces an iterative generation mechanism that pushes the limits of LLMs in killing additional mutants. Our study also provide insights into the limitations of LLM-based generation, analyzing the reasons for live and uncovered mutants, and the impact of different mutation operators on generation effectiveness.