# Test Wars: A Comparative Study of SBST, Symbolic Execution, and LLM-Based Approaches to Unit Test Generation

arXiv: https://arxiv.org/abs/2501.10200 · HTML: https://mlsanity.com/paper/2501.10200/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2025.

## Built on by (most cited first, 7 of 7)

- [EvoGPT: Leveraging LLM-Driven Seed Diversity to Improve Search-Based Test Suite Generation](https://mlsanity.com/paper/2505.12424/) (2025)
- [Towards Iterative End-to-End Software Development: A Feature-Driven Multi-Agent Framework](https://mlsanity.com/paper/2511.02399/) (2025)
- [Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmap](https://mlsanity.com/paper/2505.19625/) (2025)
- [SPARC: Scenario Planning and Reasoning for Automated C Unit Test Generation](https://mlsanity.com/paper/2602.16671/) (2026)
- [MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation](https://mlsanity.com/paper/2607.22471/) (2026)
- [Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)](https://mlsanity.com/paper/2607.22880/) (2026)
- [Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests](https://mlsanity.com/paper/2607.22883/) (2026)

## Abstract (arXiv)

Generating tests automatically is a key and ongoing area of focus in software engineering research. The emergence of Large Language Models (LLMs) has opened up new opportunities, given their ability to perform a wide spectrum of tasks. However, the effectiveness of LLM-based approaches compared to traditional techniques such as search-based software testing (SBST) and symbolic execution remains uncertain. In this paper, we perform an extensive study of automatic test generation approaches based on three tools: EvoSuite for SBST, Kex for symbolic execution, and TestSpark for LLM-based test generation. We evaluate tools performance on the GitBug Java dataset and compare them using various execution-based and feature-based metrics. Our results show that while LLM-based test generation is promising, it falls behind traditional methods in terms of coverage. However, it significantly outperforms them in mutation scores, suggesting that LLMs provide a deeper semantic understanding of code. LLM-based approach also performed worse than SBST and symbolic execution-based approaches w.r.t. fault detection capabilities. Additionally, our feature-based analysis shows that all tools are primarily affected by the complexity and internal dependencies of the class under test (CUT), with LLM-based approaches being especially sensitive to the CUT size.