# Large Language Models for Software Testing: A Research Roadmap

arXiv: https://arxiv.org/abs/2509.25043 · HTML: https://mlsanity.com/paper/2509.25043/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 151 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 30 of 62)

- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020)
- [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- [Formal Mathematical Reasoning: A New Frontier in AI](https://mlsanity.com/paper/2412.16075/) (2024)
- [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024)
- [Hypothesis Generation with Large Language Models](https://mlsanity.com/paper/2404.04326/) (2024)
- [Towards an Understanding of Large Language Models in Software Engineering Tasks](https://mlsanity.com/paper/2308.11396/) (2023)
- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [TOGA: A Neural Method for Test Oracle Generation](https://mlsanity.com/paper/2109.09262/) (2021)
- [Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?](https://mlsanity.com/paper/2310.01831/) (2023)
- [Using Large Language Models to Generate JUnit Tests: An Empirical Study](https://mlsanity.com/paper/2305.00418/) (2023)
- [SpecGen: Automated Generation of Formal Program Specifications via Large Language Models](https://mlsanity.com/paper/2401.08807/) (2024)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [On Learning Meaningful Assert Statements for Unit Test Cases](https://mlsanity.com/paper/2002.05800/) (2020)
- [Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions](https://mlsanity.com/paper/2310.15780/) (2023)
- [CoverUp: Effective High Coverage Test Generation for Python](https://mlsanity.com/paper/2403.16218/) (2024)
- [HITS: High-coverage LLM-based Unit Test Generation via Method Slicing](https://mlsanity.com/paper/2408.11324/) (2024)
- [Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing](https://mlsanity.com/paper/2212.04732/) (2022)
- [On the Evaluation of Large Language Models in Unit Test Generation](https://mlsanity.com/paper/2406.18181/) (2024)
- [Navigating the Complexity of Generative AI Adoption in Software Engineering](https://mlsanity.com/paper/2307.06081/) (2023)
- [Learning Deep Semantics for Test Completion](https://mlsanity.com/paper/2302.10166/) (2023)
- [Can Large Language Models Write Good Property-Based Tests?](https://mlsanity.com/paper/2307.04346/) (2023)
- [Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers](https://mlsanity.com/paper/2009.05634/) (2020)

## Built on by (most cited first, 2 of 2)

- [Human-AI Collaboration for Scaling Agile Regression Testing: An Agentic-AI Teammate from Manual to Automated Testing](https://mlsanity.com/paper/2603.08190/) (2026)
- [RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements](https://mlsanity.com/paper/2604.25862/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are starting to be profiled as one of the most significant disruptions in the Software Testing field. Specifically, they have been successfully applied in software testing tasks such as generating test code, or summarizing documentation. This potential has attracted hundreds of researchers, resulting in dozens of new contributions every month, hardening researchers to stay at the forefront of the wave. Still, to the best of our knowledge, no prior work has provided a structured vision of the progress and most relevant research trends in LLM-based testing. In this article, we aim to provide a roadmap that illustrates its current state, grouping the contributions into different categories, and also sketching the most promising and active research directions for the field. To achieve this objective, we have conducted a semi-systematic literature review, collecting articles and mapping them into the most prominent categories, reviewing the current and ongoing status, and analyzing the open challenges of LLM-based software testing. Lastly, we have outlined several expected long-term impacts of LLMs over the whole software testing field.