mlsanityData to 7 Oct 2026

Software Testing with Large Language Models: Survey, Landscape, and Vision

arXiv 2307.07221 ↗First posted 14 Jul 2023cs.SEcited by 163

Not on the map. Why?

Cited by 31 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained large language models (LLMs) have recently emerged as a breakthrough technology in natural language processing and artificial intelligence, with the ability to handle large-scale datasets and exhibit remarkable performance across a wide range of tasks. Meanwhile, software testing is a crucial undertaking that serves as a cornerstone for ensuring the quality and reliability of software products. As the scope and complexity of software systems continue to grow, the need for more effective software testing techniques becomes increasingly urgent, making it an area ripe for innovative approaches such as the use of LLMs. This paper provides a comprehensive review of the utilization of LLMs in software testing. It analyzes 102 relevant studies that have used LLMs for software testing, from both the software testing and LLMs perspectives. The paper presents a detailed discussion of the software testing tasks for which LLMs are commonly used, among which test case preparation and program repair are the most representative. It also analyzes the commonly used LLMs, the types of prompt engineering that are employed, as well as the accompanied techniques with these LLMs. It also summarizes the key challenges and potential opportunities in this direction. This work can serve as a roadmap for future research in this area, highlighting potential avenues for exploration, and identifying gaps in our current understanding of the use of LLMs in software testing.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jul 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 55; 2025: 53; 2026: 45.

163 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  5. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  6. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  7. 2024CoverUp: Effective High Coverage Test Generation for Pythoncited by 36
  8. 2024Participation in the age of foundation modelscited by 32
  9. 2024Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4cited by 28
  10. 2024An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Projectcited by 28
  11. 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
  12. 2024HDLdebugger: Streamlining HDL debugging with Large Language Modelscited by 25
38 more
  1. 2024How secure is AI-generated Code: A Large-Scale Comparison of Large Language Modelscited by 23
  2. 2024The Future of AI-Driven Software Engineeringcited by 23
  3. 2025Mutation-Guided LLM-based Test Generation at Metacited by 23
  4. 2024Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysiscited by 20
  5. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
  6. 2024X-lifecycle Learning for Cloud Incident Management using LLMscited by 19
  7. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  8. 2025Generative AI for Software Architecture. Applications, Challenges, and Future Directionscited by 15
  9. 2025Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineeringcited by 15
  10. 2023CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Modelcited by 14
  11. 2024The Impact of Prompt Programming on Function-Level Code Generationcited by 13
  12. 2025Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studiescited by 12
  13. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  14. 2024Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generationcited by 11
  15. 2024Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Testscited by 11
  16. 2024LLMs: A Game-Changer for Software Engineers?cited by 11
  17. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary resultscited by 10
  18. 2024RepoMasterEval: Evaluating Code Completion via Real-World Repositoriescited by 10
  19. 2024Fairness Testing of Large Language Models in Role-Playingcited by 10
  20. 2025A Contemporary Survey of Large Language Model Assisted Program Analysiscited by 10
  21. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  22. 2024VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patchingcited by 9
  23. 2024An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasetscited by 8
  24. 2024Location is Key: Leveraging Large Language Model for Functional Bug Localization in Verilogcited by 8
  25. 2025Automating a Complete Software Test Process Using LLMs: An Automotive Case Studycited by 8
  26. 2025LLM-Explorer: Towards Efficient and Affordable LLM-based Exploration for Mobile Appscited by 8
  27. 2023Assessing Evaluation Metrics for Neural Test Oracle Generationcited by 7
  28. 2025CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verificationcited by 7
  29. 2025Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystemcited by 7
  30. 2024Experimenting with Multi-Agent Software Development: Towards a Unified Platformcited by 6
  31. 2024Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Modelscited by 6
  32. 2024Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMscited by 6
  33. 2024Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programmingcited by 6
  34. 2023Testing learning-enabled cyber-physical systems with Large-Language Models: A Formal Approachcited by 5
  35. 2024API Pack: A Massive Multi-Programming Language Dataset for API Call Generationcited by 5
  36. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 5
  37. 2024Automated Prompt Engineering for Cost-Effective Code Generation Using Evolutionary Algorithmcited by 5
  38. 2024Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFastcited by 5

50 most cited of 163 All 163 (JSON)