mlsanityData to 7 Oct 2026

Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation

arXiv 2604.15270 ↗First posted 16 Apr 2026cs.SEcited by 0

On the map: Retrieval-augmented generation → RAG for code

Abstract (arXiv)

In this paper, we focus on automating two of the widely used Verification and Validation (V&V) activities in the Software Development Lifecycle (SDLC): Software testing and software inspection (also known as review). Concerning the former, we concentrate on automated test case generation using Large Language Models (LLMs). For the latter, we enable inspection of the source code by LLMs. To address the known LLM hallucination problem, in which LLMs confidently produce incorrect outputs, we implement a Retrieval Augmented Generation (RAG) pipeline to integrate supplementary knowledge sources and provide additional context to the LLM. Our experimental results indicate that incorporating external context via the RAG pipeline has a generally positive impact on both test case generation and code inspection. This novel approach reduces the total project cost by saving human testers'/inspectors' time. It also improves the effectiveness and efficiency of these V&V activities, as evidenced by our experimental study.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG for code, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 56 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

38 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Retrieval Augmented Code Generation and Summarization26 Aug

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  11. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  12. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
25 more
  1. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  2. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  3. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  4. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 94
  5. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  6. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  7. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  8. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunitiescited by 63
  9. 2023Using Large Language Models to Generate JUnit Tests: An Empirical Studycited by 53
  10. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  11. 2023Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Studycited by 44
  12. 2023Large Language Models for Code Analysis: Do LLMs Really Do Their Job?cited by 44
  13. 2024TESTEVAL: Benchmarking Large Language Models for Test Case Generationcited by 38
  14. 2023Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preferencecited by 37
  15. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 29
  16. 2024Large Language Models as Test Case Generators: Performance Evaluation and Enhancementcited by 27
  17. 2024Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automationcited by 23
  18. 2021Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generationcited by 19
  19. 2023E&V: Prompting Large Language Models to Perform Static Analysis by Pseudo-code Execution and Verificationcited by 9
  20. 2025Automating a Complete Software Test Process Using LLMs: An Automotive Case Studycited by 8
  21. 2024Improving the Learning of Code Review Successive Tasks with Cross-Task Knowledge Distillationcited by 7
  22. 2025Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actionscited by 7
  23. 2024Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stackscited by 6
  24. 2025CodeReviewQA: The Code Review Comprehension Assessment for Large Language Modelscited by 3
  25. 2025Previously on... Automating Code Reviewcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM-based Unit Test Generation via Property Retrieval
  2. 2024Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Study
  3. 2025Generating High-Level Test Cases from Requirements using LLM: An Industry Study
  4. 2025Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol
  5. 2025From Code Generation to Software Testing: AI Copilot with Context-Based RAG
  6. 2025AugmenTest: Enhancing Tests with LLM-Driven Oracles
  7. 2025Improving Retrieval-Augmented Deep Assertion Generation via Joint Training
  8. 2025Test code generation at Ericsson using Program Analysis Augmented Fine Tuned LLMs