mlsanityData to 7 Oct 2026

Retrieval-Augmented Test Generation: How Far Are We?

arXiv 2409.12682 ↗First posted 19 Sep 2024cs.SE cs.AIcited by 7

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval Augmented Generation (RAG) has advanced software engineering tasks but remains underexplored in unit test generation. To bridge this gap, we investigate the efficacy of RAG-based unit test generation for machine learning (ML/DL) APIs and analyze the impact of different knowledge sources on their effectiveness. We examine three domain-specific sources for RAG: (1) API documentation (official guidelines), (2) GitHub issues (developer-reported resolutions), and (3) StackOverflow Q&As (community-driven solutions). Our study focuses on five widely used Python-based ML/DL libraries, TensorFlow, PyTorch, Scikit-learn, Google JAX, and XGBoost, targeting the most-used APIs. We evaluate four state-of-the-art LLMs -- GPT-3.5-Turbo, GPT-4o, Mistral MoE 8x22B, and Llama 3.1 405B -- across three strategies: basic instruction prompting, Basic RAG, and API-level RAG. Quantitatively, we assess syntactical and dynamic correctness and line coverage. While RAG does not enhance correctness, RAG improves line coverage by 6.5% on average. We found that GitHub issues result in the best improvement in line coverage by providing edge cases from various issues. We also found that these generated unit tests can help detect new bugs. Specifically, 28 bugs were detected, 24 unique bugs were reported to developers, ten were confirmed, four were rejected, and ten are awaiting developers' confirmation. Our findings highlight RAG's potential in unit test generation for improving test coverage with well-targeted knowledge sources. Future work should focus on retrieval techniques that identify documents with unique program states to optimize RAG-based unit test generation further.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 95 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

35 of 95 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers22 Mar

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  6. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  7. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  8. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  9. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  10. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 94
  11. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 89
  12. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
22 more
  1. 2023GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysiscited by 65
  2. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 63
  3. 2023A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localizationcited by 50
  4. 2023RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repaircited by 43
  5. 2023Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisionscited by 41
  6. 2023A3Test: Assertion-Augmented Automated Test Case Generationcited by 33
  7. 2024On the Evaluation of Large Language Models in Unit Test Generationcited by 33
  8. 2024LLMParser: An Exploratory Study on Using Large Language Models for Log Parsingcited by 32
  9. 2024Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlesslycited by 23
  10. 2024On Mitigating Code LLM Hallucinations with API Documentationcited by 21
  11. 2024Enhancing Large Language Model Performance To Answer Questions and Extract Information More Accuratelycited by 20
  12. 2023Fuzzing Automatic Differentiation in Deep-Learning Librariescited by 16
  13. 2020Documentation of Machine Learning Softwarecited by 8
  14. 2023Domain Adaptation for Code Model-based Unit Test Case Generationcited by 7
  15. 2023Assessing Evaluation Metrics for Neural Test Oracle Generationcited by 7
  16. 2023The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Taskscited by 5
  17. 2022Basic Block Coverage for Search-based Unit Testing and Crash Reproductioncited by 4
  18. 2019A taxonomy of risk-based testingcited by 3
  19. 2017Studying the Prevalence of Exception Handling Anti-Patternscited by 2
  20. 2023Automatic Unit Test Generation for Deep Learning Frameworks based on API Knowledgecited by 2
  21. 2023Novel Preprocessing Technique for Data Embedding in Engineering Code Generation Using Large Language Modelcited by 2
  22. 2018IntegrationDistiller: Automating Integration Analysis and Testing of Object-Oriented Applicationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?cited by 12
  3. 2025Retrieval-Augmented Code Review Comment Generationcited by 6
  4. 2025AI-Driven Tools in Modern Software Quality Assurance: An Assessment of Benefits, Challenges, and Future Directionscited by 3
  5. 2026Not All RAGs Are Created Equal: A Component-Wise Empirical Study for Software Engineering Taskscited by 1
  6. 2025A Pilot Study on LLM-Based Agentic Translation from Android to iOS: Pitfalls and Insightscited by 0
  7. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines
  2. 2025EncouRAGe: Evaluating RAG Local, Fast, and Reliable
  3. 2024LLM-based Unit Test Generation via Property Retrieval
  4. 2025Improving Retrieval-Augmented Deep Assertion Generation via Joint Training
  5. 2024RAGProbe: An Automated Approach for Evaluating RAG Applications
  6. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
  7. 2025RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation
  8. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation