mlsanityData to 7 Oct 2026

CodeRAG-Bench: Can Retrieval Augment Code Generation?

arXiv 2406.14497 ↗First posted 20 Jun 2024cs.SE cs.CLcited by 89

On the map: Retrieval-augmented generation → RAG for code

Abstract (arXiv)

While language models (LMs) have proven remarkably adept at generating code, many programs are challenging for LMs to generate using their parametric knowledge alone. Providing external contexts such as library documentation can facilitate generating accurate and functional code. Despite the success of retrieval-augmented generation (RAG) in various text-oriented tasks, its potential for improving code generation remains under-explored. In this work, we conduct a systematic, large-scale analysis by asking: in what scenarios can retrieval benefit code generation models? and what challenges remain? We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks, including basic programming, open-domain, and repository-level problems. We aggregate documents from five sources for models to retrieve contexts: competition solutions, online tutorials, library documentation, StackOverflow posts, and GitHub repositories. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources. While notable gains are made in final code generation by retrieving high-quality contexts across various settings, our analysis reveals room for improvement -- current retrievers still struggle to fetch useful contexts especially with limited lexical overlap, and generators fail to improve with limited context lengths or abilities to integrate additional contexts. We hope CodeRAG-Bench serves as an effective testbed to encourage further development of advanced code-oriented RAG methods.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in RAG for code, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 52 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

38 of 52 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2018Datasheets for Datasetscited by 1,371
26 more
  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  6. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  7. 2023StarCoder: may the source be with you!cited by 790
  8. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  11. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  13. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  14. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  15. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  16. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  17. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  18. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  19. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  20. 2022Task-aware Retrieval with Instructionscited by 107
  21. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  22. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  23. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 82
  24. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  25. 2024GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuningcited by 40
  26. 2022ClueWeb22: 10 Billion Web Documents with Visual and Semantic Informationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 48; 2026: 30.

89 papers.

Sort:

By area:

  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  2. 2025LocAgent: Graph-Guided LLM Agents for Code Localizationcited by 82
  3. 2024CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Rerankingcited by 37
  4. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  5. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  7. 2024An LLM Agent for Automatic Geospatial Data Analysiscited by 21
  8. 2025Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approachescited by 20
  9. 2025Improving Code Localization with Repository Memorycited by 19
  10. 2025cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Treecited by 17
  11. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  12. 2025CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Executioncited by 14
38 more
  1. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understandingcited by 14
  2. 2025RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testingcited by 12
  3. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  4. 2025Towards an Understanding of Context Utilization in Code Intelligencecited by 10
  5. 2026SERA: Soft-Verified Efficient Repository Agentscited by 10
  6. 2025Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generationcited by 9
  7. 2025Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verificationcited by 9
  8. 2025CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoningcited by 9
  9. 2025SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehensioncited by 8
  10. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generationcited by 8
  11. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  12. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  13. 2024Retrieval-Augmented Test Generation: How Far Are We?cited by 7
  14. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systemscited by 7
  15. 2025Beyond the Prompt: An Empirical Study of Cursor Rulescited by 7
  16. 2026CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agentscited by 7
  17. 2025CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrievalcited by 6
  18. 2025ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generationcited by 5
  19. 2025Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understandingcited by 5
  20. 2026Specification-Based Code-Text-Code Reengineering for LLM-Mediated Software Evolutioncited by 5
  21. 2024Self-adaptive Multimodal Retrieval-Augmented Generationcited by 4
  22. 2024Context-Augmented Code Generation Using Programming Knowledge Graphscited by 4
  23. 2025LLM Program Optimization via Retrieval Augmented Searchcited by 4
  24. 2025OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learningcited by 4
  25. 2025Knowledge Graph Based Repository-Level Code Generationcited by 4
  26. 2025SACL: Understanding and Combating Textual Bias in Code Retrieval with Semantic-Augmented Reranking and Localizationcited by 4
  27. 2025EyeMulator: Improving Code Language Models by Mimicking Human Visual Attentioncited by 4
  28. 2024RaSeRec: Retrieval-Augmented Sequential Recommendationcited by 3
  29. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  30. 2025Enhancing Code Generation via Bidirectional Comment-Level Mutual Groundingcited by 3
  31. 2025CoRet: Improved Retriever for Code Editingcited by 3
  32. 2025Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generationcited by 3
  33. 2025Revela: Dense Retriever Learning via Language Modelingcited by 3
  34. 2025Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategiescited by 3
  35. 2026RepoShapley: Shapley-Enhanced Context Filtering for Repository-Level Code Completioncited by 3
  36. 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
  37. 2026URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Modelscited by 3
  38. 2025AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programmingcited by 2

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond
  2. 2025A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat
  3. 2025An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities
  4. 2024Evaluating the Effectiveness and Efficiency of Demonstration Retrievers in RAG for Coding Tasks
  5. 2026Do Not Treat Code as Natural Language: Implications for Repository-Level Code Generation and Beyond
  6. 2026Not All RAGs Are Created Equal: A Component-Wise Empirical Study for Software Engineering Tasks
  7. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  8. 2026Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation