# Retrieval Augmented Code Generation and Summarization

arXiv: https://arxiv.org/abs/2108.11601 · HTML: https://mlsanity.com/paper/2108.11601/

## Key facts

- Cited by 149 papers in the map (data through 14 Oct 2026).
- Placed in RAG for code, one of 15 directions in Retrieval-augmented generation.
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Placed in

- Retrieval-augmented generation / [RAG for code](https://mlsanity.com/rag/rag-for-code/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [Mapping Language to Code in Programmatic Context](https://mlsanity.com/paper/1808.09588/) (2018)
- [Abstract Syntax Networks for Code Generation and Semantic Parsing](https://mlsanity.com/paper/1704.07535/) (2017)
- [A Convolutional Attention Network for Extreme Summarization of Source Code](https://mlsanity.com/paper/1602.03001/) (2016)
- [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [A Retrieve-and-Edit Framework for Predicting Structured Outputs](https://mlsanity.com/paper/1812.01194/) (2018)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [In-IDE Code Generation from Natural Language: Promise and Challenges](https://mlsanity.com/paper/2101.11149/) (2021)
- [Deep API Learning](https://mlsanity.com/paper/1605.08535/) (2016)
- [Retrieval-Based Neural Code Generation](https://mlsanity.com/paper/1808.10025/) (2018)
- [Incorporating External Knowledge through Pre-training for Natural Language to Code Generation](https://mlsanity.com/paper/2004.09015/) (2020)
- [Quantization based Fast Inner Product Search](https://mlsanity.com/paper/1509.01469/) (2015)

## Built on by (most cited first, 30 of 149)

- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://mlsanity.com/paper/2306.15626/) (2023)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2402.16893/) (2024)
- [DocPrompting: Generating Code by Retrieving the Docs](https://mlsanity.com/paper/2207.05987/) (2022)
- [Physics of Language Models: Part 3.2, Knowledge Manipulation](https://mlsanity.com/paper/2309.14402/) (2023)
- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory](https://mlsanity.com/paper/2305.02437/) (2023)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models](https://mlsanity.com/paper/2406.00083/) (2024)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models](https://mlsanity.com/paper/2310.04027/) (2023)
- [LitLLM: A Toolkit for Scientific Literature Review](https://mlsanity.com/paper/2402.01788/) (2024)
- [RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation](https://mlsanity.com/paper/2408.08067/) (2024)
- [VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents](https://mlsanity.com/paper/2504.09795/) (2025)
- [NatGen: Generative pre-training by "Naturalizing" source code](https://mlsanity.com/paper/2206.07585/) (2022)
- [When Language Model Meets Private Library](https://mlsanity.com/paper/2210.17236/) (2022)
- [Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data](https://mlsanity.com/paper/2406.14773/) (2024)
- [Compilable Neural Code Generation with Compiler Feedback](https://mlsanity.com/paper/2203.05132/) (2022)
- [xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://mlsanity.com/paper/2303.03004/) (2023)
- [RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair](https://mlsanity.com/paper/2309.06057/) (2023)
- [Multilingual training for Software Engineering](https://mlsanity.com/paper/2112.02043/) (2021)
- [SelfEvolve: A Code Evolution Framework via Large Language Models](https://mlsanity.com/paper/2306.02907/) (2023)

## Abstract (arXiv)

Software developers write a lot of source code and documentation during software development. Intrinsically, developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them. To mimic developers' code or summary generation behavior, we propose a retrieval augmented framework, REDCODER, that retrieves relevant code or summaries from a retrieval database and provides them as a supplement to code generation or summarization models. REDCODER has a couple of uniqueness. First, it extends the state-of-the-art dense retrieval technique to search for relevant code or summaries. Second, it can work with retrieval databases that include unimodal (only code or natural language description) or bimodal instances (code-description pairs). We conduct experiments and extensive analysis on two benchmark datasets of code generation and summarization in Java and Python, and the promising results endorse the effectiveness of our proposed retrieval augmented framework.