# RepoFusion: Training Code Models to Understand Your Repository

arXiv: https://arxiv.org/abs/2306.10998 · HTML: https://mlsanity.com/paper/2306.10998/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 19 Jun 2023.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [SantaCoder: don't reach for the stars!](https://mlsanity.com/paper/2301.03988/) (2023)
- [DocPrompting: Generating Code by Retrieving the Docs](https://mlsanity.com/paper/2207.05987/) (2022)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [Repository-Level Prompt Generation for Large Language Models of Code](https://mlsanity.com/paper/2206.12839/) (2022)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Context](https://mlsanity.com/paper/2212.10007/) (2022)
- [When Language Model Meets Private Library](https://mlsanity.com/paper/2210.17236/) (2022)
- [FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference](https://mlsanity.com/paper/2212.08153/) (2022)
- [How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot](https://mlsanity.com/paper/2208.14613/) (2022)
- [Capturing Structural Locality in Non-parametric Language Models](https://mlsanity.com/paper/2110.02870/) (2021)
- [Learning to Generate Code Comments from Class Hierarchies](https://mlsanity.com/paper/2103.13426/) (2021)
- [Embedding API Dependency Graph for Neural Code Generation](https://mlsanity.com/paper/2103.15361/) (2021)

## Built on by (most cited first, 30 of 40)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases](https://mlsanity.com/paper/2408.03910/) (2024)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [Repoformer: Selective Retrieval for Repository-Level Code Completion](https://mlsanity.com/paper/2403.10059/) (2024)
- [AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology](https://mlsanity.com/paper/2406.11912/) (2024)
- [GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Model](https://mlsanity.com/paper/2406.07003/) (2024)
- [SWE-QA: Can Language Models Answer Repository-level Code Questions?](https://mlsanity.com/paper/2509.14635/) (2025)
- [REPOFUSE: Repository-Level Code Completion with Fused Dual Context](https://mlsanity.com/paper/2402.14323/) (2024)
- [SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving](https://mlsanity.com/paper/2601.01426/) (2026)
- [BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models](https://mlsanity.com/paper/2308.16458/) (2023)
- [Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches](https://mlsanity.com/paper/2510.04905/) (2025)
- [M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation](https://mlsanity.com/paper/2410.21157/) (2024)
- [In Line with Context: Repository-Level Code Generation via Context Inlining](https://mlsanity.com/paper/2601.00376/) (2026)
- [RepoHyper: Search-Expand-Refine on Semantic Graphs for Repository-Level Code Completion](https://mlsanity.com/paper/2403.06095/) (2024)
- [Enhancing LLM-Based Coding Tools through Native Integration of IDE-Derived Static Context](https://mlsanity.com/paper/2402.03630/) (2024)
- [InCoRo: In-Context Learning for Robotics Control with Feedback Loops](https://mlsanity.com/paper/2402.05188/) (2024)
- [Towards an Understanding of Context Utilization in Code Intelligence](https://mlsanity.com/paper/2504.08734/) (2025)
- [R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models](https://mlsanity.com/paper/2406.01359/) (2024)
- [RTLRepoCoder: Repository-Level RTL Code Completion through the Combination of Fine-Tuning and Retrieval Augmentation](https://mlsanity.com/paper/2504.08862/) (2025)
- [AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion](https://mlsanity.com/paper/2601.19697/) (2026)
- [CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation](https://mlsanity.com/paper/2504.13472/) (2025)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [Multi-agent Collaboration with State Management](https://mlsanity.com/paper/2605.20563/) (2026)
- [Building A Coding Assistant via the Retrieval-Augmented Language Model](https://mlsanity.com/paper/2410.16229/) (2024)
- [YABLoCo: Yet Another Benchmark for Long Context Code Generation](https://mlsanity.com/paper/2505.04406/) (2025)
- [Enhancing Project-Specific Code Completion by Inferring Internal API Information](https://mlsanity.com/paper/2507.20888/) (2025)
- [Impact-driven Context Filtering For Cross-file Code Completion](https://mlsanity.com/paper/2508.05970/) (2025)

## Abstract (arXiv)

Despite the huge success of Large Language Models (LLMs) in coding assistants like GitHub Copilot, these models struggle to understand the context present in the repository (e.g., imports, parent classes, files with similar names, etc.), thereby producing inaccurate code completions. This effect is more pronounced when using these assistants for repositories that the model has not seen during training, such as proprietary software or work-in-progress code projects. Recent work has shown the promise of using context from the repository during inference. In this work, we extend this idea and propose RepoFusion, a framework to train models to incorporate relevant repository context. Experiments on single-line code completion show that our models trained with repository context significantly outperform much larger code models as CodeGen-16B-multi (~73× larger) and closely match the performance of the ~ 70× larger StarCoderBase model that was trained with the Fill-in-the-Middle objective. We find these results to be a novel and compelling demonstration of the gains that training with repository context can bring. We carry out extensive ablation studies to investigate the impact of design choices such as context type, number of contexts, context length, and initialization within our framework. Lastly, we release Stack-Repo, a dataset of 200 Java repositories with permissive licenses and near-deduplicated files that are augmented with three types of repository contexts. Additionally, we are making available the code and trained checkpoints for our work. Our released resources can be found at urlhttps://huggingface.co/RepoFusion.