# FastCoder: Accelerating Repository-level Code Generation via Efficient Retrieval and Verification arXiv: https://arxiv.org/abs/2502.17139 ยท HTML: https://mlsanity.com/paper/2502.17139/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. - Builds on 25 of its 36 references that are in the map. - First posted on arXiv 24 Feb 2025. ## Placed in - Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/) ## Builds on (most cited first, 25 of 25) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022) - [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023) - [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024) - [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023) - [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024) - [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023) - [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023) - [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022) - [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018) - [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024) - [CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion](https://mlsanity.com/paper/2310.11248/) (2023) - [Accelerating LLM Inference with Staged Speculative Decoding](https://mlsanity.com/paper/2308.04623/) (2023) - [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023) - [DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories](https://mlsanity.com/paper/2405.19856/) (2024) - [Inference with Reference: Lossless Acceleration of Large Language Models](https://mlsanity.com/paper/2304.04487/) (2023) - [RLCoder: Reinforcement Learning for Repository-Level Code Completion](https://mlsanity.com/paper/2407.19487/) (2024) - [Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion](https://mlsanity.com/paper/2405.19782/) (2024) - [REPOFUSE: Repository-Level Code Completion with Fused Dual Context](https://mlsanity.com/paper/2402.14323/) (2024) - [Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding](https://mlsanity.com/paper/2402.13720/) (2024) - [Deep Learning-based Software Engineering: Progress, Challenges, and Opportunities](https://mlsanity.com/paper/2410.13110/) (2024) ## Built on by (most cited first, 2 of 2) - [Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches](https://mlsanity.com/paper/2510.04905/) (2025) - [AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines](https://mlsanity.com/paper/2610.01108/) (2026) ## Abstract (arXiv) Code generation is a latency-sensitive task that demands high timeliness. However, with the growing interest and inherent difficulty in repository-level code generation, most existing code generation studies focus on improving the correctness of generated code while overlooking the inference efficiency, which is substantially affected by the overhead during LLM generation. Although there has been work on accelerating LLM inference, these approaches are not tailored to the specific characteristics of code generation; instead, they treat code the same as natural language sequences and ignore its unique syntax and semantic characteristics, which are also crucial for improving efficiency. Consequently, these approaches exhibit limited effectiveness in code generation tasks, particularly for repository-level scenarios with considerable complexity and difficulty. To alleviate this issue, following draft-verification paradigm, we propose FastCoder, a simple yet highly efficient inference acceleration approach specifically designed for code generation, without compromising the quality of the output. FastCoder constructs a multi-source datastore, providing access to both general and project-specific knowledge, facilitating the retrieval of high-quality draft sequences. Moreover, FastCoder reduces the retrieval cost by controlling retrieval timing, and enhances efficiency through parallel retrieval and a context- and LLM preference-aware cache. Experimental results show that FastCoder can reach up to 2.53x and 2.54x speedup compared to autoregressive decoding in repository-level and standalone code generation tasks, respectively, outperforming state-of-the-art inference acceleration approaches by up to 88%. FastCoder can also be integrated with existing correctness-focused code generation approaches to accelerate the LLM generation process, and reach a speedup exceeding 2.6x.