# Practical Code RAG at Scale: Task-Aware Retrieval Design Choices under Compute Budgets

arXiv: https://arxiv.org/abs/2510.20609 · HTML: https://mlsanity.com/paper/2510.20609/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Builds on (most cited first, 28 of 28)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [DocPrompting: Generating Code by Retrieving the Docs](https://mlsanity.com/paper/2207.05987/) (2022)
- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [Dense X Retrieval: What Retrieval Granularity Should We Use?](https://mlsanity.com/paper/2312.06648/) (2023)
- [Repoformer: Selective Retrieval for Repository-Level Code Completion](https://mlsanity.com/paper/2403.10059/) (2024)
- [Long Code Arena: a Set of Benchmarks for Long-Context Code Models](https://mlsanity.com/paper/2406.11612/) (2024)
- [RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair](https://mlsanity.com/paper/2309.06057/) (2023)
- [Is Semantic Chunking Worth the Computational Cost?](https://mlsanity.com/paper/2410.13070/) (2024)
- [Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion](https://mlsanity.com/paper/2405.19782/) (2024)
- [A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions](https://mlsanity.com/paper/2507.18910/) (2025)
- [A Proposed Conceptual Framework for a Representational Approach to Information Retrieval](https://mlsanity.com/paper/2110.01529/) (2021)
- [ChunkRAG: Novel LLM-Chunk Filtering Method for RAG Systems](https://mlsanity.com/paper/2410.19572/) (2024)
- [Rewriting the Code: A Simple Method for Large Language Model Augmented Code Search](https://mlsanity.com/paper/2401.04514/) (2024)
- [METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation](https://mlsanity.com/paper/2412.10543/) (2024)
- [GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generation](https://mlsanity.com/paper/2504.10046/) (2025)
- [cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree](https://mlsanity.com/paper/2506.15655/) (2025)
- [Prompt-based Code Completion via Multi-Retrieval Augmented Generation](https://mlsanity.com/paper/2405.07530/) (2024)
- [Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference](https://mlsanity.com/paper/2412.11854/) (2024)
- [The Devil is in the Tails: How Long-Tailed Code Distributions Impact Large Language Models](https://mlsanity.com/paper/2309.03567/) (2023)
- [Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking](https://mlsanity.com/paper/2507.09935/) (2025)
- [Better Than Whitespace: Information Retrieval for Languages without Custom Tokenizers](https://mlsanity.com/paper/2210.05481/) (2022)
- [Multi-View Adaptive Contrastive Learning for Information Retrieval Based Fault Localization](https://mlsanity.com/paper/2409.12519/) (2024)

## Built on by (most cited first, 3 of 3)

- [A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity](https://mlsanity.com/paper/2603.06976/) (2026)
- [How Does Chunking Affect Retrieval-Augmented Code Completion? A Controlled Empirical Study](https://mlsanity.com/paper/2605.04763/) (2026)
- [The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context](https://mlsanity.com/paper/2608.14838/) (2026)

## Abstract (arXiv)

We study retrieval design for code-focused generation tasks under realistic compute budgets. Using two complementary tasks from Long Code Arena -- code completion and bug localization -- we systematically compare retrieval configurations across various context window sizes along three axes: (i) chunking strategy, (ii) similarity scoring, and (iii) splitting granularity. (1) For PL-PL, sparse BM25 with word-level splitting is the most effective and practical, significantly outperforming dense alternatives while being an order of magnitude faster. (2) For NL-PL, proprietary dense encoders (Voyager-3 family) consistently beat sparse retrievers, however requiring 100x larger latency. (3) Optimal chunk size scales with available context: 32-64 line chunks work best at small budgets, and whole-file retrieval becomes competitive at 16000 tokens. (4) Simple line-based chunking matches syntax-aware splitting across budgets. (5) Retrieval latency varies by up to 200x across configurations; BPE-based splitting is needlessly slow, and BM25 + word splitting offers the best quality-latency trade-off. Thus, we provide evidence-based recommendations for implementing effective code-oriented RAG systems based on task requirements, model constraints, and computational efficiency.