mlsanityData to 14 Oct 2026

Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference

arXiv 2412.11854 ↗First posted 16 Dec 2024cs.ARcited by 15

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

The rapid increase in the number of parameters in large language models (LLMs) has significantly increased the cost involved in fine-tuning and retraining LLMs, a necessity for keeping models up to date and improving accuracy. Retrieval-Augmented Generation (RAG) offers a promising approach to improving the capabilities and accuracy of LLMs without the necessity of retraining. Although RAG eliminates the need for continuous retraining to update model data, it incurs a trade-off in the form of slower model inference times. Resultingly, the use of RAG in enhancing the accuracy and capabilities of LLMs often involves diverse performance implications and trade-offs based on its design. In an effort to begin tackling and mitigating the performance penalties associated with RAG from a systems perspective, this paper introduces a detailed taxonomy and characterization of the different elements within the RAG ecosystem for LLMs that explore trade-offs within latency, throughput, and memory. Our study reveals underlying inefficiencies in RAG for systems deployment, that can result in TTFT latencies that are twice as long and unoptimized datastores that consume terabytes of storage.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 20 of its 23 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

20 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design8 Mar

Other references · 19

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  9. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  10. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  12. 2024The Faiss librarycited by 376
7 more
  1. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  2. 2020The Cost of Training NLP Models: A Concise Overviewcited by 98
  3. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  4. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  5. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  6. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  7. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025CacheClip: Accelerating RAG with Effective KV Cache Reusecited by 16
  2. 2026NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systemscited by 6
  3. 2026PruneRAG: Confidence-Guided Query Decomposition Trees for Efficient Retrieval-Augmented Generationcited by 5
  4. 2025LEANN: A Low-Storage Vector Indexcited by 4
  5. 2025Practical Code RAG at Scale: Task-Aware Retrieval Design Choices under Compute Budgetscited by 3
  6. 2025Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Searchcited by 2
  7. 2025CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systemscited by 1
  8. 2025DIRC-RAG: Accelerating Edge RAG with Robust High-Density and High-Loading-Bandwidth Digital In-ReRAM Computationcited by 1
  9. 2025BAR Conjecture: the Feasibility of Inference Budget-Constrained LLM Services with Authenticity and Reasoningcited by 0
  10. 2026GaiaFlow: Semantic-Guided Diffusion Tuning for Carbon-Frugal Searchcited by 0
  11. 2026uGen: An Agentic Framework for Generating Microarchitectural Attack PoCscited by 0
  12. 2026The Coverage Illusion: From Pre-retrieval Routing Failure to Post-retrieval Cascades in a Production RAG Systemcited by 0
3 more
  1. 2026Retrieval-Augmented Generation in LLMs for Mental Health: Quantifying the Incremental Contribution of Retrieval Within a Layered Safety Architecturecited by 0
  2. 2026RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systemscited by 0
  3. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  2. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  4. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  5. 2024Long Context RAG Performance of Large Language Models
  6. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  7. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  8. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach