mlsanityData to 14 Oct 2026

Accelerating Retrieval-Augmented Generation

arXiv 2412.15246 ↗First posted 14 Dec 2024cs.CL cs.AI cs.ARcited by 14

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

An evolving solution to address hallucination and enhance accuracy in large language models (LLMs) is Retrieval-Augmented Generation (RAG), which involves augmenting LLMs with information retrieved from an external knowledge source, such as the web. This paper profiles several RAG execution pipelines and demystifies the complex interplay between their retrieval and generation phases. We demonstrate that while exact retrieval schemes are expensive, they can reduce inference time compared to approximate retrieval variants because an exact retrieval model can send a smaller but more accurate list of documents to the generative model while maintaining the same end-to-end accuracy. This observation motivates the acceleration of the exact nearest neighbor search for RAG. In this work, we design Intelligent Knowledge Store (IKS), a type-2 CXL device that implements a scale-out near-memory acceleration architecture with a novel cache-coherent interface between the host CPU and near-memory accelerators. IKS offers 13.4-27.9x faster exact nearest neighbor search over a 512GB vector database compared with executing the search on Intel Sapphire Rapids CPUs. This higher search performance translates to 1.7-26.3x lower end-to-end inference time for representative RAG applications. IKS is inherently a memory expander; its internal DRAM can be disaggregated and used for other applications running on the server to prevent DRAM, which is the most expensive component in today's servers, from being stranded.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 60 of its 117 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

60 of 117 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximization5 May
  2. 2024Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models30 Apr
  3. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generation21 Apr
  4. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation9 Apr
  5. 2024Corrective Retrieval Augmented Generation29 Jan
  6. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation28 Sep
  7. 2022A Survey on Retrieval-Augmented Text Generation2 Feb

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  9. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  10. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  11. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  12. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
38 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  2. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  3. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  4. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  5. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  7. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  8. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  9. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  10. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  11. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  12. 2021Retrieval Augmented Code Generation and Summarizationcited by 149
  13. 2022On The Computational Complexity of Self-Attentioncited by 139
  14. 2018ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithmscited by 130
  15. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 127
  16. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  17. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 97
  18. 2020Retrieval-Augmented Generation for Code Summarization via Hybrid GNNcited by 89
  19. 2018Retrieve and Refine: Improved Sequence Generation Models For Dialoguecited by 82
  20. 2021A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Searchcited by 82
  21. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  22. 2018Guiding Neural Machine Translation with Retrieved Translation Piecescited by 63
  23. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
  24. 2018Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memorycited by 49
  25. 2022Retrieval-Enhanced Machine Learningcited by 47
  26. 2018Response Generation by Context-aware Prototype Editingcited by 40
  27. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  28. 2021Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented Dialogcited by 33
  29. 2021DeepScaleTool : A Tool for the Accurate Estimation of Technology Scaling in the Deep-Submicron Eracited by 30
  30. 2023Retrieval-augmented Image Captioningcited by 29
  31. 2019Text Generation with Exemplar-based Adaptive Decodingcited by 28
  32. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  33. 2023Efficient LLM Inference on CPUscited by 24
  34. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 21
  35. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  36. 2024Learning to Rank for Multiple Retrieval-Augmented Models through Iterative Utility Maximizationcited by 14
  37. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  38. 2023Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answeringcited by 10

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  2. 2025Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agentscited by 5
  3. 2025Cache Mechanism for Agent RAG Systemscited by 5
  4. 2026Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMscited by 5
  5. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processingcited by 4
  6. 2025Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspectivecited by 4
  7. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAGcited by 3
  8. 2025SINDI: An Efficient Index for Sparse Vector Approximate Maximum Inner Product Searchcited by 2
  9. 2025DIRC-RAG: Accelerating Edge RAG with Robust High-Density and High-Loading-Bandwidth Digital In-ReRAM Computationcited by 1
  10. 2026GPU-Accelerated ANNS: Quantized for Speed, Built for Changecited by 1
  11. 2026AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systemscited by 1
  12. 2026HaS: Accelerating RAG through Homology-Aware Speculative Retrievalcited by 0
2 more
  1. 2026NasZip: Software and Hardware Co-Design to Accelerate Approximate Nearest Neighbor Search with DIMM-Based Near-Data Processingcited by 0
  2. 2026Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scalingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
  2. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  3. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  4. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  6. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  7. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting
  8. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation