mlsanityData to 14 Oct 2026

RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving

arXiv 2503.14649 ↗First posted 18 Mar 2025cs.IR cs.AI cs.CLcited by 19

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

also in 1 more
Abstract (arXiv)

Retrieval-augmented generation (RAG), which combines large language models (LLMs) with retrievals from external knowledge databases, is emerging as a popular approach for reliable LLM serving. However, efficient RAG serving remains an open challenge due to the rapid emergence of many RAG variants and the substantial differences in workload characteristics across them. In this paper, we make three fundamental contributions to advancing RAG serving. First, we introduce RAGSchema, a structured abstraction that captures the wide range of RAG algorithms, serving as a foundation for performance optimization. Second, we analyze several representative RAG workloads with distinct RAGSchema, revealing significant performance variability across these workloads. Third, to address this variability and meet diverse performance requirements, we propose RAGO (Retrieval-Augmented Generation Optimizer), a system optimization framework for efficient RAG serving. Our evaluation shows that RAGO achieves up to a 2x increase in QPS per chip and a 55% reduction in time-to-first-token latency compared to RAG systems built on LLM-system extensions.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 46 of its 124 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

46 of 124 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach23 Jul
  2. 2024Multi-Head RAG: Solving Multi-Aspect Problems with LLMs7 Jun
  3. 2024CRAG -- Comprehensive RAG Benchmark7 Jun
  4. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design8 Mar
  5. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models15 Oct

Other references · 41

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2017Billion-scale similarity search with GPUscited by 2,497
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  10. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
29 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  2. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  3. 2022Galactica: A Large Language Model for Sciencecited by 744
  4. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  5. 2019ELI5: Long Form Question Answeringcited by 575
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  7. 2023Active Retrieval Augmented Generationcited by 516
  8. 2021Internet-Augmented Dialogue Generationcited by 217
  9. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 213
  10. 2020Nearest Neighbor Machine Translationcited by 209
  11. 2017Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graphcited by 104
  12. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  13. 2020Pre-training via Paraphrasingcited by 92
  14. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  15. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  16. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  17. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  18. 2024SPFresh: Incremental In-Place Update for Billion-Scale Vector Searchcited by 28
  19. 2023High-Dimensional Approximate Nearest Neighbor Search: with Reliable and Efficient Distance Comparison Operationscited by 26
  20. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 23
  21. 2024ChunkRAG: Novel LLM-Chunk Filtering Method for RAG Systemscited by 23
  22. 2021A Full-Stack Search Technique for Domain Optimized Deep Learning Acceleratorscited by 19
  23. 2023High-Throughput Vector Similarity Search in Knowledge Graphscited by 19
  24. 2019GGNN: Graph-based GPU Nearest Neighbor Searchcited by 10
  25. 2018In-RDBMS Hardware Acceleration of Advanced Analyticscited by 8
  26. 2023Automating Nearest Neighbor Search Configuration with Constrained Optimizationcited by 7
  27. 2023Co-design Hardware and Algorithm for Vector Searchcited by 6
  28. 2023JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mappingcited by 5
  29. 2024Fast Graph Vector Search via Hardware Acceleration and Delayed-Synchronization Traversalcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 12.

19 papers.

Sort:

By area:

  1. 2026How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perfcited by 19
  2. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 14
  3. 2025HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Servingcited by 3
  4. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  5. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPUcited by 2
  6. 2025PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devicescited by 1
  7. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-designcited by 1
  8. 2026Cost-Governed RAG: Unified Per-Tenant Cost Attribution Across Retrieval and Generation in Multi-Tenant LLM Systemscited by 1
  9. 2026RAG-Stack: Co-Optimizing RAG Serving Performance and Qualitycited by 1
  10. 2025Harmonia: End-to-End RAG Serving Optimizationcited by 0
  11. 2025Serve Programs, Not Promptscited by 0
  12. 2026TurboVec: A Case Study in Cost-Efficient Private Retrieval for Enterprise RAG via Codebook-Oblivious Quantizationcited by 0
7 more
  1. 2026D-NOVA: In-Storage Retrieval Accelerator via Dual-Bound 3D NAND-Optimized Similarity Search with Vector Adaptationcited by 0
  2. 2026KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systemscited by 0
  3. 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Schedulingcited by 0
  4. 2026ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platformcited by 0
  5. 2026Atlas: Optimizing Deployment of Compound AI Workflows on Heterogeneous Clusterscited by 0
  6. 2026RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systemscited by 0
  7. 2026Fast and Accurate: Improving Personalized LLM Performance by Semantic Concentration Inspired Workload Split between RAG and Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RAG-Stack: Co-Optimizing RAG Quality and Performance From the Vector Database Perspective
  2. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation
  3. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  4. 2024Toward Optimal Search and Retrieval for RAG
  5. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  6. 2025RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systems
  7. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  8. 2026RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems