mlsanityData to 7 Oct 2026

PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design

arXiv 2403.05676 ↗First posted 8 Mar 2024cs.CLcited by 28

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-augmented generation (RAG) can enhance the generation quality of large language models (LLMs) by incorporating external token databases. However, retrievals from large databases can constitute a substantial portion of the overall generation time, particularly when retrievals are periodically performed to align the retrieved content with the latest states of generation. In this paper, we introduce PipeRAG, a novel algorithm-system co-design approach to reduce generation latency and enhance generation quality. PipeRAG integrates (1) pipeline parallelism to enable concurrent retrieval and generation processes, (2) flexible retrieval intervals to maximize the efficiency of pipeline parallelism, and (3) a performance model to automatically balance retrieval quality and latency based on the generation states and underlying hardware. Our evaluation shows that, by combining the three aforementioned methods, PipeRAG achieves up to 2.6× speedup in end-to-end generation latency while improving generation quality. These promising results showcase the effectiveness of co-designing algorithms with underlying systems, paving the way for the adoption of PipeRAG in future RAG systems.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 42 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

36 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models15 Oct

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
23 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  4. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  8. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  9. 2019Passage Re-ranking with BERTcited by 629
  10. 2023Active Retrieval Augmented Generationcited by 506
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  13. 2021Internet-Augmented Dialogue Generationcited by 217
  14. 2020Nearest Neighbor Machine Translationcited by 209
  15. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 111
  16. 2020Pre-training via Paraphrasingcited by 92
  17. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  18. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answeringcited by 75
  19. 2021Fast Nearest Neighbor Machine Translationcited by 37
  20. 2023KNN-LM Does Not Improve Open-ended Text Generationcited by 10
  21. 2023Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Modelscited by 7
  22. 2023Co-design Hardware and Algorithm for Vector Searchcited by 6
  23. 2023On the Generalization Ability of Retrieval-Enhanced Transformerscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 10; 2026: 7.

28 papers.

Sort:

By area:

  1. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  2. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  3. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  4. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  5. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  6. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  7. 2025When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generationcited by 46
  8. 2024When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?cited by 20
  9. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 18
  10. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  11. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  12. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inferencecited by 15
16 more
  1. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  2. 2026Does Memory Need Graphs? A Unified Framework and Empirical Analysis for Long-Term Dialog Memorycited by 9
  3. 2024RAGProbe: An Automated Approach for Evaluating RAG Applicationscited by 7
  4. 2026Pancake: Hierarchical Memory System for Multi-Agent LLM Servingcited by 5
  5. 2024RIRAG: Regulatory Information Retrieval and Answer Generationcited by 3
  6. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAGcited by 3
  7. 2025After Retrieval, Before Generation: Enhancing the Trustworthiness of Large Language Models in Retrieval-Augmented Generationcited by 3
  8. 2025HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Servingcited by 3
  9. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPUcited by 2
  10. 2025PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devicescited by 1
  11. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-designcited by 1
  12. 2026SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariancecited by 1
  13. 2024Enhancing E-commerce Product Title Translation with Retrieval-Augmented Generation and Large Language Modelscited by 0
  14. 2026Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generationcited by 0
  15. 2026Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)cited by 0
  16. 2026RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  2. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  3. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
  4. 2025FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  6. 2024Accelerating Retrieval-Augmented Generation
  7. 2024AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline
  8. 2026Bridge-RAG: An Abstract Bridge Tree Based Retrieval Augmented Generation Algorithm