mlsanityData to 7 Oct 2026

TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval

arXiv 2502.20969 ↗First posted 28 Feb 2025cs.DC cs.LGcited by 9

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely on large datastores, creating a significant system challenge: achieving high throughput and low latency is difficult, especially when GPU memory is limited. To address these challenges, we propose TeleRAG, an efficient inference system that reduces latency and improves throughput with minimal GPU memory requirements. The core innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that predicts required data and transfers them from CPU to GPU in parallel with LLM generation. In addition, TeleRAG adopts a prefetching scheduler and a cache-aware scheduler to support efficient multi-GPU inference with minimal overhead. Evaluations show TeleRAG achieves up to a 1.53x average end-to-end latency reduction (single-query) and 1.83x higher average throughput (batched), as well as good scalability in throughput. This confirms the practical utility of TeleRAG for faster and more memory-efficient deployments of RAG applications.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 65 of its 138 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

65 of 138 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Accelerating Retrieval-Augmented Generation14 Dec
  2. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting11 Jul
  3. 2024FACTS About Building Retrieval Augmented Generation-based Chatbots10 Jul
  4. 2024MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generation2 Jul
  5. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research22 May
  6. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models10 May
  7. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design8 Mar
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  9. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models15 Oct

Other references · 56

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
38 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  5. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  8. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  10. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  11. 2024The Faiss librarycited by 370
  12. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  13. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  14. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  15. 2023An LLM Compiler for Parallel Function Callingcited by 170
  16. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 157
  17. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  18. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  19. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  20. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
  21. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  22. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  23. 2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
  24. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 64
  25. 2023Large Language Model based Long-tail Query Rewriting in Taobao Searchcited by 61
  26. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  27. 2023Query Expansion by Prompting Large Language Modelscited by 56
  28. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 54
  29. 2023Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Rankingcited by 44
  30. 2021SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Searchcited by 35
  31. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
  32. 2022TPU-KNN: K Nearest Neighbor Search at Peak FLOP/scited by 18
  33. 2023Challenges of Large Language Models for Mental Health Counselingcited by 16
  34. 2023Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspectivecited by 16
  35. 2024BANG: Billion-Scale Approximate Nearest Neighbor Search using a Single GPUcited by 14
  36. 2024AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Executioncited by 14
  37. 2023Adapting LLMs for Efficient, Personalized Information Retrieval: Methods and Implicationscited by 8
  38. 2023ESPN: Memory-Efficient Multi-Vector Information Retrievalcited by 7

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
  2. 2026Pancake: Hierarchical Memory System for Multi-Agent LLM Servingcited by 5
  3. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAGcited by 3
  4. 2025HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Servingcited by 3
  5. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2
  6. 2026SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariancecited by 1
  7. 2025Harmonia: End-to-End RAG Serving Optimizationcited by 0
  8. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  9. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems
  2. 2024JORA: JAX Tensor-Parallel LoRA Library for Retrieval Augmented Fine-Tuning
  3. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  4. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  5. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU
  6. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  7. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  8. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding