Data to 14 Oct 2026

RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation

arXiv 2404.12457 ↗First posted 18 Apr 2024cs.DC cs.CL cs.LGcited by 98

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has shown significant improvements in various natural language processing tasks by integrating the strengths of large language models (LLMs) and external knowledge databases. However, RAG introduces long sequence generation and leads to high computation and memory costs. We propose RAGCache, a novel multilevel dynamic caching system tailored for RAG. Our analysis benchmarks current RAG systems, pinpointing the performance bottleneck (i.e., long sequence due to knowledge injection) and optimization opportunities (i.e., caching knowledge's intermediate states). Based on these insights, we design RAGCache, which organizes the intermediate states of retrieved knowledge in a knowledge tree and caches them in the GPU and host memory hierarchy. RAGCache proposes a replacement policy that is aware of LLM inference characteristics and RAG retrieval patterns. It also dynamically overlaps the retrieval and inference steps to minimize the end-to-end latency. We implement RAGCache and evaluate it on vLLM, a state-of-the-art LLM inference system and Faiss, a state-of-the-art vector database. The experimental results show that RAGCache reduces the time to first token (TTFT) by up to 4x and improves the throughput by up to 2.1x compared to vLLM integrated with Faiss.

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 72 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

38 of 72 references.

Closest ancestors · 4

Same direction, newest first

Other references · 34

2017Attention Is All You Needcited by 60,361
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2020Measuring Massive Multitask Language Understandingcited by 7,963
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
2021Improving language models by retrieving from trillions of tokenscited by 1,293

22 more

2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
2020Towards a Human-like Open-Domain Chatbotcited by 470
2023Benchmarking Large Language Models for News Summarizationcited by 445
2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
2023Fast Distributed Inference Serving for Large Language Modelscited by 145
2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 127
2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 119
2017Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graphcited by 104
2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 34; 2026: 48.

98 papers.

Sort:

By area:

2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 109
2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 101
2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 63
2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47
2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 40
2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 36

38 more

2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 27
2024Compute Or Load KV Cache? Why Not Both?cited by 25
2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
2024Do Large Language Models Need a Content Delivery Network?cited by 16
2025CacheClip: Accelerating RAG with Effective KV Cache Reusecited by 16
2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inferencecited by 15
2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 14
2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
2024MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generationcited by 8
2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8
2025HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reusecited by 8
2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devicescited by 8
2024LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contextscited by 7
2026KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasingcited by 6
2026Models Take Notes at Prefill: KV Cache Can Be Editable and Composablecited by 6
2025Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimizationcited by 5
2025Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Modelscited by 5
2025Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agentscited by 5
2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
2026TS-Memory: Plug-and-Play Memory for Time Series Foundation Modelscited by 5
2026Pancake: Hierarchical Memory System for Multi-Agent LLM Servingcited by 5
2025A Survey of LLM × DATAcited by 4
2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
2026ClawVM: Harness-Managed Virtual Memory for Stateful Tool-Using LLM Agentscited by 4
2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 4
2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
2024FAME: Towards Factual Multi-Task Model Editingcited by 3
2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAGcited by 3
2025Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMscited by 3
2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 3
2024Empower Vision Applications with LoRA LMMcited by 2
2025RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systemscited by 2
2025MatKV: Trading Compute for Flash Storage in LLM Inferencecited by 2
2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way