mlsanityData to 7 Oct 2026

TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text

arXiv 2410.07590 ↗First posted 10 Oct 2024cs.CV cs.CLcited by 60

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

Current Retrieval-Augmented Generation (RAG) systems concatenate and process numerous retrieved document chunks for prefill which requires a large volume of computation, therefore leading to significant latency in time-to-first-token (TTFT). To reduce the computation overhead as well as TTFT, we introduce TurboRAG, a novel RAG system that redesigns the inference paradigm of the current RAG system by first pre-computing and storing the key-value (KV) caches of documents offline, and then directly retrieving the saved KV cache for prefill. Hence, online computation of KV caches is eliminated during inference. In addition, we provide a number of insights into the mask matrix and positional embedding mechanisms, plus fine-tune a pretrained language model to maintain model accuracy of TurboRAG. Our approach is applicable to most existing large language models and their applications without any requirement in modification of models and inference systems. Experimental results across a suite of RAG benchmarks demonstrate that TurboRAG reduces TTFT by up to 9.4x compared to the conventional RAG systems (on an average of 8.6x), but reserving comparable performance to the standard RAG systems.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 49 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

35 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Block-Attention for Efficient Prefilling14 Sep
  2. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection25 May
  3. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation18 Apr
  4. 2023In-Context Retrieval-Augmented Language Models31 Jan
  5. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP28 Dec
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories20 Dec
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  12. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
16 more
  1. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  6. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  7. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  9. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  10. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  11. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  12. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  13. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
  14. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28
  15. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  16. 2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 21; 2026: 35.

60 papers.

Sort:

By area:

  1. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  2. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
  3. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  4. 2025CacheClip: Accelerating RAG with Effective KV Cache Reusecited by 14
  5. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  6. 2024Improving Factuality with Explicit Working Memorycited by 10
  7. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  8. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 9
  9. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  10. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
  11. 2025Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlapscited by 6
  12. 2026You Need an Encoder for Native Position-Independent Cachingcited by 6
38 more
  1. 2025Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agentscited by 5
  2. 2025ImpRAG: Retrieval-Augmented Generation with Implicit Queriescited by 5
  3. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
  4. 2026An experimental study of KV cache reuse strategies in chunk-level caching systemscited by 5
  5. 2025OSCAR: Online Soft Compression And Rerankingcited by 4
  6. 2025Log-Augmented Generation: Scaling Test-Time Reasoning with Reusable Computationcited by 4
  7. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  8. 2026KV Cache Translation across Heterogeneous Large Language Modelscited by 4
  9. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generationcited by 3
  10. 2025Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMscited by 3
  11. 2026When Knowledge Is Not Free: Cost-Aware Evidence Selection in Retrieval-Augmented Generationcited by 3
  12. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inferencecited by 3
  13. 2025RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systemscited by 2
  14. 2025MatKV: Trading Compute for Flash Storage in LLM Inferencecited by 2
  15. 2026On the Effectiveness of Proposed Techniques to Reduce Energy Consumption in RAG Systems: A Controlled Experimentcited by 2
  16. 2026When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judgescited by 2
  17. 2026TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQLcited by 2
  18. 2026CSAttention: Centroid-Scoring Attention for Accelerating LLM Inferencecited by 2
  19. 2026SemPIC: Learning Semantic Position-Independent KV Cachescited by 2
  20. 2025SubGCache: Accelerating Graph-based RAG with Subgraph-level KV Cachecited by 1
  21. 2026Parallel Context-of-Experts Decoding for Retrieval Augmented Generationcited by 1
  22. 2026InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Contextcited by 1
  23. 2026EnComp: Lightweight Encoder-Only Context Compression for Retrieval-Augmented Question Answeringcited by 1
  24. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-designcited by 1
  25. 2026Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?cited by 1
  26. 2026LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encodingcited by 1
  27. 2026SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariancecited by 1
  28. 2026CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inferencecited by 1
  29. 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
  30. 2025Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integrationcited by 0
  31. 2025PEVLM: Parallel Encoding for Vision-Language Modelscited by 0
  32. 2025Accelerating LLM Inference with Precomputed Query Storagecited by 0
  33. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  34. 2025Democratizing LLM Efficiency: From Hyperscale Optimizations to Universal Deployabilitycited by 0
  35. 2025Accelerating Language Model Workflows with Prompt Choreographycited by 0
  36. 2026HaS: Accelerating RAG through Homology-Aware Speculative Retrievalcited by 0
  37. 2026Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agentscited by 0
  38. 2026KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systemscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse
  2. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
  3. 2024Accelerating Retrieval-Augmented Generation
  4. 2025Less Is More: Elevating RAG via Performance-Driven Context Compression
  5. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  6. 2025FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation
  7. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
  8. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models