mlsanityData to 14 Oct 2026

CacheClip: Accelerating RAG with Effective KV Cache Reuse

arXiv 2510.10129 ↗First posted 11 Oct 2025cs.LG cs.AIcited by 16

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems suffer from severe time-to-first-token (TTFT) bottlenecks due to long input sequences. Existing KV cache reuse methods face a fundamental trade-off: prefix caching requires identical prefixes that rarely occur in RAG scenarios, while direct precomputation sacrifices quality due to missing inter-chunk attention and repeated attention sinks. Recent methods like APE and CacheBlend partially address these issues but remain inadequate for robust RAG applications. This paper presents CacheClip, a novel framework that achieves both fast TTFT and high generation quality. Our key insight is that small auxiliary LLMs exhibit similar last-layer attention distributions to primary LLMs (the target model for generation), enabling efficient identification of tokens critical for restoring inter-chunk attention, thereby significantly improving response quality on cross-chunk reasoning tasks. CacheClip integrates four techniques: (1) auxiliary-model-guided token selection for selective KV cache recomputation, (2) shared prefixes to eliminate redundant attention sinks, (3) a sliding-window grouping strategy to maintain local coherence during partial KV cache updates, and (4) a CPU-GPU hybrid design that offloads auxiliary model inference to idle CPU resources, avoiding additional GPU overhead. The recomputation ratio is adjustable, allowing users to flexibly balance efficiency and quality for different deployment requirements. Experiments show CacheClip retains up to 85.2% and 91.1% of full-attention performance on NIAH and LongBench, outperforming CacheBlend and APE by 16.1 and 12.8 points on NIAH, and by 4.5 and 4.2 points on LongBench (with recomp% = 20%). Meanwhile, CacheClip accelerates LLM inference by up to 3.33× in prefill time (with recomp% = 20%), providing a practical solution to the efficiency-quality trade-off in RAG systems.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 37 of its 45 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

37 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation5 Feb
  2. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text10 Oct
  3. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation18 Apr
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  12. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
21 more
  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  3. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  5. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  6. 2023Large Language Models in Finance: A Surveycited by 226
  7. 2025Qwen2.5-1M Technical Reportcited by 164
  8. 2022On The Computational Complexity of Self-Attentioncited by 139
  9. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  10. 2023Large language models in healthcare and medical domain: A reviewcited by 127
  11. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 93
  12. 2025LLM4SR: A Survey on Large Language Models for Scientific Researchcited by 83
  13. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  14. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 51
  15. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  16. 2025APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encodingcited by 29
  17. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  18. 2024LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Modelscited by 17
  19. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inferencecited by 15
  20. 2020On the Importance of Local Information in Transformer Based Modelscited by 3
  21. 2025SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 15.

16 papers.

Sort:

By area:

  1. 2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generationcited by 13
  2. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 11
  3. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11
  4. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
  5. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
  6. 2026An experimental study of KV cache reuse strategies in chunk-level caching systemscited by 5
  7. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 3
  8. 2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
  9. 2026LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMscited by 3
  10. 2026SemPIC: Learning Semantic Position-Independent KV Cachescited by 2
  11. 2026Can I Buy Your KV Cache?cited by 0
  12. 2026Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?cited by 0
4 more
  1. 2026Evaluating the accuracy of KV cache reuse techniquescited by 0
  2. 2026CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusioncited by 0
  3. 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0
  4. 2026Budgeted Cache Repair for Cross-Context KV-Cache Reusecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
  2. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
  4. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  5. 2025HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse
  6. 2026Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?
  7. 2026Decoupled Attention Fusion: Accelerating RAG with Efficient KV Cache Reuse
  8. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decoding