# CacheClip: Accelerating RAG with Effective KV Cache Reuse

arXiv: https://arxiv.org/abs/2510.10129 · HTML: https://mlsanity.com/paper/2510.10129/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)
- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [Qwen2.5-1M Technical Report](https://mlsanity.com/paper/2501.15383/) (2025)
- [On The Computational Complexity of Self-Attention](https://mlsanity.com/paper/2209.04881/) (2022)
- [Scatterbrain: Unifying Sparse and Low-rank Attention Approximation](https://mlsanity.com/paper/2110.15343/) (2021)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.12457/) (2024)
- [Reducing hallucination in structured outputs via Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.08189/) (2024)
- [LLM4SR: A Survey on Large Language Models for Scientific Research](https://mlsanity.com/paper/2501.04306/) (2025)
- [KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse](https://mlsanity.com/paper/2502.16002/) (2025)
- [TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text](https://mlsanity.com/paper/2410.07590/) (2024)
- [Dated Data: Tracing Knowledge Cutoffs in Large Language Models](https://mlsanity.com/paper/2403.12958/) (2024)
- [Knowledge Boundary of Large Language Models: A Survey](https://mlsanity.com/paper/2412.12472/) (2024)

## Built on by (most cited first, 16 of 16)

- [ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation](https://mlsanity.com/paper/2602.02579/) (2026)
- [MEPIC: Memory Efficient Position Independent Caching for LLM Serving](https://mlsanity.com/paper/2512.16822/) (2025)
- [Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation](https://mlsanity.com/paper/2604.10098/) (2026)
- [KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs](https://mlsanity.com/paper/2604.13226/) (2026)
- [HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching](https://mlsanity.com/paper/2607.01299/) (2026)
- [An experimental study of KV cache reuse strategies in chunk-level caching systems](https://mlsanity.com/paper/2603.20218/) (2026)
- [Adaptive KV Cache Reuse for Fast Long-Context LLM Serving](https://mlsanity.com/paper/2605.24022/) (2026)
- [Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse](https://mlsanity.com/paper/2606.23581/) (2026)
- [LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs](https://mlsanity.com/paper/2608.11231/) (2026)
- [SemPIC: Learning Semantic Position-Independent KV Caches](https://mlsanity.com/paper/2607.28069/) (2026)
- [Can I Buy Your KV Cache?](https://mlsanity.com/paper/2606.13361/) (2026)
- [Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?](https://mlsanity.com/paper/2609.09768/) (2026)
- [Evaluating the accuracy of KV cache reuse techniques](https://mlsanity.com/paper/2609.31415/) (2026)
- [CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion](https://mlsanity.com/paper/2609.35139/) (2026)
- [ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptation](https://mlsanity.com/paper/2609.36722/) (2026)
- [Budgeted Cache Repair for Cross-Context KV-Cache Reuse](https://mlsanity.com/paper/2610.02233/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems suffer from severe time-to-first-token (TTFT) bottlenecks due to long input sequences. Existing KV cache reuse methods face a fundamental trade-off: prefix caching requires identical prefixes that rarely occur in RAG scenarios, while direct precomputation sacrifices quality due to missing inter-chunk attention and repeated attention sinks. Recent methods like APE and CacheBlend partially address these issues but remain inadequate for robust RAG applications. This paper presents CacheClip, a novel framework that achieves both fast TTFT and high generation quality. Our key insight is that small auxiliary LLMs exhibit similar last-layer attention distributions to primary LLMs (the target model for generation), enabling efficient identification of tokens critical for restoring inter-chunk attention, thereby significantly improving response quality on cross-chunk reasoning tasks. CacheClip integrates four techniques: (1) auxiliary-model-guided token selection for selective KV cache recomputation, (2) shared prefixes to eliminate redundant attention sinks, (3) a sliding-window grouping strategy to maintain local coherence during partial KV cache updates, and (4) a CPU-GPU hybrid design that offloads auxiliary model inference to idle CPU resources, avoiding additional GPU overhead. The recomputation ratio is adjustable, allowing users to flexibly balance efficiency and quality for different deployment requirements. Experiments show CacheClip retains up to 85.2% and 91.1% of full-attention performance on NIAH and LongBench, outperforming CacheBlend and APE by 16.1 and 12.8 points on NIAH, and by 4.5 and 4.2 points on LongBench (with recomp% = 20%). Meanwhile, CacheClip accelerates LLM inference by up to 3.33× in prefill time (with recomp% = 20%), providing a practical solution to the efficiency-quality trade-off in RAG systems.