mlsanityData to 7 Oct 2026

An experimental study of KV cache reuse strategies in chunk-level caching systems

arXiv 2603.20218 ↗First posted 3 Mar 2026cs.CL cs.LGcited by 5

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precomputing KV caches for these retrieved chunks and reusing them. However, these caches miss cross-attention dependencies between chunks, which can reduce output quality. Several methods try to improve CLC accuracy using different techniques. We make two main contributions. First, we show that existing CLC approaches have fundamental limitations that limit their accuracy or their applicability. We back this conclusion with an extensive CLC system experimental evaluation. Second, we observe that existing CLC techniques are complementary. We leverage this insight to propose a new CLC design that carefully combines them and achieves better accuracy.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 18 of its 27 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

18 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025CacheClip: Accelerating RAG with Effective KV Cache Reuse11 Oct
  2. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation5 Feb
  3. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text10 Oct
  4. 2024Block-Attention for Efficient Prefilling14 Sep

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2023Mistral 7Bcited by 3,065
  4. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  5. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  7. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  8. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  9. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  10. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  11. 2024DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Servingcited by 30
  12. 2025APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encodingcited by 29
2 more
  1. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  2. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?cited by 0
  2. 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpointscited by 0
  3. 2026Evaluating the accuracy of KV cache reuse techniquescited by 0
  4. 2026Memory as a cache: Exact context reuse and deletion by constructioncited by 0
  5. 2026Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing
  2. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving
  3. 2026Recency/Frequency Adaptive KV Caching for Large Language Model Serving
  4. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving
  5. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
  6. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
  7. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
  8. 2026CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion