mlsanityData to 14 Oct 2026

Evaluating the accuracy of KV cache reuse techniques

arXiv 2609.31415 ↗First posted 25 Sep 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show that current evaluations of KV cache reuse techniques rely on measurements that fail to faithfully capture the loss of accuracy attributable to reuse, often artificially inflating the reported effectiveness. We also show that existing datasets do not exhibit the reuse dynamics needed to thoroughly evaluate such techniques. To address these issues, we propose an evaluation methodology that measures this accuracy loss without ambiguity and we introduce Boxoffice, a tool that programmatically generates evaluation datasets that exercise challenging KV cache reuse patterns.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 58 of its 66 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

58 of 66 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026An experimental study of KV cache reuse strategies in chunk-level caching systems3 Mar
  2. 2026ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation31 Jan
  3. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation19 Jan
  4. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Serving18 Dec
  5. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving16 Dec
  6. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving13 Nov
  7. 2025CacheClip: Accelerating RAG with Effective KV Cache Reuse11 Oct
  8. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  9. 2025Sparse Attention across Multiple-context KV Cache6 Aug
  10. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse21 Feb
  11. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation5 Feb
  12. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text10 Oct
3 more
  1. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2023Mistral 7Bcited by 3,164
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
31 more
  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  5. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  6. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  7. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  8. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  10. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  11. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  12. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  13. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  14. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  15. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  16. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  17. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  18. 2022Data Contamination: From Memorization to Exploitationcited by 236
  19. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  20. 2020Characterising Bias in Compressed Modelscited by 167
  21. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  22. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 40
  23. 2024Block-Attention for Efficient Prefillingcited by 37
  24. 2025APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encodingcited by 29
  25. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  26. 2025Benchmarking Deep Search over Heterogeneous Enterprise Datacited by 23
  27. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  28. 2024When Context Leads but Parametric Memory Follows in Large Language Modelscited by 11
  29. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  30. 2026You Need an Encoder for Native Position-Independent Cachingcited by 6
  31. 2025RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systemscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse
  2. 2026Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
  3. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving
  4. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
  5. 2026Probing the Prompt KV Cache: Where It Becomes Dispensable
  6. 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
  7. 2026RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse
  8. 2026KV Cache Offloading for Context-Intensive Tasks