mlsanityData to 7 Oct 2026

KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models

arXiv 2608.09412 ↗First posted 10 Aug 2026cs.AIcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why. We present KVDiagnosis, a diagnostic dataset and benchmark with three contributions. First, a 25-method taxonomy groups methods into five mechanism families and links them to eight verified implementations and their valid diagnostic measurements. Second, for every supported method setting, we evaluate all sources in each fixed split against a per-source FullCache control before selecting FullCache-correct/compressed-wrong (C-to-W) rows separately for each method-setting, so no compressor defines another's test set. Third, a common record format links paired outputs and run metadata to cache, likelihood, attention, and decoding measurements with explicit applicability states. On Qwen3-8B, four evidence-aware workloads yield 59 800 supported compressed runs over 2600 sources and 12 520 C-to-W rows. Under fixed diagnostic rules, 63.2% have low or partial measured/projected coverage. Only 19 rows (0.2%) combine high measured/projected coverage with strong likelihood drift; another 2,126 (17.0%) preserve structural position addressability, for which representation fidelity remains unknown, while showing the same drift. Against C-to-C success controls, all ten diagnostics separate failed from successful compression (stratified AUROC 0.684-0.871). Among 96 reproducible low-EAR failures, a controlled 4x evidence-attention boost repairs 29.2%, versus 6.3% under a count-matched sham intervention and 3.3% degradation on matched C-to-C controls. Code and data are available at https://github.com/ChosenQC/KVDiagnosis.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 50 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

36 of 50 references.

Closest ancestors · 24

Same direction, newest first

  1. 2026Exploring a Layer-Wise Design Space for KV Cache Eviction13 Jun
  2. 2026Training Transformers for KV Cache Compressibility7 May
  3. 2026CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference24 Feb
  4. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs5 Feb
  5. 2025SALS: Sparse Attention in Latent Space for KV cache Compression28 Oct
  6. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  7. 2025The Pitfalls of KV Cache Compression30 Sep
  8. 2025Inference-Time Hyper-Scaling with KV Cache Compression5 Jun
  9. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  10. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  11. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  12. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
12 more
  1. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  3. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  4. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  6. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  7. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct

Other references · 12

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  8. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  10. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  11. 2025HiBench: Benchmarking LLMs Capability on Hierarchical Structure Reasoningcited by 12
  12. 2025VFLAIR-LLM: A Comprehensive Framework and Benchmark for Split Learning of LLMscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Behavior-Preserving KV Cache Compression
  2. 2026More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression
  3. 2026Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
  4. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
  5. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
  6. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  7. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  8. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache