mlsanityData to 14 Oct 2026

Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression

arXiv 2502.01941 ↗First posted 4 Feb 2025cs.CL cs.AIcited by 16

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

While Key-Value (KV) cache compression is essential for efficient LLM inference, current evaluations disproportionately focus on sparse retrieval tasks, potentially masking the degradation of High-Density Reasoning where Chain-of-Thought (CoT) coherence is critical. We introduce KVFundaBench to systematically evaluate this gap, revealing a sharp dichotomy: while retrieval tasks remain robust, reasoning tasks exhibit severe Task-Dependent Degradation under aggressive compression due to disrupted CoT links. Extending our analysis to the DeepSeek-R1 model, we uncover that its specialized attention patterns offer unique insights into the fragility of reasoning chains. Guided by these findings -- specifically the necessity of preserving few-shot examples as indivisible Semantic Units -- we propose ShotKV. This approach explicitly separates prefill and decoding phases to prioritize semantic integrity. Empirical results demonstrate that ShotKV achieves 9%-18% accuracy improvements on long-context generation tasks and effectively generalizes to document QA, all while delivering an 11% latency reduction compared to full cache inference.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 70 of its 101 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

70 of 101 references.

References · 70

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023Mistral 7Bcited by 3,164
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  13. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  14. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  15. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  16. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  17. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  18. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  19. 2022Efficiently Scaling Transformer Inferencecited by 539
  20. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  21. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  22. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  23. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  24. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  25. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  26. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  27. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  28. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  29. 2023Adapting Language Models to Compress Contextscited by 356
  30. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  31. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  32. 2022UL2: Unifying Language Learning Paradigmscited by 295
  33. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  34. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  35. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  36. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  37. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  38. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  2. 2025The Pitfalls of KV Cache Compressioncited by 15
  3. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  4. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  5. 2025DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inferencecited by 11
  6. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 10
  7. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Managementcited by 6
  8. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Sizecited by 4
  9. 2025AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Modelscited by 3
  10. 2026SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Cachingcited by 3
  11. 2026ST-Lite: Training-Free KV Cache Compression with Spatio-Trajectory Guidance for Long-Horizon GUI Agentscited by 2
  12. 2026NestedKV: Nested Memory Routing for Long-Context KV Cache Compressioncited by 2
4 more
  1. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 2
  2. 2025Reasoning Language Model Inference Serving Unveiled: An Empirical Studycited by 1
  3. 2026Small Frequency Corrections Can Change What Survives KV Cache Compressioncited by 0
  4. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
  2. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
  3. 2026Information-Aware KV Cache Compression for Long Reasoning
  4. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
  5. 2025Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoning
  6. 2026BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
  7. 2026CASK: Core-Aware Selective KV Compression for Reasoning Traces
  8. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference