mlsanityData to 7 Oct 2026

Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution

arXiv 2510.00636 ↗First posted 1 Oct 2025cs.AI cs.CLcited by 70

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Memory consumption of the Key-Value (KV) cache represents a major bottleneck for efficient large language model inference. While attention-score-based KV cache pruning shows promise, it faces critical practical limitations: attention scores from future tokens are unavailable during compression, and modern implementations like Flash Attention do not materialize the full attention matrix, making past scores inaccessible. To overcome these challenges, we introduce Expected Attention, a training-free compression method that estimates KV pairs importance by predicting how future queries will attend to them. Our approach leverages the distributional properties of LLM activations to compute expected attention scores in closed form for each KV pair. These scores enable principled ranking and pruning of KV pairs with minimal impact on the residual stream, achieving effective compression without performance degradation. Importantly, our method operates seamlessly across both prefilling and decoding phases, consistently outperforming state-of-the-art baselines in both scenarios. Finally, we release KVPress, a comprehensive library to enable researchers to implement and benchmark KV cache compression methods, already including more than 20 techniques.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 57 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

47 of 57 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Inference-Time Hyper-Scaling with KV Cache Compression5 Jun
  2. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments21 Apr
  3. 2025Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression4 Mar
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  8. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
1 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
22 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  8. 2024Massive Activations in Large Language Modelscited by 277
  9. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  10. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  11. 2024The Illusion of State in State-Space Modelscited by 181
  12. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  13. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  14. 2024Transformers are Multi-State RNNscited by 120
  15. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  16. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  17. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  18. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  19. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  20. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationcited by 10
  21. 2025Limitations of Normalization in Attention Mechanismcited by 7
  22. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristicscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 67.

70 papers.

Sort:

By area:

  1. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
  2. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 14
  3. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 14
  4. 2026Learning to Evict from Key-Value Cachecited by 12
  5. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 11
  6. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
  7. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 9
  8. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  9. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  10. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  11. 2026WildCat: Near-Linear Attention in Theory and Practicecited by 6
  12. 2026Cartridges at Scale: Training Modular KV Caches over Large Document Collectionscited by 6
38 more
  1. 2026IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inferencecited by 6
  2. 2025Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beaconscited by 5
  3. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  4. 2026Why Attention Patterns Exist: A Unifying Temporal Perspective Analysiscited by 4
  5. 2026Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Evictioncited by 4
  6. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objectivecited by 4
  7. 2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
  8. 2026When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compressioncited by 4
  9. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
  10. 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMscited by 4
  11. 2026End-to-End Context Compression at Scalecited by 4
  12. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  13. 2026SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Cachingcited by 3
  14. 2026Sub-Token Routing for KV Cache Compressioncited by 3
  15. 2026A Simple Plug-in for Improving Eviction-Based KV Cache Compressioncited by 3
  16. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  17. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3
  18. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMscited by 2
  19. 2026NestedKV: Nested Memory Routing for Long-Context KV Cache Compressioncited by 2
  20. 2026Rethinking LoRA Memory Through the Lens of KV Cache Compressioncited by 2
  21. 2026QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Servingcited by 2
  22. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Servingcited by 2
  23. 2026Information-Aware KV Cache Compression for Long Reasoningcited by 2
  24. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  25. 2026Practical Online KV Cache Compaction for LLM Agents: An Empirical Studycited by 2
  26. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Evictioncited by 2
  27. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compressioncited by 2
  28. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transformscited by 2
  29. 2026CodeComp: Structural KV Cache Compression for Agentic Codingcited by 1
  30. 2026Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utilitycited by 1
  31. 2026Adaptive Mass-Segmented KV Compression for Long-Context Reasoningcited by 1
  32. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  33. 2026Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluationcited by 1
  34. 2026TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruningcited by 1
  35. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Modelscited by 1
  36. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  37. 2026Trust the Mass: Forced Weights in KV-Cache Evictioncited by 1
  38. 2026ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compactioncited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
  2. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
  3. 2025SALS: Sparse Attention in Latent Space for KV cache Compression
  4. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
  5. 2026ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
  6. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  7. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  8. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression