mlsanityData to 7 Oct 2026

Beyond Sparse Weights: When Is Attention Compressible?

arXiv 2608.21541 ↗First posted 21 Aug 2026cs.LG cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV-cache compression is often justified by attention maps with a few large weights. This is incomplete: large weights may not contain most of the mass, omitted values can cancel, and preserving the attention output may not preserve the task. We separate these questions. Global score gaps -- not threshold counts -- determine how many tokens are needed to retain a target mass. For a realized row, the weighted sum of omitted values is the exact missing statistic. A controlled retrieval--aggregation model explains when truncation helps and when it hurts. These results motivate CertKV, a training-free compressor that reserves one tail-summary slot per head and allocates the rest by value dispersion. Under matched budgets, CertKV is top-two in seven of nine LongBench-v2 settings, remains in the leading compressed tier on 128K RULER, and realizes a ten-fold cache budget in a packed Llama prototype. Compressibility depends on the mass, values, future queries, and task -- not on a sparse-looking map alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 67 of its 80 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

67 of 80 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning8 Aug
  2. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference14 Apr
  3. 2025WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models3 Mar
  4. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolation20 Feb
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  6. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  7. 2024When Attention Sink Emerges in Language Models: An Empirical View14 Oct
  8. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  9. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models18 Jun
  10. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  12. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
6 more
  1. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  6. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 49

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
37 more
  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  6. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  7. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  8. 2020Hopfield Networks is All You Needcited by 461
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  10. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  12. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  13. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  14. 2024Massive Activations in Large Language Modelscited by 277
  15. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  16. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  17. 2019Sparse Sequence-to-Sequence Modelscited by 167
  18. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  19. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  20. 2024Transformers are Multi-State RNNscited by 120
  21. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  22. 2023Fast Attention Requires Bounded Entriescited by 102
  23. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  24. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  25. 2023KDEformer: Accelerating Transformers via Kernel Density Estimationcited by 58
  26. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  27. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  28. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  29. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  30. 2025Scalable-Softmax Is Superior for Attentioncited by 38
  31. 2023Algorithm and Hardness for Dynamic Attention Maintenance in Large Language Modelscited by 37
  32. 2024The Fine-Grained Complexity of Gradient Computation for Training Large Language Modelscited by 30
  33. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  34. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 23
  35. 2024How Sparse Attention Approximates Exact Attention? Your Attention is Naturally nC-Sparsecited by 16
  36. 2023One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Spacecited by 6
  37. 2026Focus and Dilution: The Multi-stage Learning Process of Attentioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Attention-Weighted Value Projection for KV-Cache Compression
  2. 2026Trust the Mass: Forced Weights in KV-Cache Eviction
  3. 2026Small Frequency Corrections Can Change What Survives KV Cache Compression
  4. 2026VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch
  5. 2025EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
  6. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
  7. 2026Beyond Mean Attention: Diversity-Aware, Layer-Wise Scoring for KV Cache Eviction
  8. 2026Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation