mlsanityData to 7 Oct 2026

Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

arXiv 2609.33889 ↗First posted 27 Sep 2026cs.LG cs.CL cs.PFcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

At each step, decoding one sequence with a large language model rereads the projection weights, whose traffic is fixed, and the key-value (KV) cache, whose traffic grows with context. Activation sparsity trims the first term and KV-cache sparsity the second, yet their reported speedups are hard to compare because each depends on context length and on the dense attention kernel it is measured against. We derive a byte crossover, the context length at which the two savings are equal, together with ideal speedup bounds for each branch and for their composition, from model dimensions and keep ratios alone. We then time both branches and their composition from 2K to 128K tokens on two GPUs after a dense prefill of real text, with dense and sparse modes reading the cache through the same split-K attention kernel. The projection branch leads at short context and the KV branch at long context, with speedups that follow their byte bounds up to fixed kernel costs. Adding these costs, measured in separate sweeps, lets the byte account predict the measured crossings of three keep-ratio pairs, a second model, and a second GPU to within 4.1K tokens. Timing the dense baseline with masked instead of split-K attention inflates the apparent speedup of the same KV policy about fivefold. An attention-scored KV selection answers the same passkey and multi-key placements as dense decoding up to 127K tokens, whereas a KV window misses most of them. Under matched perplexity budgets, activation sparsity composed with this selection decodes 14 to 26% faster than the best single branch on both GPUs. Code is available at https://github.com/js-lee-AI/ByteCross.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 51 of its 59 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

51 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 44

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  12. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
32 more
  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  3. 2022Efficiently Scaling Transformer Inferencecited by 529
  4. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  5. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  6. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  7. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  8. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  9. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  10. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  11. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  12. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  13. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  14. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  15. 2024Transformers are Multi-State RNNscited by 120
  16. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  17. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  18. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  19. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  20. 2024Faster Cascades via Speculative Decodingcited by 38
  21. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
  22. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 24
  23. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
  24. 2024ShadowLLM: Predictor-based Contextual Sparsity for Large Language Modelscited by 13
  25. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  26. 2026DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Modelscited by 7
  27. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
  28. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 5
  29. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 5
  30. 2025CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMscited by 4
  31. 2026KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problemcited by 4
  32. 2026To Isolate or to Score? Model-Adaptive Assessment for Cost-Efficient Multi-Agent RAGcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  2. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  3. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  4. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
  5. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
  6. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  7. 2026HieraSparse: Hierarchical Semi-Structured Sparse KV Attention
  8. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch