mlsanityData to 7 Oct 2026

Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning

arXiv 2502.02770 ↗First posted 4 Feb 2025cs.LG cs.CLcited by 48

Not on the map. Why?

Cited by 23 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been a hot research topic. However, current algorithms such as sparse attention or key-value (KV) cache compression tend to use a fixed budget, which presents a significant challenge during deployment because it fails to account for the dynamic nature of real-world scenarios, where the optimal balance between accuracy and efficiency can vary greatly. In this paper, we find that borrowing top-p sampling (nucleus sampling) to sparse attention can surprisingly achieve adaptive budgeting. Based on this, we propose Twilight, a framework to bring adaptive sparsity to any existing sparse attention algorithm without sacrificing their accuracy. Empirical results show that Twilight can adaptively prune at most 98% of redundant tokens, leading to 15.4× acceleration in self-attention operations and 3.9× acceleration in end-to-end per token latency in long context LLM decoding.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 51 of its 64 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

51 of 64 references.

References · 51

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
38 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  6. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  7. 2018CoQA: A Conversational Question Answering Challengecited by 674
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  10. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  11. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  12. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  13. 2021Efficient Attentions for Long Document Summarizationcited by 342
  14. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  15. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  16. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  17. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  18. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  19. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  20. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  21. 2025Qwen2.5-1M Technical Reportcited by 158
  22. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  23. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  24. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  25. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  26. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  27. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  28. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  29. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  30. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  31. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  32. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  33. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  34. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  35. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  36. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  37. 2024TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attentioncited by 28
  38. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 24

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 35.

48 papers.

Sort:

By area:

  1. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  2. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  3. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  4. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 21
  5. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 16
  6. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  7. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  8. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 10
  9. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  10. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 6
  11. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  12. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 5
36 more
  1. 2026Runtime-Certified Bounded-Error Quantized Attentioncited by 4
  2. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformerscited by 4
  3. 2026Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decodingcited by 4
  4. 2025Sparsity Forcing: Reinforcing Token Sparsity of MLLMscited by 3
  5. 2025A Preliminary Study on the Promises and Challenges of Native Top-k Sparse Attentioncited by 3
  6. 2025Focus: A Streaming Concentration Architecture for Efficient Vision-Language Modelscited by 3
  7. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  8. 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modelingcited by 3
  9. 2026RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inferencecited by 3
  10. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMscited by 2
  11. 2026RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inferencecited by 2
  12. 2026VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inferencecited by 2
  13. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compressioncited by 2
  14. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  15. 2025ReFreeKV: Towards Threshold-Free KV Cache Compressioncited by 1
  16. 2025SpecAttn: Speculating Sparse Attentioncited by 1
  17. 2025OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMscited by 1
  18. 2026MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLMcited by 1
  19. 2026ART: Adaptive Relational Transformer for Pedestrian Trajectory Prediction with Temporal-Aware Relationscited by 1
  20. 2026AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attentioncited by 1
  21. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  22. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cachecited by 1
  23. 2026Vortex: Efficient and Programmable Sparse Attention Serving for AI Agentscited by 1
  24. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  25. 2026Trust the Mass: Forced Weights in KV-Cache Evictioncited by 1
  26. 2026LoGo: Token-Level Dynamic Local-Global Attentioncited by 1
  27. 2026SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attentioncited by 0
  28. 2026Long Context Pre-Training with Lighthouse Attentioncited by 0
  29. 2026EntmaxKV: Support-Aware Decoding for Entmax Attentioncited by 0
  30. 2026Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsitycited by 0
  31. 2026Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understandingcited by 0
  32. 2026CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routingcited by 0
  33. 2026SAS: Simple Attention Sparsification via End-to-End Optimization of Context Rankingcited by 0
  34. 2026MassAlloc Attention: Let Attention Allocate Its Own Computecited by 0
  35. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0
  36. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0