# Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning

arXiv: https://arxiv.org/abs/2502.02770 · HTML: https://mlsanity.com/paper/2502.02770/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 64 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 30 of 51)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://mlsanity.com/paper/2407.11550/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)

## Built on by (most cited first, 30 of 49)

- [Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation](https://mlsanity.com/paper/2505.18875/) (2025)
- [SeerAttention-R: Sparse Attention Adaptation for Long Reasoning](https://mlsanity.com/paper/2506.08889/) (2025)
- [The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs](https://mlsanity.com/paper/2504.17768/) (2025)
- [SLA2: Sparse-Linear Attention with Learnable Routing and QAT](https://mlsanity.com/paper/2602.12675/) (2026)
- [SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning](https://mlsanity.com/paper/2602.13515/) (2026)
- [FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference](https://mlsanity.com/paper/2505.13109/) (2025)
- [Kinetics: Rethinking Test-Time Scaling Laws](https://mlsanity.com/paper/2506.05333/) (2025)
- [HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention](https://mlsanity.com/paper/2603.28458/) (2026)
- [Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding](https://mlsanity.com/paper/2512.01278/) (2025)
- [Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification](https://mlsanity.com/paper/2607.24027/) (2026)
- [Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention](https://mlsanity.com/paper/2602.07223/) (2026)
- [Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs](https://mlsanity.com/paper/2602.05191/) (2026)
- [SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers](https://mlsanity.com/paper/2606.22874/) (2026)
- [KV Admission: Learning What to Write for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2512.17452/) (2025)
- [Runtime-Certified Bounded-Error Quantized Attention](https://mlsanity.com/paper/2605.20868/) (2026)
- [Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding](https://mlsanity.com/paper/2609.00097/) (2026)
- [Sparsity Forcing: Reinforcing Token Sparsity of MLLMs](https://mlsanity.com/paper/2504.18579/) (2025)
- [A Preliminary Study on the Promises and Challenges of Native Top-k Sparse Attention](https://mlsanity.com/paper/2512.03494/) (2025)
- [Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models](https://mlsanity.com/paper/2512.14661/) (2025)
- [SPLA: Block Sparse Plus Linear Attention for Long Context Modeling](https://mlsanity.com/paper/2601.22379/) (2026)
- [OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs](https://mlsanity.com/paper/2511.12201/) (2025)
- [ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs](https://mlsanity.com/paper/2602.07721/) (2026)
- [MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM](https://mlsanity.com/paper/2602.14209/) (2026)
- [RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference](https://mlsanity.com/paper/2602.18196/) (2026)
- [VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference](https://mlsanity.com/paper/2603.29494/) (2026)
- [HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression](https://mlsanity.com/paper/2606.28831/) (2026)
- [RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference](https://mlsanity.com/paper/2606.31519/) (2026)
- [KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression](https://mlsanity.com/paper/2607.01237/) (2026)
- [SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking](https://mlsanity.com/paper/2609.13141/) (2026)
- [ReFreeKV: Towards Threshold-Free KV Cache Compression](https://mlsanity.com/paper/2502.16886/) (2025)

## Abstract (arXiv)

Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been a hot research topic. However, current algorithms such as sparse attention or key-value (KV) cache compression tend to use a fixed budget, which presents a significant challenge during deployment because it fails to account for the dynamic nature of real-world scenarios, where the optimal balance between accuracy and efficiency can vary greatly. In this paper, we find that borrowing top-p sampling (nucleus sampling) to sparse attention can surprisingly achieve adaptive budgeting. Based on this, we propose Twilight, a framework to bring adaptive sparsity to any existing sparse attention algorithm without sacrificing their accuracy. Empirical results show that Twilight can adaptively prune at most 98% of redundant tokens, leading to 15.4× acceleration in self-attention operations and 3.9× acceleration in end-to-end per token latency in long context LLM decoding.