mlsanityData to 14 Oct 2026

Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

arXiv 2502.08363 ↗First posted 12 Feb 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

We present Top-Theta (Top-θ) Attention, a training-free method for sparsifying transformer attention during inference. Our key insight is that static, per-head thresholds can be calibrated to retain the desired constant number of significant elements per attention row. This approach enables content-based sparsity without retraining, and it remains robust across data domains. We further introduce compensation techniques to preserve accuracy under aggressive sparsification, establishing attention thresholding as a practical and principled alternative to top-k attention. We provide extensive evaluation on natural language processing tasks, showing that Top-θ achieves 3-10x reduction in V-cache usage and up to 10x fewer attention elements during inference while degrading no more than 1% in accuracy.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 39 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

33 of 39 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
21 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  2. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  3. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  4. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  5. 2019Sparse Sequence-to-Sequence Modelscited by 170
  6. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  7. 2021Learned Token Pruning for Transformerscited by 139
  8. 2022On The Computational Complexity of Self-Attentioncited by 139
  9. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  10. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  11. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
  12. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  13. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  14. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  15. 2021Memory-efficient Transformers via Top-k Attentioncited by 60
  16. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 52
  17. 2021Sparse Attention with Linear Unitscited by 41
  18. 2021Energon: Towards Efficient Acceleration of Transformers Using Dynamic Sparse Attentioncited by 26
  19. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  20. 2023Linear Log-Normal Attention with Unbiased Concentrationcited by 11
  21. 2019StreamBox-HBM: Stream Analytics on High Bandwidth Hybrid Memorycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Power Law Guided Dynamic Sifting for Efficient Attentioncited by 1
  2. 2026MassAlloc Attention: Let Attention Allocate Its Own Computecited by 0