# Sparse and Continuous Attention Mechanisms

arXiv: https://arxiv.org/abs/2006.07214 · HTML: https://mlsanity.com/paper/2006.07214/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 30 of 39)

- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Uncovering mesa-optimization algorithms in Transformers](https://mlsanity.com/paper/2309.05858/) (2023)
- [The Transient Nature of Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2311.08360/) (2023)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers](https://mlsanity.com/paper/2305.15805/) (2023)
- [MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer](https://mlsanity.com/paper/2305.00355/) (2023)
- [Continuum Attention for Neural Operators](https://mlsanity.com/paper/2406.06486/) (2024)
- [HRVQA: A Visual Question Answering Benchmark for High-Resolution Aerial Images](https://mlsanity.com/paper/2301.09460/) (2023)
- [Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literature](https://mlsanity.com/paper/2305.11033/) (2023)
- [Sparse Continuous Distributions and Fenchel-Young Losses](https://mlsanity.com/paper/2108.01988/) (2021)
- [SPANet: Frequency-balancing Token Mixer using Spectral Pooling Aggregation Modulation](https://mlsanity.com/paper/2308.11568/) (2023)
- [∞-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation](https://mlsanity.com/paper/2501.19098/) (2025)
- [IDF: Iterative Dynamic Filtering Networks for Generalizable Image Denoising](https://mlsanity.com/paper/2508.19649/) (2025)
- [Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2203.04421/) (2022)
- [ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models](https://mlsanity.com/paper/2403.20262/) (2024)
- [∞-former: Infinite Memory Transformer](https://mlsanity.com/paper/2109.00301/) (2021)
- [Quasi-optimal Reinforcement Learning with Continuous Actions](https://mlsanity.com/paper/2301.08940/) (2023)
- [Multimodal Continuous Visual Attention Mechanisms](https://mlsanity.com/paper/2104.03046/) (2021)
- [Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost](https://mlsanity.com/paper/2210.15541/) (2022)
- [Reranking Laws for Language Generation: A Communication-Theoretic Perspective](https://mlsanity.com/paper/2409.07131/) (2024)
- [Learning Hierarchical Relational Representations through Relational Convolutions](https://mlsanity.com/paper/2310.03240/) (2023)
- [Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models](https://mlsanity.com/paper/2412.07171/) (2024)
- [Detect the Interactions that Matter in Matter: Geometric Attention for Many-Body Systems](https://mlsanity.com/paper/2106.02549/) (2021)
- [Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More](https://mlsanity.com/paper/2502.07490/) (2025)
- [Improving the Efficiency of Long Document Classification using Sentence Ranking Approach](https://mlsanity.com/paper/2506.07248/) (2025)
- [LEDGER: Scaling Agentic Document Editing with Dependency-aware Graph Retrieval](https://mlsanity.com/paper/2606.28379/) (2026)
- [Perspectives on Tsallis Statistics for Artificial Intelligence](https://mlsanity.com/paper/2608.01223/) (2026)
- [DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence](https://mlsanity.com/paper/2110.04821/) (2021)
- [Kernel Deformed Exponential Families for Sparse Continuous Attention](https://mlsanity.com/paper/2111.01222/) (2021)
- [Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC](https://mlsanity.com/paper/2306.06345/) (2023)

## Abstract (arXiv)

Exponential families are widely used in machine learning; they include many distributions in continuous and discrete domains (e.g., Gaussian, Dirichlet, Poisson, and categorical distributions via the softmax transformation). Distributions in each of these families have fixed support. In contrast, for finite domains, there has been recent work on sparse alternatives to softmax (e.g. sparsemax and alpha-entmax), which have varying support, being able to assign zero probability to irrelevant categories. This paper expands that work in two directions: first, we extend alpha-entmax to continuous domains, revealing a link with Tsallis statistics and deformed exponential families. Second, we introduce continuous-domain attention mechanisms, deriving efficient gradient backpropagation algorithms for alpha in {1,2}. Experiments on attention-based text classification, machine translation, and visual question answering illustrate the use of continuous attention in 1D and 2D, showing that it allows attending to time intervals and compact regions.