# A Regularized Framework for Sparse and Structured Neural Attention

arXiv: https://arxiv.org/abs/1705.07704 · HTML: https://mlsanity.com/paper/1705.07704/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2017.

## Built on by (most cited first, 30 of 38)

- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [Neurons in Large Language Models: Dead, N-gram, Positional](https://mlsanity.com/paper/2309.04827/) (2023)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Sparse Attention with Linear Units](https://mlsanity.com/paper/2104.07012/) (2021)
- [SPECTRA: Sparse Structured Text Rationalization](https://mlsanity.com/paper/2109.04552/) (2021)
- [Attention layers provably solve single-location regression](https://mlsanity.com/paper/2410.01537/) (2024)
- [Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution](https://mlsanity.com/paper/2101.06658/) (2021)
- [Sparse and Structured Hopfield Networks](https://mlsanity.com/paper/2402.13725/) (2024)
- [Towards Self-Supervised Learning of Global and Object-Centric Representations](https://mlsanity.com/paper/2203.05997/) (2022)
- [Regularized Frank-Wolfe for Dense CRFs: Generalizing Mean Field and Beyond](https://mlsanity.com/paper/2110.14759/) (2021)
- [Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2203.04421/) (2022)
- [Learning Self-Modulating Attention in Continuous Time Space with Applications to Sequential Recommendation](https://mlsanity.com/paper/2204.06517/) (2022)
- [Massively Parallel Continuous Local Search for Hybrid SAT Solving on GPUs](https://mlsanity.com/paper/2308.15020/) (2023)
- [Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models](https://mlsanity.com/paper/2110.14182/) (2021)
- [A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search](https://mlsanity.com/paper/2202.07071/) (2022)
- [DataSP: A Differential All-to-All Shortest Path Algorithm for Learning Costs and Predicting Paths with Context](https://mlsanity.com/paper/2405.04923/) (2024)
- [MultiMax: Sparse and Multi-Modal Attention Learning](https://mlsanity.com/paper/2406.01189/) (2024)
- [Give Me More Details: Improving Fact-Checking with Latent Retrieval](https://mlsanity.com/paper/2305.16128/) (2023)
- [Yet Another Representation of Binary Decision Trees: A Mathematical Demonstration](https://mlsanity.com/paper/2101.07077/) (2021)
- [Learning with Latent Structures in Natural Language Processing: A Survey](https://mlsanity.com/paper/2201.00490/) (2022)
- [Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.15176/) (2022)
- [Convolution-enhanced Evolving Attention Networks](https://mlsanity.com/paper/2212.08330/) (2022)
- [Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognition](https://mlsanity.com/paper/2304.01905/) (2023)
- [Learning Hierarchical Relational Representations through Relational Convolutions](https://mlsanity.com/paper/2310.03240/) (2023)
- [MGHF: Multi-Granular High-Frequency Perceptual Loss for Image Super-Resolution](https://mlsanity.com/paper/2411.13548/) (2024)
- [Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators](https://mlsanity.com/paper/2602.08216/) (2026)
- [Modeling Semantic Composition with Syntactic Hypergraph for Video Question Answering](https://mlsanity.com/paper/2205.06530/) (2022)
- [Bregman Conditional Random Fields: Sequence Labeling with Parallelizable Inference Algorithms](https://mlsanity.com/paper/2506.00732/) (2025)
- [Fairness Meets Privacy: Integrating Differential Privacy and Demographic Parity in Multi-class Classification](https://mlsanity.com/paper/2511.18876/) (2025)
- [Geometric Analysis of Token Selection in Multi-Head Attention](https://mlsanity.com/paper/2602.01893/) (2026)

## Abstract (arXiv)

Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real values to probabilities, suitable as an attention mechanism. Our framework includes softmax and a slight generalization of the recently-proposed sparsemax as special cases. However, we also show how our framework can incorporate modern structured penalties, resulting in more interpretable attention mechanisms, that focus on entire segments or groups of an input. We derive efficient algorithms to compute the forward and backward passes of our attention mechanisms, enabling their use in a neural network trained with backpropagation. To showcase their potential as a drop-in replacement for existing ones, we evaluate our attention mechanisms on three large-scale tasks: textual entailment, machine translation, and sentence summarization. Our attention mechanisms improve interpretability without sacrificing performance; notably, on textual entailment and summarization, we outperform the standard attention mechanisms based on softmax and sparsemax.