mlsanityData to 7 Oct 2026

On the Regularity of Attention

arXiv 2102.05628 ↗First posted 10 Feb 2021stat.ML cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Attention is a powerful component of modern neural networks across a wide variety of domains. In this paper, we seek to quantify the regularity (i.e. the amount of smoothness) of the attention operation. To accomplish this goal, we propose a new mathematical framework that uses measure theory and integral operators to model attention. We show that this framework is consistent with the usual definition, and that it captures the essential properties of attention. Then we use this framework to prove that, on compact domains, the attention operation is Lipschitz continuous and provide an estimate of its Lipschitz constant. Additionally, by focusing on a specific type of attention, we extend these Lipschitz continuity results to non-compact domains. We also discuss the effects regularity can have on NLP models, and applications to invertible and infinitely-deep networks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

21 of 38 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017Graph Attention Networkscited by 8,854
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2018Image Transformercited by 770
  8. 2014Neural Turing Machinescited by 630
  9. 2019Deep Equilibrium Modelscited by 603
  10. 2018Invertible Residual Networkscited by 364
  11. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  12. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
9 more
  1. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  2. 2020The Lipschitz Constant of Self-Attentioncited by 171
  3. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  4. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  5. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  6. 2016Understanding Deep Convolutional Networkscited by 76
  7. 2020The Depth-to-Width Interplay in Self-Attentioncited by 39
  8. 2015Perturbation theory for Markov chains via Wasserstein distancecited by 16
  9. 2020Superbloom: Bloom filter meets Transformercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 2; 2024: 1; 2025: 0; 2026: 4.

9 papers.

  1. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  2. 2023How Smooth Is Attention?cited by 44
  3. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  4. 2024Clustering in Causal Attention Maskingcited by 31
  5. 2023Towards Verifying the Geometric Robustness of Large-scale Neural Networkscited by 11
  6. 2026Clustering in Deep Stochastic Transformerscited by 5
  7. 2026A Unified Framework for In-Context Learning with Causal and Masked Language Modelscited by 0
  8. 2026Refinement Symmetry in Multimodal Transformerscited by 0
  9. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0