mlsanityData to 7 Oct 2026

A Mathematical Theory of Attention

arXiv 2007.02876 ↗First posted 6 Jul 2020stat.ML cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Attention is a powerful component of modern neural networks across a wide variety of domains. However, despite its ubiquity in machine learning, there is a gap in our understanding of attention from a theoretical point of view. We propose a framework to fill this gap by building a mathematically equivalent model of attention using measure theory. With this model, we are able to interpret self-attention as a system of self-interacting particles, we shed light on self-attention from a maximum entropy perspective, and we show that attention is actually Lipschitz-continuous (with an appropriate metric) under suitable assumptions. We then apply these insights to the problem of mis-specified input data; infinitely-deep, weight-sharing self-attention networks; and more general Lipschitz estimates for a specific type of attention studied in concurrent work.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 12; 2024: 8; 2025: 12; 2026: 7.

43 papers.

Sort:

  1. 2023A mathematical perspective on Transformerscited by 160
  2. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  3. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  4. 2022Vision Transformers provably learn spatial structurecited by 92
  5. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  6. 2023An Over-parameterized Exponential Regressioncited by 45
  7. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  8. 2023How Smooth Is Attention?cited by 44
  9. 2024Transformers are Universal In-context Learnerscited by 42
  10. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  11. 2023Universality and Limitations of Prompt Tuningcited by 30
  12. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 29
31 more
  1. 2023A Mathematical Abstraction for Balancing the Trade-off Between Creativity and Reality in Large Language Modelscited by 29
  2. 2024Measure-to-measure interpolation using Transformerscited by 28
  3. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  4. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 15
  5. 2023Why Softmax Attention Outperforms Linear Attentioncited by 15
  6. 2024Interpretable Lightweight Transformer via Unrolling of Learned Graph Smoothness Priorscited by 8
  7. 2024Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamicscited by 7
  8. 2024Universal Approximation of Mean-Field Models via Transformerscited by 7
  9. 2024Global Convergence in Training Large-Scale Transformerscited by 5
  10. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5
  11. 2025Transformers through the lens of support-preserving maps between measurescited by 5
  12. 2023Learning to Decouple Complex Systemscited by 3
  13. 2025Token Sample Complexity of Attentioncited by 3
  14. 2025Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspectivecited by 3
  15. 2024Towards understanding how attention mechanism works in deep learningcited by 2
  16. 2025Sliced ReLU attention: Quasi-linear contextual expressivity via sortingcited by 2
  17. 2026Propagation of Chaos in Contextual Flow Mapscited by 2
  18. 2025Towards Sampling Data Structures for Tensor Products in Turnstile Streamscited by 1
  19. 2026Algebraic Invariants of Lightning Self-Attentioncited by 1
  20. 2026The physics of AI weather modelscited by 1
  21. 2023Manifold-Preserving Transformers are Effective for Short-Long Range Encodingcited by 0
  22. 2024Local Attention Mechanism: Boosting the Transformer Architecture for Long-Sequence Time Series Forecastingcited by 0
  23. 2025Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Modelscited by 0
  24. 2025Attention with Trained Embeddings Provably Selects Important Tokenscited by 0
  25. 2025Revisiting Transformers with Insights from Image Filtering and Boostingcited by 0
  26. 2025Physical models realizing the transformer architecture of large language modelscited by 0
  27. 2025Change of Thought: Adaptive Test-Time Computationcited by 0
  28. 2026Extreme Meta-Classification for Large-Scale Zero-Shot Retrievalcited by 0
  29. 2026Measuring Optimal Transport in Transformer Depthcited by 0
  30. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0
  31. 2026Attention Kernels for Learning Maps Between Heavy-Tailed Measurescited by 0