mlsanityData to 7 Oct 2026

How Smooth Is Attention?

arXiv 2312.14820 ↗First posted 22 Dec 2023cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Self-attention and masked self-attention are at the heart of Transformers' outstanding success. Still, our mathematical understanding of attention, in particular of its Lipschitz properties - which are key when it comes to analyzing robustness and expressive power - is incomplete. We provide a detailed study of the Lipschitz constant of self-attention in several practical scenarios, discussing the impact of the sequence length n and layer normalization on the local Lipschitz constant of both unmasked and masked self-attention. In particular, we show that for inputs of length n in any compact set, the Lipschitz constant of self-attention is bounded by √n up to a constant factor and that this bound is tight for reasonable sequence lengths. When the sequence length n is too large for the previous bound to be tight, which we refer to as the mean-field regime, we provide an upper bound and a matching lower bound which are independent of n. Our mean-field framework for masked self-attention is novel and of independent interest. Our experiments on pretrained and randomly initialized BERT and GPT-2 support our theoretical findings.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 55 of its 67 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

55 of 67 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2016Layer Normalizationcited by 5,579
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2018Neural Ordinary Differential Equationscited by 3,733
  10. 2023Mistral 7Bcited by 3,065
  11. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
38 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  3. 2020Point Transformercited by 1,462
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2021Scaling Vision Transformerscited by 993
  6. 2016Adversarial Machine Learning at Scalecited by 934
  7. 2020On Layer Normalization in the Transformer Architecturecited by 906
  8. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  13. 2018Invertible Residual Networkscited by 364
  14. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  15. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  16. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  17. 2019Residual Flows for Invertible Generative Modelingcited by 256
  18. 2018Sorting out Lipschitz function approximationcited by 211
  19. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  20. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
  21. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  22. 2020The Lipschitz Constant of Self-Attentioncited by 171
  23. 2023A mathematical perspective on Transformerscited by 160
  24. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  25. 2023The emergence of clusters in self-attention dynamicscited by 133
  26. 2016Robust Large Margin Deep Neural Networkscited by 117
  27. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  28. 2021Globally-Robust Neural Networkscited by 106
  29. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  30. 2015Distributional Smoothing with Virtual Adversarial Trainingcited by 61
  31. 2021On the Expressive Power of Self-Attention Matricescited by 44
  32. 2020A Mathematical Theory of Attentioncited by 43
  33. 2020A case for new neural network smoothness constraintscited by 43
  34. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 38
  35. 2023Conditional Optimal Transport on Function Spacescited by 26
  36. 2020A Functional Perspective on Learning Symmetric Functions with Neural Networkscited by 16
  37. 2023Mitigating Transformer Overconfidence via Lipschitz Regularizationcited by 15
  38. 2023Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averagingcited by 14

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 18; 2026: 21.

44 papers.

Sort:

By area:

  1. 2024Transformers are Universal In-context Learnerscited by 42
  2. 2024Clustering in Causal Attention Maskingcited by 31
  3. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  4. 2025Terminal Velocity Matchingcited by 17
  5. 2025A Theoretical Analysis of Discrete Flow Matching Generative Modelscited by 13
  6. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  7. 2025Approximation theory for 1-Lipschitz ResNetscited by 7
  8. 2025Transformers through the lens of support-preserving maps between measurescited by 5
  9. 2025Optimal and Diffusion Transports in Machine Learningcited by 5
  10. 2026Clustering in Deep Stochastic Transformerscited by 5
  11. 2026Approximation Theory for Lipschitz Continuous Transformerscited by 5
  12. 2025DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flowscited by 4
32 more
  1. 2025Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobianscited by 4
  2. 2025Memory Limitations of Prompt Tuning in Transformerscited by 4
  3. 2026Mixture of Sequence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendationcited by 4
  4. 2026Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patchingcited by 4
  5. 2025Quantum latent distributions in deep generative modelscited by 3
  6. 2026Exact Attention Sensitivity and the Geometry of Transformer Stabilitycited by 3
  7. 2025Distributionally robust minimization in meta-learning for system identificationcited by 2
  8. 2025Sliced ReLU attention: Quasi-linear contextual expressivity via sortingcited by 2
  9. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1
  10. 2025Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Driftcited by 1
  11. 2026On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functionscited by 1
  12. 2026Measure-to-measure Regression with Transformerscited by 1
  13. 2026MKEvolve: A Modular Multi-Agent Framework for Kernel Code Generationcited by 1
  14. 2024Improving Fine-Grained Control via Aggregation of Multiple Diffusion Modelscited by 0
  15. 2024Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metriccited by 0
  16. 2025Understanding Sensitivity of Differential Attention through the Lens of Adversarial Robustnesscited by 0
  17. 2025Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropycited by 0
  18. 2025Tight Robustness Certificates and Wasserstein Distributional Attacks for Deep Neural Networkscited by 0
  19. 2025ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equationcited by 0
  20. 2026Deep Neural Networks as Iterated Function Systems and a Generalization Boundcited by 0
  21. 2026CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generationcited by 0
  22. 2026Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Modelscited by 0
  23. 2026Lossless Anti-Distillation Samplingcited by 0
  24. 2026How Many Different Outputs Can a Transformer Generate?cited by 0
  25. 2026Improving Adversarial Robustness of Attribution via Implicit Regularizationcited by 0
  26. 2026General Quantification of Covariate and Concept Shiftscited by 0
  27. 2026To Solve Bilevel Optimization with Nonconvex Lower Levels, We Need Second-Order Stationaritycited by 0
  28. 2026Refinement Symmetry in Multimodal Transformerscited by 0
  29. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0
  30. 2026STCFormer: Adaptive Spatio-Temporal Modeling with Dynamic Cluster Transformer for Station-based Weather Forecastingcited by 0
  31. 2026ATI-VLA: Action-Centric Predictive Vision-Language-Action Models via Actionable Alignment Then Adaptive Injectioncited by 0
  32. 2026Exploring Weaknesses of Generative Image Watermarks against Latent Frequency Maskingcited by 0