Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers

arXiv 2507.07814 ↗First posted 10 Jul 2025cs.LG cs.NA math.NAcited by 11

Not on the map. Why?

Abstract (arXiv)

We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions. The proposed bound is not only tighter than the prior art, but for the first time, reveals how the distribution of attention probabilities shapes the local Lipschitz constant of the self-attention block. The theoretical basis of the proposed upper bound lies in the refined closed-form upper bounds on singular values of the Jacobian of softmax function. Leveraging these theoretical insights, we introduce JaSMin (Jacobian Softmax norm Minimization), a lightweight regularizer that directly controls the local Lipschitz constant of each block and, consequently, the entire model. Additionally, we discuss how the nature of the attention map distribution contributes to the gradient dynamics and, consequently, transformer training stability.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

46 of 65 references.

References · 46

2017Attention Is All You Needcited by 60,361
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
2020Language Models are Few-Shot Learnerscited by 32,419
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
2014Explaining and Harnessing Adversarial Examplescited by 7,364
2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
2023Mistral 7Bcited by 3,164
2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884

34 more

2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
2020On Layer Normalization in the Transformer Architecturecited by 922
2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
2020The Lipschitz Constant of Self-Attentioncited by 174
2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
2022Understanding The Robustness in Vision Transformerscited by 144
2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 142
2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
2021Towards Robust Vision Transformercited by 137
2025Why do LLMs attend to the first token?cited by 127
2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
2023Adversarial Demonstration Attacks on Large Language Modelscited by 63
2024CViT: Continuous Vision Transformer for Operator Learningcited by 51
2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
2023How Smooth Is Attention?cited by 44
2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 39
2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 34
2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
2022When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanismcited by 30
2018On Controllable Sparse Alternatives to Softmaxcited by 27
2025NeoBERT: A Next-Generation BERTcited by 16
2023Transformers and Large Language Models for Chemistry and Drug Discoverycited by 13
2023Sparsity-aware generalization theory for deep neural networkscited by 8
2024SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalizationcited by 5
2024Learning Physical Simulation with Message Passing Transformercited by 3
2025LipShiFT: A Certifiably Robust Shift-based Vision Transformercited by 3
2025EDIT: Enhancing Vision Transformers by Mitigating Attention Sink through an Encoder-Decoder Architecturecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.