mlsanityData to 7 Oct 2026

The Lipschitz Constant of Self-Attention

arXiv 2006.04710 ↗First posted 8 Jun 2020stat.ML cs.LGcited by 171

Not on the map. Why?

Abstract (arXiv)

Lipschitz constants of neural networks have been explored in various contexts in deep learning, such as provable adversarial robustness, estimating Wasserstein distance, stabilising training of GANs, and formulating invertible neural networks. Such works have focused on bounding the Lipschitz constant of fully connected or convolutional networks, composed of linear maps and pointwise non-linearities. In this paper, we investigate the Lipschitz constant of self-attention, a non-linear neural network module widely used in sequence modelling. We prove that the standard dot-product self-attention is not Lipschitz for unbounded input domain, and propose an alternative L2 self-attention that is Lipschitz. We derive an upper bound on the Lipschitz constant of L2 self-attention and provide empirical evidence for its asymptotic tightness. To demonstrate the practical relevance of our theoretical work, we formulate invertible self-attention and use it in a Transformer-based architecture for a character-level language modelling task.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 13; 2023: 23; 2024: 29; 2025: 40; 2026: 50.

171 papers.

Sort:

By area:

  1. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  2. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  3. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  4. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 258
  5. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 251
  6. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 200
  7. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  8. 2022On The Computational Complexity of Self-Attentioncited by 137
  9. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  10. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  11. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  12. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
38 more
  1. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  2. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 59
  3. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  4. 2022Reachability Constrained Reinforcement Learningcited by 52
  5. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  6. 2023How Smooth Is Attention?cited by 44
  7. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 43
  8. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  9. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 39
  10. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 38
  11. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
  12. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 35
  13. 2024Learning Continually by Spectral Regularizationcited by 34
  14. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 33
  15. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  16. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  17. 2023Universality and Limitations of Prompt Tuningcited by 30
  18. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 30
  19. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  20. 2022Neural Integral Equationscited by 23
  21. 2025Spiking Vision Transformer with Saccadic Attentioncited by 21
  22. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 21
  23. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  24. 2025Training Transformers with Enforced Lipschitz Constantscited by 18
  25. 2025Softmax is 1/2-Lipschitz: A tight bound across all ℓp normscited by 18
  26. 2025Terminal Velocity Matchingcited by 17
  27. 2021LipBaB: Computing exact Lipschitz constant of ReLU networkscited by 16
  28. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  29. 2024Law of Vision Representation in MLLMscited by 16
  30. 2023Mitigating Transformer Overconfidence via Lipschitz Regularizationcited by 15
  31. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  32. 2021Controlling Wasserstein Distances by Kernel Norms with Application to Compressive Statistical Learningcited by 13
  33. 2023Unlocking Deterministic Robustness Certification on ImageNetcited by 13
  34. 2023Exploring Sparse MoE in GANs for Text-conditioned Image Synthesiscited by 13
  35. 2025T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuningcited by 12
  36. 2022Semi-Discrete Normalizing Flows through Differentiable Tessellationcited by 11
  37. 2023Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generationcited by 11
  38. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10

50 most cited of 171 All 171 (JSON)