mlsanityData to 7 Oct 2026

Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networks

arXiv 2103.04886 ↗First posted 8 Mar 2021cs.LG stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

Attention based neural networks are state of the art in a large range of applications. However, their performance tends to degrade when the number of layers increases. In this work, we show that enforcing Lipschitz continuity by normalizing the attention scores can significantly improve the performance of deep attention models. First, we show that, for deep graph attention networks (GAT), gradient explosion appears during training, leading to poor performance of gradient-based training algorithms. To address this issue, we derive a theoretical analysis of the Lipschitz continuity of attention modules and introduce LipschitzNorm, a simple and parameter-free normalization for self-attention mechanisms that enforces the model to be Lipschitz continuous. We then apply LipschitzNorm to GAT and Graph Transformers and show that their performance is substantially improved in the deep setting (10 to 30 layers). More specifically, we show that a deep GAT model with LipschitzNorm achieves state of the art results for node label prediction tasks that exhibit long-range dependencies, while showing consistent improvements over their unnormalized counterparts in benchmark node classification tasks.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

33 of 44 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2017Graph Attention Networkscited by 8,854
  7. 2017Inductive Representation Learning on Large Graphscited by 6,284
  8. 2016Layer Normalizationcited by 5,579
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2018How Powerful are Graph Neural Networks?cited by 4,355
  11. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  12. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
21 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  3. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  4. 2020Simple and Deep Graph Convolutional Networkscited by 857
  5. 2015Gated Graph Sequence Neural Networkscited by 792
  6. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  7. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  8. 2019Graph Transformer Networkscited by 520
  9. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
  10. 2017Representation Learning on Graphs: Methods and Applicationscited by 456
  11. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  12. 2018Invariant and Equivariant Graph Networkscited by 339
  13. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  14. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  15. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  16. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
  17. 2019What graph neural networks cannot learn: depth vs widthcited by 195
  18. 2020The Lipschitz Constant of Self-Attentioncited by 171
  19. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  20. 2016A Convolutional Encoder Model for Neural Machine Translationcited by 54
  21. 2019Coloring graph neural networks for node disambiguationcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 2; 2023: 14; 2024: 4; 2025: 9; 2026: 6.

38 papers.

Sort:

By area:

  1. 2023How Smooth Is Attention?cited by 44
  2. 2021VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantizationcited by 39
  3. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 33
  4. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  5. 2022On the Robustness of Graph Neural Diffusion to Topology Perturbationscited by 30
  6. 2023Universality and Limitations of Prompt Tuningcited by 30
  7. 2023Mitigating Transformer Overconfidence via Lipschitz Regularizationcited by 15
  8. 2023Unlocking Deterministic Robustness Certification on ImageNetcited by 13
  9. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  10. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  11. 2023Are GATs Out of Balance?cited by 9
  12. 2024Fair Bilevel Neural Network (FairBiNN): On Balancing fairness and accuracy via Stackelberg Equilibriumcited by 9
26 more
  1. 2025Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief Statescited by 9
  2. 2023Centered Self-Attention Layerscited by 7
  3. 2025Approximation theory for 1-Lipschitz ResNetscited by 7
  4. 2024DuSEGO: Dual Second-order Equivariant Graph Ordinary Differential Equationcited by 6
  5. 2025Pool Me Wisely: On the Effect of Pooling in Transformer-Based Modelscited by 6
  6. 2023Interpretable Graph Neural Networks for Tabular Datacited by 5
  7. 2024SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalizationcited by 5
  8. 2024Global Convergence in Training Large-Scale Transformerscited by 5
  9. 2023SwinGar: Spectrum-Inspired Neural Dynamic Deformation for Free-Swinging Garmentscited by 3
  10. 2026Exact Attention Sensitivity and the Geometry of Transformer Stabilitycited by 3
  11. 2023CLiSA: A Hierarchical Hybrid Transformer Model using Orthogonal Cross Attention for Satellite Image Cloud Segmentationcited by 1
  12. 2025LEL: Lipschitz Continuity Constrained Ensemble Learning for Efficient EEG-Based Intra-subject Emotion Recognitioncited by 1
  13. 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networkscited by 1
  14. 2026Information-Theoretic Limits of Safety Verification for Self-Improving Systemscited by 1
  15. 2021Attentive Contractive Flow with Lipschitz-constrained Self-Attentioncited by 0
  16. 2022Density-Aware Hyper-Graph Neural Networks for Graph-based Semi-supervised Node Classificationcited by 0
  17. 2023Adaptive Depth Graph Attention Networkscited by 0
  18. 2023Supervised Attention Using Homophily in Graph Neural Networkscited by 0
  19. 2023Promoting Fairness in GNNs: A Characterization of Stabilitycited by 0
  20. 2025Understanding Sensitivity of Differential Attention through the Lens of Adversarial Robustnesscited by 0
  21. 2025On the Convergence and Size Transferability of Continuous-depth Graph Neural Networkscited by 0
  22. 2025ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equationcited by 0
  23. 2026Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Modelscited by 0
  24. 2026Polynomial Context-Truncation Sensitivity in Autoregressive Language Models: Sequential Wyner-Ziv Bounds for KV Cache Compressioncited by 0
  25. 2026Harnessing non-adversarial robustness in large language modelscited by 0
  26. 2026Reference-Tail Trust:Certified Probability Floors for Learned Updates Inside a Deployed Networkcited by 0