mlsanityData to 7 Oct 2026

Demystifying Oversmoothing in Attention-Based Graph Neural Networks

arXiv 2305.16102 ↗First posted 25 May 2023cs.LG cs.SI stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Oversmoothing in Graph Neural Networks (GNNs) refers to the phenomenon where increasing network depth leads to homogeneous node representations. While previous work has established that Graph Convolutional Networks (GCNs) exponentially lose expressive power, it remains controversial whether the graph attention mechanism can mitigate oversmoothing. In this work, we provide a definitive answer to this question through a rigorous mathematical analysis, by viewing attention-based GNNs as nonlinear time-varying dynamical systems and incorporating tools and techniques from the theory of products of inhomogeneous matrices and the joint spectral radius. We establish that, contrary to popular belief, the graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially. The proposed framework extends the existing results on oversmoothing for symmetric GCNs to a significantly broader class of GNN models, including random walk GCNs, Graph Attention Networks (GATs) and (graph) transformers. In particular, our analysis accounts for asymmetric, state-dependent and time-varying aggregation operators and a wide range of common nonlinear activation functions, such as ReLU, LeakyReLU, GELU and SiLU.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

35 of 49 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2017Graph Attention Networkscited by 8,854
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  8. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  9. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
  10. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  11. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,097
  12. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
23 more
  1. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  2. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  3. 2020Simple and Deep Graph Convolutional Networkscited by 857
  4. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
  5. 2021How Attentive are Graph Attention Networks?cited by 818
  6. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 751
  7. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  8. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  9. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  10. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 493
  11. 2019Diffusion Improves Graph Learningcited by 414
  12. 2018Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generationcited by 363
  13. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  14. 2023A Survey on Oversmoothing in Graph Neural Networkscited by 246
  15. 2020A Note on Over-Smoothing for Graph Neural Networkscited by 245
  16. 2022Graph-Coupled Oscillator Networkscited by 123
  17. 2022Not too little, not too much: a theoretical analysis of graph (over)smoothingcited by 106
  18. 2020Bayesian Graph Neural Networks with Adaptive Connection Samplingcited by 81
  19. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
  20. 2020Scattering GCN: Overcoming Oversmoothness in Graph Convolutional Networkscited by 57
  21. 2022A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networkscited by 49
  22. 2019Improving Graph Attention Networks with Large Margin-based Constraintscited by 34
  23. 2022Graph Attention Retrospectivecited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 18; 2025: 18; 2026: 8.

47 papers.

Sort:

By area:

  1. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  2. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 42
  3. 2024Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNscited by 31
  4. 2024Clustering in Causal Attention Maskingcited by 31
  5. 2024Measure-to-measure interpolation using Transformerscited by 28
  6. 2023Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networkscited by 18
  7. 2025Oversmoothing, Oversquashing, Heterophily, Long-Range, and more: Demystifying Common Beliefs in Graph Machine Learningcited by 18
  8. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  9. 2024Bundle Neural Networks for message diffusion on graphscited by 13
  10. 2024MambaTS: Improved Selective State Space Models for Long-term Time Series Forecastingcited by 13
  11. 2024Revisiting Random Walks for Learning on Graphscited by 13
  12. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
35 more
  1. 2024Graph Neural Networks Do Not Always Oversmoothcited by 7
  2. 2024Combining Graph Neural Network and Mamba to Capture Local and Global Tissue Spatial Relationships in Whole Slide Imagescited by 7
  3. 2025Are We Measuring Oversmoothing in Graph Neural Networks Correctly?cited by 6
  4. 2023Attentional Graph Neural Network Is All You Need for Robust Massive Network Localizationcited by 5
  5. 2024GRASP: GRAph-Structured Pyramidal Whole Slide Image Representationcited by 5
  6. 2025Residual connections provably mitigate oversmoothing in graph neural networkscited by 5
  7. 2024Simplifying the Theory on Over-Smoothingcited by 4
  8. 2024Lambda-Skip Connections: the architectural component that prevents Rank Collapsecited by 4
  9. 2025Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Spacecited by 4
  10. 2024Improving Graph Neural Networks by Learning Continuous Edge Directionscited by 3
  11. 2025Transformer Meets Twicing: Harnessing Unattended Residual Informationcited by 3
  12. 2025Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlationcited by 3
  13. 2025The Oversmoothing Fallacy: A Misguided Narrative in GNN Researchcited by 3
  14. 2024Taming Gradient Oversmoothing and Expansion in Graph Neural Networkscited by 2
  15. 2024Making Classic GNNs Strong Baselines Across Varying Homophily: A Smoothness-Generalization Perspectivecited by 2
  16. 2025A Signed Graph Approach to Understanding and Mitigating Oversmoothing in GNNscited by 2
  17. 2025Hyper-SET: Designing Transformers via Hyperspherical Energy Minimizationcited by 2
  18. 2025Backward Oversmoothing: why is it hard to train deep Graph Neural Networks?cited by 2
  19. 2025Scaling Kinetic Monte-Carlo Simulations of Grain Growth with Combined Convolutional and Graph Neural Networkscited by 2
  20. 2024Bridging Smoothness and Approximation: Theoretical Insights into Over-Smoothing in Graph Neural Networkscited by 1
  21. 2024A Dynamical Systems-Inspired Pruning Strategy for Addressing Oversmoothing in Graph Neural Networkscited by 1
  22. 2025Resolving Oversmoothing with Opinion Dissensuscited by 1
  23. 2025Wavy Transformercited by 1
  24. 2025Measuring Over-smoothing beyond Dirichlet energycited by 1
  25. 2026Gaussian Process Limit Reveals Structural Benefits of Graph Transformerscited by 1
  26. 2026Machine Learning Multiscale Interactionscited by 1
  27. 2025How do Probabilistic Graphical Models and Graph Neural Networks Look at Network Data?cited by 0
  28. 2025Solving Oversmoothing in GNNs via Nonlocal Message Passing: Algebraic Smoothing and Depth Scalabilitycited by 0
  29. 2025How Smoothing is N-simplicial Attention?cited by 0
  30. 2026MRC-GAT: A Meta-Relational Copula-Based Graph Attention Network for Interpretable Multimodal Alzheimer's Disease Diagnosiscited by 0
  31. 2026Graph neural network for colliding particles with an application to sea ice floe modelingcited by 0
  32. 2026ST-ResGAT: Explainable Spatio-Temporal Graph Neural Network for Road Condition Prediction and Priority-Driven Maintenancecited by 0
  33. 2026Distance-Misaligned Training in Graph Transformers and Adaptive Graph-Aware Controlcited by 0
  34. 2026BrainDyn: A Sheaf Neural ODE for Generative Brain Dynamicscited by 0
  35. 2026Exploring Oversmoothing with Householder Matricescited by 0