mlsanityData to 14 Oct 2026

Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks

arXiv 2308.16800 ↗First posted 31 Aug 2023cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Our study reveals new theoretical insights into over-smoothing and feature over-correlation in graph neural networks. Specifically, we demonstrate that with increased depth, node representations become dominated by a low-dimensional subspace that depends on the aggregation function but not on the feature transformations. For all aggregation functions, the rank of the node representations collapses, resulting in over-smoothing for particular aggregation functions. Our study emphasizes the importance for future research to focus on rank collapse rather than over-smoothing. Guided by our theory, we propose a sum of Kronecker products as a beneficial property that provably prevents over-smoothing, over-correlation, and rank collapse. We empirically demonstrate the shortcomings of existing models in fitting target functions of node classification tasks.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 48 of its 67 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

48 of 67 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2017Graph Attention Networkscited by 8,923
  6. 2017Inductive Representation Learning on Large Graphscited by 6,320
  7. 2018How Powerful are Graph Neural Networks?cited by 4,393
  8. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  9. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  10. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
  11. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
  12. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
36 more
  1. 2020Simple and Deep Graph Convolutional Networkscited by 860
  2. 2021How Attentive are Graph Attention Networks?cited by 835
  3. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 826
  4. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 680
  5. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
  6. 2019Multi-scale Attributed Node Embeddingcited by 548
  7. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 545
  8. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  10. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  11. 2021Beyond Low-frequency Information in Graph Convolutional Networkscited by 362
  12. 2017Residual Gated Graph ConvNetscited by 346
  13. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  14. 2020Towards Deeper Graph Neural Networkscited by 296
  15. 2019Revisiting Graph Neural Networks: All We Have is Low-Pass Filterscited by 261
  16. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  17. 2020A Note on Over-Smoothing for Graph Neural Networkscited by 249
  18. 2023A Survey on Oversmoothing in Graph Neural Networkscited by 247
  19. 2022Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNscited by 214
  20. 2021Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networkscited by 182
  21. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  22. 2022EquiBind: Geometric Deep Learning for Drug Binding Structure Predictioncited by 144
  23. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  24. 2022Graph-Coupled Oscillator Networkscited by 124
  25. 2022Not too little, not too much: a theoretical analysis of graph (over)smoothingcited by 106
  26. 2021Dirichlet Energy Constrained Learning for Deep Graph Neural Networkscited by 97
  27. 2023Edge Directionality Improves Learning on Heterophilic Graphscited by 90
  28. 2022Gradient Gating for Deep Multi-Rate Learning on Graphscited by 49
  29. 2023Demystifying Oversmoothing in Attention-Based Graph Neural Networkscited by 49
  30. 2023A Fractional Graph Laplacian Approach to Oversmoothingcited by 45
  31. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  32. 2023Some Might Say All You Need Is Sumcited by 21
  33. 2021Orthogonal Graph Neural Networkscited by 19
  34. 2022Feature Overcorrelation in Deep Graph Neural Networks: A New Perspectivecited by 19
  35. 2022Transforming PageRank into an Infinite-Depth Graph Neural Networkcited by 11
  36. 2022Forecasting Unobserved Node States with spatio-temporal Graph Neural Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 9; 2026: 2.

18 papers.

Sort:

By area:

  1. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 43
  2. 2024Spectral Graph Pruning Against Over-Squashing and Over-Smoothingcited by 24
  3. 2025Oversmoothing, Oversquashing, Heterophily, Long-Range, and more: Demystifying Common Beliefs in Graph Machine Learningcited by 19
  4. 2025Rethinking Functional Brain Connectome Analysis: Do Graph Deep Learning Models Helpcited by 9
  5. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  6. 2024Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graphcited by 6
  7. 2025Are We Measuring Oversmoothing in Graph Neural Networks Correctly?cited by 6
  8. 2025Improving Long-Range Interactions in Graph Neural Simulators via Hamiltonian Dynamicscited by 6
  9. 2024Simplifying the Theory on Over-Smoothingcited by 4
  10. 2024Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modelingcited by 3
  11. 2025On the Relationship Between Robustness and Expressivity of Graph Neural Networkscited by 3
  12. 2024Generalized Probabilistic Attention Mechanism in Transformerscited by 2
6 more
  1. 2026Position: Don't be Afraid of Over-Smoothing And Over-Squashingcited by 2
  2. 2025What Can We Learn From MIMO Graph Convolutions?cited by 1
  3. 2023Distilling Influences to Mitigate Prediction Churn in Graph Neural Networkscited by 0
  4. 2024Differential Encoding for Improved Representation Learning over Graphscited by 0
  5. 2025How Smoothing is N-simplicial Attention?cited by 0
  6. 2026LiftGCN: Efficient Energy-Preserving Graph Learning via Joukowski Spectral Lifting for Finite Element Stress Predictioncited by 0