mlsanityData to 7 Oct 2026

Are GATs Out of Balance?

arXiv 2310.07235 ↗First posted 11 Oct 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

While the expressive power and computational capabilities of graph neural networks (GNNs) have been theoretically studied, their optimization and learning dynamics, in general, remain largely unexplored. Our study undertakes the Graph Attention Network (GAT), a popular GNN architecture in which a node's neighborhood aggregation is weighted by parameterized attention coefficients. We derive a conservation law of GAT gradient flow dynamics, which explains why a high portion of parameters in GATs with standard initialization struggle to change during training. This effect is amplified in deeper GATs, which perform significantly worse than their shallow counterparts. To alleviate this problem, we devise an initialization scheme that balances the GAT network. Our approach i) allows more effective propagation of gradients and in turn enables trainability of deeper networks, and ii) attains a considerable speedup in training and convergence time in comparison to the standard initialization. Our main theorem serves as a stepping stone to studying the learning dynamics of positive homogeneous models with attention mechanisms.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 61 of its 67 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

61 of 67 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2016Layer Normalizationcited by 5,579
  6. 2018How Powerful are Graph Neural Networks?cited by 4,355
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  9. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  10. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  11. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
  12. 2019Graph WaveNet for Deep Spatial-Temporal Graph Modelingcited by 820
38 more
  1. 2021How Attentive are Graph Attention Networks?cited by 818
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  4. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  5. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  6. 2019Graph Transformer Networkscited by 520
  7. 2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 332
  8. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  9. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  10. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  11. 2020Towards Deeper Graph Neural Networkscited by 296
  12. 2016Molecular Graph Convolutions: Moving Beyond Fingerprintscited by 238
  13. 2020Design Space for Graph Neural Networkscited by 182
  14. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  15. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  16. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  17. 2019Understanding Attention and Generalization in Graph Neural Networkscited by 169
  18. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  19. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  20. 2020Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networkscited by 152
  21. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  22. 2015All you need is a good initcited by 124
  23. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  24. 2021DropGNN: Random Dropouts Increase the Expressiveness of Graph Neural Networkscited by 117
  25. 2020GraphNorm: A Principled Approach to Accelerating Graph Neural Network Trainingcited by 116
  26. 2020Towards Deeper Graph Neural Networks with Differentiable Group Normalizationcited by 113
  27. 2021PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equationscited by 106
  28. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  29. 2022How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervisioncited by 102
  30. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  31. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  32. 2019Inductive Matrix Completion Based on Graph Neural Networkscited by 85
  33. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  34. 2022Theory of Graph Neural Networks: Representation and Learningcited by 60
  35. 2021On Provable Benefits of Depth in Training Graph Convolutional Networkscited by 58
  36. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  37. 2020Understanding and Resolving Performance Degradation in Graph Convolutional Networkscited by 53
  38. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 2.

9 papers.

  1. 2025Symmetry in Neural Network Parameter Spacescited by 31
  2. 2024Spectral Graph Pruning Against Over-Squashing and Over-Smoothingcited by 23
  3. 2025GNNs Getting ComFy: Community and Feature Similarity Guided Rewiringcited by 13
  4. 2024Unitary convolutions for learning on graphs and groupscited by 8
  5. 2024GATE: How to Keep Out Intrusive Neighborscited by 4
  6. 2026Fixed Aggregation Features Can Rival GNNscited by 2
  7. 2024Control the GNN: Utilizing Neural Controller with Lyapunov Stability for Test-Time Feature Reconstructioncited by 1
  8. 2025Memorization in Graph Neural Networkscited by 0
  9. 2026Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer?cited by 0