mlsanityData to 7 Oct 2026

Towards Deep Attention in Graph Neural Networks: Problems and Remedies

arXiv 2306.02376 ↗First posted 4 Jun 2023cs.LG cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Graph neural networks (GNNs) learn the representation of graph-structured data, and their expressiveness can be further enhanced by inferring node relations for propagation. Attention-based GNNs infer neighbor importance to manipulate the weight of its propagation. Despite their popularity, the discussion on deep graph attention and its unique challenges has been limited. In this work, we investigate some problematic phenomena related to deep graph attention, including vulnerability to over-smoothed features and smooth cumulative attention. Through theoretical and empirical analyses, we show that various attention-based GNNs suffer from these problems. Motivated by our findings, we propose AEROGNN, a novel GNN architecture designed for deep graph attention. AERO-GNN provably mitigates the proposed problems of deep graph attention, which is further empirically demonstrated with (a) its adaptive and less smooth attention functions and (b) higher performance at deep layers (up to 64). On 9 out of 12 node classification benchmarks, AERO-GNN outperforms the baseline GNNs, highlighting the advantages of deep graph attention. Our code is available at https://github.com/syleeheal/AERO-GNN.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 65 of its 89 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

65 of 89 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Graph Attention Networkscited by 8,854
  5. 2016Layer Normalizationcited by 5,579
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  8. 2019Simplifying Graph Convolutional Networkscited by 1,470
  9. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,134
  10. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  11. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  12. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
38 more
  1. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
  2. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  3. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  4. 2020Simple and Deep Graph Convolutional Networkscited by 857
  5. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
  6. 2021How Attentive are Graph Attention Networks?cited by 818
  7. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  8. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  9. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 592
  10. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 544
  11. 2019Multi-scale Attributed Node Embeddingcited by 543
  12. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  13. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
  14. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 493
  15. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 471
  16. 2019MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixingcited by 442
  17. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  18. 2021Beyond Low-frequency Information in Graph Convolutional Networkscited by 361
  19. 2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 332
  20. 2021Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methodscited by 309
  21. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  22. 2020Towards Deeper Graph Neural Networkscited by 296
  23. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  24. 2021GRAND: Graph Neural Diffusioncited by 258
  25. 2020Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networkscited by 253
  26. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
  27. 2022Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNscited by 208
  28. 2021Is Homophily a Necessity for Graph Neural Networks?cited by 182
  29. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  30. 2022How Powerful are Spectral Graph Neural Networkscited by 169
  31. 2021ETA Prediction with Graph Neural Networks in Google Mapscited by 130
  32. 2018DeepInf: Social Influence Prediction with Deep Learningcited by 128
  33. 2020Towards Deeper Graph Neural Networks with Differentiable Group Normalizationcited by 113
  34. 2022How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervisioncited by 102
  35. 2022Decoupling the Depth and Scope of Graph Neural Networkscited by 94
  36. 2021New Benchmarks for Learning on Non-Homophilous Graphscited by 81
  37. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
  38. 2021Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalizationcited by 70

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 13; 2025: 9.

24 papers.

Sort:

  1. 2024A Survey on Hypergraph Neural Networks: An In-Depth and Step-By-Step Guidecited by 22
  2. 2023Breaking the Entanglement of Homophily and Heterophily in Semi-supervised Node Classificationcited by 19
  3. 2024How Interpretable Are Interpretable Graph Neural Networks?cited by 13
  4. 2024Rethinking Reconstruction-based Graph-Level Anomaly Detection: Limitations and a Simple Remedycited by 13
  5. 2024Feature Distribution on Graph Topology Mediates the Effect of Graph Convolution: Homophily Perspectivecited by 9
  6. 2024Sign is Not a Remedy: Multiset-to-Multiset Message Passing for Learning on Heterophilic Graphscited by 8
  7. 2025Depth-Adaptive Graph Neural Networks via Learnable Bakry-'Emery Curvaturecited by 8
  8. 2024Layer-diverse Negative Sampling for Graph Neural Networkscited by 7
  9. 2024GATE: How to Keep Out Intrusive Neighborscited by 4
  10. 2025Discrepancy-Aware Graph Mask Auto-Encodercited by 4
  11. 2023Triplet Edge Attention for Algorithmic Reasoningcited by 2
  12. 2024Unleash Graph Neural Networks from Heavy Tuningcited by 2
12 more
  1. 2024Scalable Graph Compressed Convolutionscited by 2
  2. 2025ScaleGNN: Towards Scalable Graph Neural Networks via Adaptive High-order Neighboring Feature Fusioncited by 2
  3. 2025Hierarchical Uncertainty-Aware Graph Neural Networkcited by 1
  4. 2025Local Virtual Nodes for Alleviating Over-Squashing in Graph Neural Networkscited by 1
  5. 2024Neighbour-level Message Interaction Encoding for Improved Representation Learning on Graphscited by 0
  6. 2024Differential Encoding for Improved Representation Learning over Graphscited by 0
  7. 2024Graph Dimension Attention Networks for Enterprise Credit Assessmentcited by 0
  8. 2024Better Not to Propagate: Understanding Edge Uncertainty and Over-smoothing in Signed Graph Neural Networkscited by 0
  9. 2025Channel-Attentive Graph Neural Networkscited by 0
  10. 2025H3GNNs: Harmonizing Heterophily and Homophily in GNNs via Joint Structural Node Encoding and Self-Supervised Learningcited by 0
  11. 2025PyTorch-based Geometric Learning with Non-CUDA Processing Units: Experiences from Intel Gaudi-v2 HPUscited by 0
  12. 2025GraphTorque: Torque-Driven Rewiring Graph Neural Networkcited by 0