mlsanityData to 14 Oct 2026

How Attentive are Graph Attention Networks?

arXiv 2105.14491 ↗First posted 30 May 2021cs.LGcited by 835

Not on the map. Why?

Cited by 28 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Graph Attention Networks (GATs) are one of the most popular GNN architectures and are considered as the state-of-the-art architecture for representation learning with graphs. In GAT, every node attends to its neighbors given its own representation as the query. However, in this paper we show that GAT computes a very limited kind of attention: the ranking of the attention scores is unconditioned on the query node. We formally define this restricted kind of attention as static attention and distinguish it from a strictly more expressive dynamic attention. Because GATs use a static attention mechanism, there are simple graph problems that GAT cannot express: in a controlled problem, we show that static attention hinders GAT from even fitting the training data. To remove this limitation, we introduce a simple fix by modifying the order of operations and propose GATv2: a dynamic graph attention variant that is strictly more expressive than GAT. We perform an extensive evaluation and show that GATv2 outperforms GAT across 11 OGB and other benchmarks while we match their parametric costs. Our code is available at https://github.com/tech-srl/how_attentive_are_gats . GATv2 is available as part of the PyTorch Geometric library, the Deep Graph Library, and the TensorFlow GNN library.

Key facts. Cited by 835 papers in the map (data through 14 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 30 May 2021.

Builds on

52 of 72 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2017Graph Attention Networkscited by 8,923
  5. 2017Inductive Representation Learning on Large Graphscited by 6,320
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2018How Powerful are Graph Neural Networks?cited by 4,393
  8. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  9. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  10. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  11. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  12. 2019Simplifying Graph Convolutional Networkscited by 1,477
38 more
  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  2. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
  3. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  5. 2015Gated Graph Sequence Neural Networkscited by 796
  6. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 754
  7. 2019Heterogeneous Graph Attention Networkcited by 734
  8. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 680
  9. 2020Benchmarking Graph Neural Networkscited by 680
  10. 2020A Generalization of Transformer Networks to Graphscited by 661
  11. 2020Contrastive Multi-View Representation Learning on Graphscited by 652
  12. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 631
  13. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  14. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  15. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  16. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  17. 2016Geometric deep learning on graphs and manifolds using mixture model CNNscited by 468
  18. 2017A simple neural network module for relational reasoningcited by 346
  19. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  20. 2015Diffusion-Convolutional Neural Networkscited by 268
  21. 2017One-Shot Imitation Learningcited by 233
  22. 2019Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networkscited by 229
  23. 2017Learning to Represent Programs with Graphscited by 220
  24. 2020Combining Label Propagation and Simple Models Out-performs Graph Neural Networkscited by 202
  25. 2018GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphscited by 165
  26. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  27. 2018DeepInf: Social Influence Prediction with Deep Learningcited by 128
  28. 2019Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphscited by 125
  29. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  30. 2018Attention-based Graph Neural Network for Semi-supervised Learningcited by 101
  31. 2020Minimum Width for Universal Approximationcited by 94
  32. 2019Relational Graph Attention Networkscited by 82
  33. 2019GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulationcited by 75
  34. 2017VAIN: Attentional Multi-agent Predictive Modelingcited by 53
  35. 2020Pointer Graph Networkscited by 53
  36. 2018Attention Models in Graphs: A Surveycited by 41
  37. 2019Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networkscited by 40
  38. 2019Improving Graph Attention Networks with Large Margin-based Constraintscited by 34

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 97; 2023: 150; 2024: 181; 2025: 206; 2026: 189.

835 papers.

Sort:

By area:

  1. 2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 343
  2. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
  3. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  4. 2022Graph neural networks for materials science and chemistrycited by 182
  5. 2023Graph Inductive Biases in Transformers without Message Passingcited by 153
  6. 2023GraphText: Graph Reasoning in Text Spacecited by 111
  7. 2021Causal Attention for Interpretable and Generalizable Graph Classificationcited by 108
  8. 2021PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equationscited by 106
  9. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 106
  10. 2022Periodic Graph Transformers for Crystal Material Property Predictioncited by 87
  11. 2023Everything is Connected: Graph Neural Networkscited by 85
  12. 2023Decoding Natural Images from EEG for Object Recognitioncited by 81
38 more
  1. 2022DiffWire: Inductive Graph Rewiring via the Lov'asz Boundcited by 79
  2. 2021JoinABLe: Learning Bottom-up Assembly of Parametric CAD Jointscited by 78
  3. 2021Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction predictioncited by 68
  4. 2022GoRela: Go Relative for Viewpoint-Invariant Motion Forecastingcited by 68
  5. 2022Global Explainability of GNNs via Logic Combination of Learned Conceptscited by 65
  6. 2023On the Connection Between MPNN and Graph Transformercited by 61
  7. 2022GraphWorld: Fake Graphs Bring Real Insights for GNNscited by 56
  8. 2023How does over-squashing affect the power of GNNs?cited by 53
  9. 2024GKAN: Graph Kolmogorov-Arnold Networkscited by 53
  10. 2024AnyGraph: Graph Foundation Model in the Wildcited by 53
  11. 2023InGram: Inductive Knowledge Graph Embedding via Relation Graphscited by 52
  12. 2023Demystifying Oversmoothing in Attention-Based Graph Neural Networkscited by 49
  13. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 46
  14. 2023TacticAI: an AI assistant for football tacticscited by 44
  15. 2025FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrievalcited by 42
  16. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  17. 2024SemlaFlow -- Efficient 3D Molecular Generation with Latent Attention and Equivariant Flow Matchingcited by 41
  18. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  19. 2023Searching Large Neighborhoods for Integer Linear Programs with Contrastive Learningcited by 37
  20. 2022Cell Attention Networkscited by 35
  21. 2023HINormer: Representation Learning On Heterogeneous Information Networks with Graph Transformercited by 35
  22. 2023Graph Positional and Structural Encodercited by 33
  23. 2024Kolmogorov-Arnold Graph Neural Networkscited by 32
  24. 2024Graph Transformers: A Surveycited by 32
  25. 2022MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initializationcited by 31
  26. 2023Inductive Graph Unlearningcited by 31
  27. 2022Direct Molecular Conformation Generationcited by 30
  28. 2022DropMessage: Unifying Random Dropping for Graph Neural Networkscited by 29
  29. 2022Graph Neural Networks with Adaptive Readoutscited by 29
  30. 2023Learning on Graphs with Out-of-Distribution Nodescited by 29
  31. 2023Generalized Simplicial Attention Neural Networkscited by 27
  32. 2024Circuit Complexity Bounds for RoPE-based Transformer Architecturecited by 26
  33. 2022Can Hybrid Geometric Scattering Networks Help Solve the Maximum Clique Problem?cited by 25
  34. 2022Graph Attention Retrospectivecited by 24
  35. 2023Towards Deep Attention in Graph Neural Networks: Problems and Remediescited by 24
  36. 2022Exploring Edge Disentanglement for Node Classificationcited by 23
  37. 2022Latent Graph Inference using Product Manifoldscited by 23
  38. 2024Self-Labeling the Job Shop Scheduling Problemcited by 23

50 most cited of 835 All 835 (JSON)