A Bird's-Eye Tutorial of Graph Attention Architectures
Not on the map. Why?
Abstract (arXiv)
Graph Neural Networks (GNNs) have shown tremendous strides in performance for graph-structured problems especially in the domains of natural language processing, computer vision and recommender systems. Inspired by the success of the transformer architecture, there has been an ever-growing body of work on attention variants of GNNs attempting to advance the state of the art in many of these problems. Incorporating "attention" into graph mining has been viewed as a way to overcome the noisiness, heterogenity and complexity associated with graph-structured data as well as to encode soft-inductive bias. It is hence crucial and advantageous to study these variants from a bird's-eye view to assess their strengths and weaknesses. We provide a systematic and focused tutorial centered around attention based GNNs in a hope to benefit researchers dealing with graph-structured problems. Our tutorial looks at GNN variants from the point of view of the attention function and iteratively builds the reader's understanding of different graph attention variants.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 48 of its 68 references that are in the map. First posted on arXiv 6 Jun 2022.
Builds on
48 of 68 references.
References · 48
- 2017Attention Is All You Needcited by 59,932
- 2017Graph Attention Networkscited by 8,854
- 2017Inductive Representation Learning on Large Graphscited by 6,284
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
- 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
- 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
- 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
- 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
- 2018Pitfalls of Graph Neural Network Evaluationcited by 994
- 2021E(n) Equivariant Graph Neural Networkscited by 951
36 more
- 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 878
- 2021How Attentive are Graph Attention Networks?cited by 818
- 2019Heterogeneous Graph Attention Networkcited by 732
- 2020Benchmarking Graph Neural Networkscited by 672
- 2020A Generalization of Transformer Networks to Graphscited by 653
- 2018Junction Tree Variational Autoencoder for Molecular Graph Generationcited by 647
- 2020Heterogeneous Graph Transformercited by 581
- 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
- 2019Provably Powerful Graph Networkscited by 385
- 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
- 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
- 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
- 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
- 2021Combinatorial optimization and reasoning with graph neural networkscited by 267
- 2017A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applicationscited by 260
- 2020A Note on Over-Smoothing for Graph Neural Networkscited by 245
- 2019Hyperbolic Graph Neural Networkscited by 244
- 2019Position-aware Graph Neural Networkscited by 203
- 2018GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphscited by 164
- 2019What Can Neural Networks Reason About?cited by 155
- 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
- 2015A Survey of Heterogeneous Information Network Analysiscited by 109
- 2021Scaling Up Graph Neural Networks Via Graph Coarseningcited by 80
- 2019Hyperbolic Graph Attention Networkcited by 69
- 2021Graph Learning based Recommender Systems: A Reviewcited by 59
- 2020HittER: Hierarchical Transformers for Knowledge Graph Embeddingscited by 54
- 2018Attention Models in Graphs: A Surveycited by 41
- 2021Graph Pointer Neural Networkscited by 27
- 2021Learning Conjoint Attentions for Graph Neural Netscited by 26
- 2019GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animationcited by 21
- 2021MELONS: generating melody with long-term structure using transformers and structure graphcited by 18
- 2021Neuromorphic Camera Denoising using Graph Neural Network-driven Transformerscited by 15
- 2021Self-attention Presents Low-dimensional Knowledge Graph Embeddings for Link Predictioncited by 8
- 2020Segmented Graph-Bert for Graph Instance Modelingcited by 3
- 2020Graph Neural Distance Metric Learning with Graph-Bertcited by 2
- 2020G5: A Universal GRAPH-BERT for Graph-to-Graph Transfer and Apocalypse Learningcited by 1
Built on by
Citing papers per year: 2022: 1.
1 paper.