Graph External Attention Enhanced Transformer
Not on the map. Why?
Abstract (arXiv)
The Transformer architecture has recently gained considerable attention in the field of graph representation learning, as it naturally overcomes several limitations of Graph Neural Networks (GNNs) with customized attention mechanisms or positional and structural encodings. Despite making some progress, existing works tend to overlook external information of graphs, specifically the correlation between graphs. Intuitively, graphs with similar structures should have similar representations. Therefore, we propose Graph External Attention (GEA) -- a novel attention mechanism that leverages multiple external node/edge key-value units to capture inter-graph correlations implicitly. On this basis, we design an effective architecture called Graph External Attention Enhanced Transformer (GEAET), which integrates local structure and global interaction information for more comprehensive graph representations. Extensive experiments on benchmark datasets demonstrate that GEAET achieves state-of-the-art empirical performance. The source code is available for reproducibility at: https://github.com/icm1018/GEAET.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 49 of its 70 references that are in the map. First posted on arXiv 31 May 2024.
Builds on
49 of 70 references.
References · 49
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2017Graph Attention Networkscited by 8,923
- 2017Inductive Representation Learning on Large Graphscited by 6,320
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2018How Powerful are Graph Neural Networks?cited by 4,393
- 2017Neural Message Passing for Quantum Chemistrycited by 3,487
- 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
- 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
- 2019Strategies for Pre-training Graph Neural Networkscited by 971
- 2020PCT: Point cloud transformercited by 732
37 more
- 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 713
- 2020Benchmarking Graph Neural Networkscited by 680
- 2020A Generalization of Transformer Networks to Graphscited by 661
- 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 631
- 2018Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizoncited by 623
- 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 545
- 2020Principal Neighbourhood Aggregation for Graph Netscited by 478
- 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
- 2017Residual Gated Graph ConvNetscited by 346
- 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 333
- 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
- 2017Community Detection and Stochastic Block Modelscited by 319
- 2020Improving Graph Neural Network Expressivity via Subgraph Isomorphism Countingcited by 313
- 2021Combinatorial optimization and reasoning with graph neural networkscited by 268
- 2022Long Range Graph Benchmarkcited by 263
- 2021Weisfeiler and Lehman Go Cellular: CW Networkscited by 218
- 2017Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2veccited by 198
- 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 195
- 2022Structure-Aware Transformer for Graph Representation Learningcited by 194
- 2020Random Features Strengthen Graph Neural Networkscited by 186
- 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 177
- 2023Exphormer: Sparse Transformers for Graphscited by 170
- 2021From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awarenesscited by 155
- 2023Graph Inductive Biases in Transformers without Message Passingcited by 153
- 2019Relational Pooling for Graph Representationscited by 150
- 2021GraphiT: Encoding Graph Structure in Transformerscited by 147
- 2020Directional Graph Networkscited by 133
- 2021ETA Prediction with Graph Neural Networks in Google Mapscited by 130
- 2019Fake News Detection on Social Media using Geometric Deep Learningcited by 118
- 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 114
- 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
- 2022A Generalization of ViT/MLP-Mixer to Graphscited by 93
- 2023Where Did the Gap Go? Reassessing the Long-Range Graph Benchmarkcited by 85
- 2023DRew: Dynamically Rewired Message Passing with Delaycited by 82
- 2020PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterestcited by 75
- 2020Hierarchical Inter-Message Passing for Learning on Molecular Graphscited by 59
- 2019Learning Symbolic Physics with Graph Networkscited by 53
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.