Scaling Graph Transformers: A Comparative Study of Sparse and Dense Attention
Not on the map. Why?
Abstract (arXiv)
Graphs have become a central representation in machine learning for capturing relational and structured data across various domains. Traditional graph neural networks often struggle to capture long-range dependencies between nodes due to their local structure. Graph transformers overcome this by using attention mechanisms that allow nodes to exchange information globally. However, there are two types of attention in graph transformers: dense and sparse. In this paper, we compare these two attention mechanisms, analyze their trade-offs, and highlight when to use each. We also outline current challenges and problems in designing attention for graph transformers.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 11 of its 14 references that are in the map. First posted on arXiv 24 Aug 2025.
Builds on
11 of 14 references.
References · 11
- 2017Attention Is All You Needcited by 60,361
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2017Neural Message Passing for Quantum Chemistrycited by 3,487
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
- 2020Rethinking Attention with Performerscited by 1,601
- 2020Benchmarking Graph Neural Networkscited by 680
- 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 631
- 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
- 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
- 2023NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classificationcited by 210
- 2023Exphormer: Sparse Transformers for Graphscited by 170
Built on by
Citing papers per year: 2025: 1.
1 paper.