mlsanityData to 7 Oct 2026

A Survey of Graph Transformers: Architectures, Theories and Applications

arXiv 2502.16533 ↗First posted 23 Feb 2025cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Graph Transformers (GTs) have demonstrated a strong capability in modeling graph structures by addressing the intrinsic limitations of graph neural networks (GNNs), such as over-smoothing and over-squashing. Recent studies have proposed diverse architectures, enhanced explainability, and practical applications for Graph Transformers. In light of these rapid developments, we conduct a comprehensive review of Graph Transformers, covering aspects such as their architectures, theoretical foundations, and applications. In this survey, we first categorize the architecture of Graph Transformers according to their strategies for processing structural information, including graph tokenization, positional encoding, structure-aware attention, and model ensemble. Then, from the theoretical perspective, we examine the expressivity of Graph Transformers in various discussed architectures and contrast them with other advanced graph learning algorithms to discover their connections. For applications, we organize the literature around four graph organization forms, from relational, geometric, dynamic to heterogeneous. A Practical Guidance table then maps architectural components to these graph forms by adoption frequency, so practitioners can narrow down which design families to consider for a given input structure. Lastly, we will discuss the current challenges and prospective directions in Graph Transformers for potential future research.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 164 of its 307 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

164 of 307 references.

References · 164

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2017Graph Attention Networkscited by 8,854
  6. 2016Layer Normalizationcited by 5,579
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2018How Powerful are Graph Neural Networks?cited by 4,355
  10. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  11. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  12. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2016Modeling Context in Referring Expressionscited by 1,054
  4. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
  7. 2021E(n) Equivariant Graph Neural Networkscited by 951
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  9. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 878
  10. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  11. 2020Simple and Deep Graph Convolutional Networkscited by 857
  12. 2017Convolutional 2D Knowledge Graph Embeddingscited by 729
  13. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  14. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  15. 2020A Generalization of Transformer Networks to Graphscited by 653
  16. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  17. 2017Convolutional Sequence to Sequence Learningcited by 550
  18. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 544
  19. 2019Multi-scale Attributed Node Embeddingcited by 543
  20. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  21. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  22. 2022Transformers in Time Series: A Surveycited by 489
  23. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  24. 2019GMAN: A Graph Multi-Attention Network for Traffic Predictioncited by 433
  25. 2018Invariant and Equivariant Graph Networkscited by 339
  26. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  27. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  28. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
  29. 2020Learning from Protein Structure with Geometric Vector Perceptronscited by 308
  30. 2018GuacaMol: Benchmarking Models for De Novo Molecular Designcited by 294
  31. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  32. 2018Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Modelscited by 270
  33. 2017DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoningcited by 265
  34. 2021Mesh Graphormercited by 264
  35. 2022Long Range Graph Benchmarkcited by 258
  36. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  37. 2019Evaluating Protein Transfer Learning with TAPEcited by 229
  38. 2020Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Predictioncited by 218

50 most cited of 164 All 164 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 3.

12 papers.

  1. 2025DiffSpectra: Molecular Structure Elucidation from Spectra using Diffusion Modelscited by 5
  2. 2025Universally Invariant Learning in Equivariant GNNscited by 3
  3. 2025Geometric Mixture Models for Electrolyte Conductivity Predictioncited by 2
  4. 2026Teaching LLMs to See Graphs: Unifying Text and Structural Reasoningcited by 2
  5. 2025Transformer-Empowered Actor-Critic Reinforcement Learning for Sequence-Aware Service Function Chain Partitioningcited by 1
  6. 2025ASD Classification on Dynamic Brain Connectome using Temporal Random Walk with Transformer-based Dynamic Network Embeddingcited by 0
  7. 2025Unifying Structural Proximity and Equivalence for Enhanced Dynamic Network Embeddingcited by 0
  8. 2025PAC-Bayesian Generalization Bounds for Graph Convolutional Networks on Inductive Node Classificationcited by 0
  9. 2025DARTS-GT: Differentiable Architecture Search for Graph Transformers with Quantifiable Instance-Specific Interpretability Analysiscited by 0
  10. 20253D-GSRD: 3D Molecular Graph Auto-Encoder with Selective Re-mask Decodingcited by 0
  11. 2026Agentic Fusion of Large Atomic and Language Models to Accelerate Superconductor Discoverycited by 0
  12. 2026Different Statistical Perspectives for Understanding Generalisation in Graph Neural Networkscited by 0