mlsanityData to 7 Oct 2026

Transformer for Graphs: An Overview from Architecture Perspective

arXiv 2202.08455 ↗First posted 17 Feb 2022cs.LG cs.AIcited by 107

Not on the map. Why?

Abstract (arXiv)

Recently, Transformer model, which has achieved great success in many artificial intelligence fields, has demonstrated its great potential in modeling graph-structured data. Till now, a great variety of Transformers has been proposed to adapt to the graph-structured data. However, a comprehensive literature review and systematical evaluation of these Transformer variants for graphs are still unavailable. It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks. In this survey, we provide a comprehensive review of various Graph Transformer models from the architectural design perspective. We first disassemble the existing models and conclude three typical ways to incorporate the graph information into the vanilla Transformer: 1) GNNs as Auxiliary Modules, 2) Improved Positional Embedding from Graphs, and 3) Improved Attention Matrix from Graphs. Furthermore, we implement the representative components in three groups and conduct a comprehensive comparison on various kinds of famous graph data benchmarks to investigate the real performance gain of each component. Our experiments confirm the benefits of current graph-specific modules on Transformer and reveal their advantages on different kinds of graph tasks.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

35 of 51 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2016Layer Normalizationcited by 5,579
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  8. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  9. 2018Link Prediction Based on Graph Neural Networkscited by 829
  10. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  11. 2020A Generalization of Transformer Networks to Graphscited by 653
  12. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
23 more
  1. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
  2. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  3. 2019Provably Powerful Graph Networkscited by 385
  4. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  5. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  6. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  7. 2021Mesh Graphormercited by 264
  8. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
  9. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  10. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 193
  11. 2022TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentialscited by 179
  12. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 176
  13. 2021GraphiT: Encoding Graph Structure in Transformerscited by 146
  14. 2019Deep Learning for Audio Signal Processingcited by 92
  15. 2019Graph Transformer for Graph-to-Sequence Learningcited by 82
  16. 2021Gophormer: Ego-Graph Transformer for Node Classificationcited by 63
  17. 2020Interpretable Rumor Detection in Microblogs by Attending to User Interactionscited by 44
  18. 2019Universal Graph Transformer Self-Attention Networkscited by 41
  19. 2020Convolutional Kernel Networks for Graph-Structured Datacited by 38
  20. 2019Modeling Graph Structure in Transformer for Better AMR-to-Text Generationcited by 35
  21. 2019Self-Attention with Structural Position Representationscited by 21
  22. 2019GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animationcited by 21
  23. 2020Deep Graph Neural Networks with Shallow Subgraph Samplerscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 29; 2024: 40; 2025: 18; 2026: 11.

107 papers.

Sort:

By area:

  1. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  2. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  3. 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
  4. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 131
  5. 2022A Generalization of ViT/MLP-Mixer to Graphscited by 93
  6. 2022Trustworthy Graph Neural Networks: Aspects, Methods and Trendscited by 88
  7. 2023Where Did the Gap Go? Reassessing the Long-Range Graph Benchmarkcited by 83
  8. 2024Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classificationcited by 83
  9. 2023Attending to Graph Transformerscited by 81
  10. 2024Discrete-state Continuous-time Diffusion for Graph Generationcited by 73
  11. 2024Position: Topological Deep Learning is the New Frontier for Relational Learningcited by 65
  12. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
38 more
  1. 2023Transformers Meet Directed Graphscited by 39
  2. 2024Graph Transformers: A Surveycited by 32
  3. 2022Graph Lifelong Learning: A Surveycited by 30
  4. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  5. 2023Learning Subpocket Prototypes for Generalizable Structure-based Drug Designcited by 25
  6. 2023Learning Probabilistic Symmetrization for Architecture Agnostic Equivariancecited by 25
  7. 2024HAMLET: Graph Transformer Neural Operator for Partial Differential Equationscited by 24
  8. 2025Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellencecited by 24
  9. 2022Transformers over Directed Acyclic Graphscited by 23
  10. 2023Devil's on the Edges: Selective Quad Attention for Scene Graph Generationcited by 23
  11. 2024MART: MultiscAle Relational Transformer Networks for Multi-agent Trajectory Predictioncited by 21
  12. 2022BayesFormer: Transformer with Uncertainty Estimationcited by 19
  13. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  14. 2024Graph Sparsification via Mixture of Graphscited by 17
  15. 2023Enhancing Graph Transformers with Hierarchical Distance Structural Encodingcited by 16
  16. 2024Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniquescited by 16
  17. 2025Graph Generative Pre-trained Transformercited by 16
  18. 2023Dynamic Graph Representation Learning with Neural Networks: A Surveycited by 15
  19. 2023GraphGPT: Generative Pre-trained Graph Eulerian Transformercited by 15
  20. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  21. 2024Graph Neural Networks in Histopathology: Emerging Trends and Future Directionscited by 13
  22. 2025A Survey of Graph Transformers: Architectures, Theories and Applicationscited by 12
  23. 2024Attending to Topological Spaces: The Cellular Transformercited by 11
  24. 2023Neural Atoms: Propagating Long-range Interaction in Molecular Graphs through Efficient Communication Channelcited by 10
  25. 2024A Graph is Worth K Words: Euclideanizing Graph using Pure Transformercited by 10
  26. 2024Gradformer: Graph Transformer with Exponential Decaycited by 10
  27. 2023MulGT: Multi-task Graph-Transformer with Task-aware Knowledge Injection and Domain Knowledge-driven Pooling for Whole Slide Image Analysiscited by 9
  28. 2024STGformer: Efficient Spatiotemporal Graph Transformer for Traffic Forecastingcited by 9
  29. 2023Graph Agent: Explicit Reasoning Agent for Graphscited by 8
  30. 2024Cluster-wise Graph Transformer with Dual-granularity Kernelized Attentioncited by 8
  31. 2022Not All Neighbors Are Worth Attending to: Graph Selective Attention Networks for Semi-supervised Learningcited by 6
  32. 2024Graph Language Modelscited by 6
  33. 2024Exploring the Limitations of Mamba in COPY and CoT Reasoningcited by 6
  34. 2023CLGT: A Graph Transformer for Student Performance Prediction in Collaborative Learningcited by 5
  35. 2023Tailoring Self-Attention for Graph via Rooted Subtreescited by 5
  36. 2024Powerformer: A Section-adaptive Transformer for Power Flow Adjustmentcited by 5
  37. 2024KnowFormer: Revisiting Transformers for Knowledge Graph Reasoningcited by 5
  38. 2025Spectro-Riemannian Graph Neural Networkscited by 5

50 most cited of 107 All 107 (JSON)