mlsanityData to 7 Oct 2026

Attending to Graph Transformers

arXiv 2302.04181 ↗First posted 8 Feb 2023cs.LG cs.AI cs.NEcited by 81

Not on the map. Why?

Abstract (arXiv)

Recently, transformer architectures for graphs emerged as an alternative to established techniques for machine learning with graphs, such as (message-passing) graph neural networks. So far, they have shown promising empirical results, e.g., on molecular prediction datasets, often attributed to their ability to circumvent graph neural networks' shortcomings, such as over-smoothing and over-squashing. Here, we derive a taxonomy of graph transformer architectures, bringing some order to this emerging field. We overview their theoretical properties, survey structural and positional encodings, and discuss extensions for important graph classes, e.g., 3D molecular graphs. Empirically, we probe how well graph transformers can recover various graph properties, how well they can deal with heterophilic graphs, and to what extent they prevent over-squashing. Further, we outline open challenges and research direction to stimulate future work. Our code is available at https://github.com/luis-mueller/probing-graph-transformers.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 73 of its 95 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

73 of 95 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Graph Attention Networkscited by 8,854
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  10. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 878
  11. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
38 more
  1. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  2. 2020Benchmarking Graph Neural Networkscited by 672
  3. 2020A Generalization of Transformer Networks to Graphscited by 653
  4. 2020A Survey on Visual Transformercited by 653
  5. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  6. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  7. 2020Heterogeneous Graph Transformercited by 581
  8. 2019Multi-scale Attributed Node Embeddingcited by 543
  9. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 471
  10. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  11. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 438
  12. 2021A Survey of Transformerscited by 399
  13. 2019Provably Powerful Graph Networkscited by 385
  14. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
  15. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  16. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
  17. 2020Improving Graph Neural Network Expressivity via Subgraph Isomorphism Countingcited by 312
  18. 2017Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphscited by 307
  19. 2021Mesh Graphormercited by 264
  20. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  21. 2021Weisfeiler and Lehman Go Cellular: CW Networkscited by 218
  22. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 193
  23. 2022Structure-Aware Transformer for Graph Representation Learningcited by 191
  24. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  25. 2022Graph neural networks for materials science and chemistrycited by 181
  26. 2022TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentialscited by 179
  27. 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 176
  28. 2019On the equivalence between graph isomorphism testing and function approximation with GNNscited by 172
  29. 2019Understanding Attention and Generalization in Graph Neural Networkscited by 169
  30. 2022Revisiting Heterophily For Graph Neural Networkscited by 169
  31. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 163
  32. 2020Machine Learning on Graphs: A Model and Comprehensive Taxonomycited by 156
  33. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  34. 2019Relational Pooling for Graph Representationscited by 150
  35. 2021GraphiT: Encoding Graph Structure in Transformerscited by 146
  36. 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
  37. 2020Directional Graph Networkscited by 132
  38. 2021Weisfeiler and Leman go Machine Learning: The Story so farcited by 121

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 38; 2025: 19; 2026: 7.

81 papers.

Sort:

By area:

  1. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  2. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
  3. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  4. 2024Position: Graph Foundation Models are Already Herecited by 84
  5. 2024Graph Mamba: Towards Learning on Graphs with State Space Modelscited by 84
  6. 2024Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classificationcited by 83
  7. 2024Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insightscited by 44
  8. 2024Transformers are Universal In-context Learnerscited by 42
  9. 2024On the Scalability of GNNs for Molecular Graphscited by 36
  10. 2024VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connectionscited by 34
  11. 2024Graph Transformers: A Surveycited by 32
  12. 2025Relational Graph Transformercited by 30
38 more
  1. 2023Probabilistically Rewired Message-Passing Neural Networkscited by 26
  2. 2023Learning Probabilistic Symmetrization for Architecture Agnostic Equivariancecited by 25
  3. 2025Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellencecited by 24
  4. 2023Adaptive Message Passing: A General Framework to Mitigate Oversmoothing, Oversquashing, and Underreachingcited by 23
  5. 2023Exposition on over-squashing problem on GNNs: Current Methods, Benchmarks and Challengescited by 22
  6. 2024Comparing Graph Transformers via Positional Encodingscited by 20
  7. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  8. 2024Distinguished In Uniform: Self Attention Vs. Virtual Nodescited by 19
  9. 2024Learning Graph Quantized Tokenizerscited by 19
  10. 2023Dynamic Graph Representation Learning with Neural Networks: A Surveycited by 15
  11. 2023Expressive Sign Equivariant Networks for Spectral Geometric Learningcited by 15
  12. 2023GraphGPT: Generative Pre-trained Graph Eulerian Transformercited by 15
  13. 2024Probabilistic Graph Rewiring via Virtual Nodescited by 15
  14. 2024MANTRA: The Manifold Triangulations Assemblagecited by 12
  15. 2024Even Sparser Graph Transformerscited by 12
  16. 2025Towards Mechanistic Interpretability of Graph Transformers via Attention Graphscited by 12
  17. 2025A Survey of Graph Transformers: Architectures, Theories and Applicationscited by 12
  18. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  19. 2024Towards Principled Graph Transformerscited by 11
  20. 2024Node Identifiers: Compact, Discrete Representations for Efficient Graph Learningcited by 10
  21. 2023Graph Transformers for Large Graphscited by 9
  22. 2024Multi-Grid Graph Neural Networks with Self-Attention for Computational Mechanicscited by 9
  23. 2024STGformer: Efficient Spatiotemporal Graph Transformer for Traffic Forecastingcited by 9
  24. 2025Message-Passing State-Space Models: Improving Graph Learning with Modern Sequence Modelingcited by 9
  25. 2024Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networkscited by 8
  26. 2024On the Theoretical Expressive Power and the Design Space of Higher-Order Graph Transformerscited by 8
  27. 2024Aligning Transformers with Weisfeiler-Lemancited by 8
  28. 2025The quest for the GRAph Level autoEncoder (GRALE)cited by 8
  29. 2024DE-HNN: An effective neural model for Circuit Netlist representationcited by 7
  30. 2024Representation Learning on Heterophilic Graph with Directional Neighborhood Attentioncited by 6
  31. 2024Diss-l-ECT: Dissecting Graph Data with Local Euler Characteristic Transformscited by 6
  32. 2024Best of Both Worlds: Advantages of Hybrid Graph Sequence Modelscited by 6
  33. 2023Tailoring Self-Attention for Graph via Rooted Subtreescited by 5
  34. 2024Future Directions in the Theory of Graph Machine Learningcited by 5
  35. 2025Transformers through the lens of support-preserving maps between measurescited by 5
  36. 2024Automatic Graph Topology-Aware Transformercited by 4
  37. 2025Training Transformers for Mesh-Based Simulationscited by 4
  38. 2024Flexible Diffusion Scopes with Parameterized Laplacian for Heterophilic Graph Learningcited by 3

50 most cited of 81 All 81 (JSON)