mlsanityData to 7 Oct 2026

Pure Transformers are Powerful Graph Learners

arXiv 2207.02505 ↗First posted 6 Jul 2022cs.LG cs.AIcited by 188

Not on the map. Why?

Abstract (arXiv)

We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice. Given a graph, we simply treat all nodes and edges as independent tokens, augment them with token embeddings, and feed them to a Transformer. With an appropriate choice of token embeddings, we prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers, which is already more expressive than all message-passing Graph Neural Networks (GNN). When trained on a large-scale graph dataset (PCQM4Mv2), our method coined Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results compared to Transformer variants with sophisticated graph-specific inductive bias. Our implementation is available at https://github.com/jw9730/tokengt.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Builds on 73 of its 97 references that are in the map. First posted on arXiv 6 Jul 2022.

Builds on

73 of 97 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2017Graph Attention Networkscited by 8,854
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2018How Powerful are Graph Neural Networks?cited by 4,355
  10. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2020Rethinking Attention with Performerscited by 1,584
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2017Deep Setscited by 1,394
  5. 2016Group Equivariant Convolutional Networkscited by 1,154
  6. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  8. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  9. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  10. 2016Deep Networks with Stochastic Depthcited by 973
  11. 2020On Layer Normalization in the Transformer Architecturecited by 906
  12. 2021How Attentive are Graph Attention Networks?cited by 818
  13. 2022A Generalist Agentcited by 802
  14. 2020Efficient Transformers: A Surveycited by 784
  15. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  16. 2020Benchmarking Graph Neural Networkscited by 672
  17. 2020A Generalization of Transformer Networks to Graphscited by 653
  18. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  19. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  20. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  21. 2019Multi-scale Attributed Node Embeddingcited by 543
  22. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  23. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  24. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  25. 2019Provably Powerful Graph Networkscited by 385
  26. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  27. 2021Random Feature Attentioncited by 350
  28. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  29. 2018Invariant and Equivariant Graph Networkscited by 339
  30. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  31. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  32. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  33. 2016Steerable CNNscited by 298
  34. 2021Mesh Graphormercited by 264
  35. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  36. 2020A Note on Over-Smoothing for Graph Neural Networkscited by 245
  37. 2020Can Graph Neural Networks Count Substructures?cited by 235
  38. 2014Efficient Object Localization Using Convolutional Networkscited by 226

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 54; 2024: 58; 2025: 35; 2026: 26.

188 papers.

Sort:

By area:

  1. 2023Language is All a Graph Needscited by 219
  2. 2023Graph Inductive Biases in Transformers without Message Passingcited by 150
  3. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 131
  4. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  5. 2024Position: Graph Foundation Models are Already Herecited by 84
  6. 2024Graph Mamba: Towards Learning on Graphs with State Space Modelscited by 84
  7. 2023Where Did the Gap Go? Reassessing the Long-Range Graph Benchmarkcited by 83
  8. 2023Attending to Graph Transformerscited by 81
  9. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77
  10. 2024Transformers, parallel computation, and logarithmic depthcited by 75
  11. 2024Discrete-state Continuous-time Diffusion for Graph Generationcited by 73
  12. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
38 more
  1. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  2. 2023On the Connection Between MPNN and Graph Transformercited by 60
  3. 2022Does GNN Pretraining Help Molecular Representation?cited by 59
  4. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  5. 2025Position: Graph Learning Will Lose Relevance Due To Poor Benchmarkscited by 52
  6. 2025PyG 2.0: Scalable Learning on Real World Graphscited by 49
  7. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  8. 2023Transformers Meet Directed Graphscited by 39
  9. 2024Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Spacecited by 39
  10. 2022Explaining the Explainers in Graph Neural Networks: a Comparative Studycited by 34
  11. 2024VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connectionscited by 34
  12. 2024Graph Transformers: A Surveycited by 32
  13. 2023On the Stability of Expressive Positional Encodings for Graphscited by 31
  14. 2023WL meet VCcited by 30
  15. 2025Relational Graph Transformercited by 30
  16. 2022Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Taskscited by 29
  17. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  18. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  19. 2023Learning Probabilistic Symmetrization for Architecture Agnostic Equivariancecited by 25
  20. 2023Automated 3D Pre-Training for Molecular Property Predictioncited by 24
  21. 2024Unifying Generation and Prediction on Graphs with Latent Graph Diffusioncited by 24
  22. 2023Highly Accurate Quantum Chemical Property Prediction with Uni-Mol+cited by 22
  23. 2022Graph Neural Networks for Moleculescited by 21
  24. 2023Calibrating Transformers via Sparse Gaussian Processescited by 21
  25. 2025Learning Efficient Positional Encodings with Graph Neural Networkscited by 21
  26. 2022A Practical, Progressively-Expressive GNNcited by 20
  27. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  28. 2023Embedding in Recommender Systems: A Surveycited by 19
  29. 2024PowerGraph: A power grid benchmark dataset for graph neural networkscited by 19
  30. 2024Distinguished In Uniform: Self Attention Vs. Virtual Nodescited by 19
  31. 2024Learning Graph Quantized Tokenizerscited by 19
  32. 2024Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detectioncited by 18
  33. 2022Astronomia ex machina: a history, primer, and outlook on neural networks in astronomycited by 17
  34. 2023LPFormer: An Adaptive Graph Transformer for Link Predictioncited by 17
  35. 2025Graph Generative Pre-trained Transformercited by 16
  36. 2023Learning Situation Hyper-Graphs for Video Question Answeringcited by 15
  37. 2023Curve Your Attention: Mixed-Curvature Transformers for Graph Representation Learningcited by 15
  38. 2023Expressive Sign Equivariant Networks for Spectral Geometric Learningcited by 15

50 most cited of 188 All 188 (JSON)