mlsanityData to 7 Oct 2026

Do Transformers Really Perform Bad for Graph Representation?

arXiv 2106.05234 ↗First posted 9 Jun 2021cs.LG cs.AIcited by 318

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance on popular leaderboards of graph-level prediction compared to mainstream GNN variants. Therefore, it remains a mystery how Transformers could perform well for graph representation learning. In this paper, we solve this mystery by presenting Graphormer, which is built upon the standard Transformer architecture, and could attain excellent results on a broad range of graph representation learning tasks, especially on the recent OGB Large-Scale Challenge. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model. To this end, we propose several simple yet effective structural encoding methods to help Graphormer better model graph-structured data. Besides, we mathematically characterize the expressive power of Graphormer and exhibit that with our ways of encoding the structural information of graphs, many popular GNN variants could be covered as the special cases of Graphormer.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). Builds on 48 of its 65 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

48 of 65 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  9. 2017Graph Attention Networkscited by 8,854
  10. 2017Inductive Representation Learning on Large Graphscited by 6,284
  11. 2018How Powerful are Graph Neural Networks?cited by 4,355
  12. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
36 more
  1. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2018Self-Attention with Relative Position Representationscited by 1,112
  5. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2020Benchmarking Graph Neural Networkscited by 672
  8. 2020A Generalization of Transformer Networks to Graphscited by 653
  9. 2020Heterogeneous Graph Transformercited by 581
  10. 2019Graph Transformer Networkscited by 520
  11. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
  12. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  13. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  14. 2019Provably Powerful Graph Networkscited by 385
  15. 2017Residual Gated Graph ConvNetscited by 341
  16. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  17. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  18. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
  19. 2019Position-aware Graph Neural Networkscited by 203
  20. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  21. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 176
  22. 2020Directional Graph Networkscited by 132
  23. 2021Accurate Learning of Graph Representations with Graph Multiset Poolingcited by 120
  24. 2020GraphNorm: A Principled Approach to Accelerating Graph Neural Network Trainingcited by 116
  25. 2018Multi-Hop Knowledge Graph Reasoning with Reward Shapingcited by 102
  26. 2020Molecule Attention Transformercited by 92
  27. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  28. 2019Graph Transformer for Graph-to-Sequence Learningcited by 82
  29. 2021Language-Agnostic Representation Learning of Source Code from Structure and Contextcited by 63
  30. 2017Learning Graph-Level Representation for Drug Discoverycited by 43
  31. 2020Graph convolutions that can finally model local structurecited by 31
  32. 2021SPAGAN: Shortest Path Graph Attention Networkcited by 28
  33. 2021How could Neural Networks understand Programs?cited by 26
  34. 2019Path-Augmented Graph Transformer Networkcited by 22
  35. 2021LazyFormer: Self Attention with Lazy Updatecited by 16
  36. 2020Breaking the Expressive Bottlenecks of Graph Neural Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 82; 2023: 78; 2024: 52; 2025: 32; 2026: 51.

318 papers.

Sort:

By area:

  1. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  2. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
  3. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  4. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  5. 2023Exphormer: Sparse Transformers for Graphscited by 166
  6. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 163
  7. 2021From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awarenesscited by 154
  8. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  9. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 132
  10. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 131
  11. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  12. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 113
38 more
  1. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  2. 2023GraphText: Graph Reasoning in Text Spacecited by 110
  3. 2022Transformer for Graphs: An Overview from Architecture Perspectivecited by 107
  4. 2023Specformer: Spectral Graph Neural Networks Meet Transformerscited by 102
  5. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  6. 2022The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalystscited by 95
  7. 2022Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networkscited by 94
  8. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 90
  9. 2023DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusioncited by 85
  10. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77
  11. 2022Hierarchical Graph Transformer with Adaptive Node Samplingcited by 72
  12. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  13. 2023Energy Transformercited by 69
  14. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  15. 2022GRPE: Relative Positional Encoding for Graph Transformercited by 67
  16. 2021Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction predictioncited by 66
  17. 2022MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Designcited by 66
  18. 2021Gophormer: Ego-Graph Transformer for Node Classificationcited by 63
  19. 2022Benchmarking Graphormer on Large-Scale Molecular Modeling Datasetscited by 63
  20. 2022Video Graph Transformer for Video Question Answeringcited by 63
  21. 2023On the Connection Between MPNN and Graph Transformercited by 60
  22. 2022Does GNN Pretraining Help Molecular Representation?cited by 59
  23. 2023CktGNN: Circuit Graph Neural Network for Electronic Design Automationcited by 55
  24. 2022DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotationscited by 54
  25. 2022Multimodal learning with graphscited by 54
  26. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  27. 2023InGram: Inductive Knowledge Graph Embedding via Relation Graphscited by 50
  28. 2022Message passing all the way upcited by 47
  29. 2022Spherical Channels for Modeling Atomic Interactionscited by 47
  30. 2023Ewald-based Long-Range Message Passing for Molecular Graphscited by 46
  31. 2021Simple GNN Regularisation for 3D Molecular Property Prediction & Beyondcited by 44
  32. 2021Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphscited by 44
  33. 2023GNNDelete: A General Strategy for Unlearning in Graph Neural Networkscited by 43
  34. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  35. 2023Transformers Meet Directed Graphscited by 39
  36. 2021From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformerscited by 38
  37. 2022Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrastcited by 37
  38. 2022Energy-Motivated Equivariant Pretraining for 3D Molecular Graphscited by 37

50 most cited of 318 All 318 (JSON)