mlsanityData to 7 Oct 2026

A Generalization of Transformer Networks to Graphs

arXiv 2012.09699 ↗First posted 17 Dec 2020cs.LGcited by 653

Not on the map. Why?

Cited by 28 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a generalization of transformer neural network architecture for arbitrary graphs. The original transformer was designed for Natural Language Processing (NLP), which operates on fully connected graphs representing all connections between the words in a sequence. Such architecture does not leverage the graph connectivity inductive bias, and can perform poorly when the graph topology is important and has not been encoded into the node features. We introduce a graph transformer with four new properties compared to the standard model. First, the attention mechanism is a function of the neighborhood connectivity for each node in the graph. Second, the positional encoding is represented by the Laplacian eigenvectors, which naturally generalize the sinusoidal positional encodings often used in NLP. Third, the layer normalization is replaced by a batch normalization layer, which provides faster training and better generalization performance. Finally, the architecture is extended to edge feature representation, which can be critical to tasks s.a. chemistry (bond type) or link prediction (entity relationship in knowledge graphs). Numerical experiments on a graph benchmark demonstrate the performance of the proposed graph transformer architecture. This work closes the gap between the original transformer, which was designed for the limited case of line graphs, and graph neural networks, that can work with arbitrary graphs. As our architecture is simple and generic, we believe it can be used as a black box for future applications that wish to consider transformer and graphs.

Key facts. Cited by 653 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 90; 2023: 135; 2024: 166; 2025: 138; 2026: 96.

653 papers.

Sort:

By area:

  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  2. 2021How Attentive are Graph Attention Networks?cited by 818
  3. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  4. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  5. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  6. 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
  7. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  8. 2022Long Range Graph Benchmarkcited by 258
  9. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  10. 2023Language is All a Graph Needscited by 219
  11. 2023NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classificationcited by 207
  12. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 201
38 more
  1. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 193
  2. 2022Structure-Aware Transformer for Graph Representation Learningcited by 191
  3. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  4. 2024LLaGA: Large Language and Graph Assistantcited by 177
  5. 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 176
  6. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
  7. 2023Exphormer: Sparse Transformers for Graphscited by 166
  8. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  9. 2023Graph Inductive Biases in Transformers without Message Passingcited by 150
  10. 2021GraphiT: Encoding Graph Structure in Transformerscited by 146
  11. 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
  12. 2023Temporal Graph Benchmark for Machine Learning on Temporal Graphscited by 137
  13. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  14. 2023PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Predictioncited by 135
  15. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 132
  16. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 131
  17. 2023Rethinking the Expressive Power of GNNs via Graph Biconnectivitycited by 119
  18. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 113
  19. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 110
  20. 2022Transformer for Graphs: An Overview from Architecture Perspectivecited by 107
  21. 2022FoSR: First-order spectral rewiring for addressing oversquashing in GNNscited by 104
  22. 2021Position Information in Transformers: An Overviewcited by 103
  23. 2023Specformer: Spectral Graph Neural Networks Meet Transformerscited by 102
  24. 2023MiDi: Mixed Graph and 3D Denoising Diffusion for Molecule Generationcited by 96
  25. 2022Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networkscited by 94
  26. 2022A Generalization of ViT/MLP-Mixer to Graphscited by 93
  27. 2022Brain Network Transformercited by 92
  28. 2021Matrix Encoding Networks for Neural Combinatorial Optimizationcited by 89
  29. 2023DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusioncited by 85
  30. 2021Beltrami Flow and Neural Diffusion on Graphscited by 84
  31. 2023Where Did the Gap Go? Reassessing the Long-Range Graph Benchmarkcited by 83
  32. 2024Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classificationcited by 83
  33. 2023Attending to Graph Transformerscited by 81
  34. 2023DRew: Dynamically Rewired Message Passing with Delaycited by 81
  35. 2022DiffWire: Inductive Graph Rewiring via the Lov'asz Boundcited by 79
  36. 2023Generative Diffusion Models on Graphs: Methods and Applicationscited by 79
  37. 2024Variational Flow Matching for Graph Generationcited by 78
  38. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77

50 most cited of 653 All 653 (JSON)