mlsanityData to 14 Oct 2026

Transformers are Graph Neural Networks

arXiv 2506.22084 ↗First posted 27 Jun 2025cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

We establish connections between the Transformer architecture, originally introduced for natural language processing, and Graph Neural Networks (GNNs) for representation learning on graphs. We show how Transformers can be viewed as message passing GNNs operating on fully connected graphs of tokens, where the self-attention mechanism capture the relative importance of all tokens w.r.t. each-other, and positional encodings provide hints about sequential ordering or structure. Thus, Transformers are expressive set processing networks that learn relationships among input elements without being constrained by apriori graphs. Despite this mathematical connection to GNNs, Transformers are implemented via dense matrix operations that are significantly more efficient on modern hardware than sparse message passing. This leads to the perspective that Transformers are GNNs currently winning the hardware lottery.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 27 Jun 2025.

Builds on

29 of 37 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  7. 2017Graph Attention Networkscited by 8,923
  8. 2016Layer Normalizationcited by 5,669
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  10. 2018How Powerful are Graph Neural Networks?cited by 4,393
  11. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
17 more
  1. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  4. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  7. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  8. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  9. 2020On Layer Normalization in the Transformer Architecturecited by 922
  10. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 713
  11. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
  12. 2020A Generalization of Transformer Networks to Graphscited by 661
  13. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  14. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  15. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  16. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 166
  17. 2021ETA Prediction with Graph Neural Networks in Google Mapscited by 130

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 9.

29 papers.

Sort:

By area:

  1. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  2. 2025Learning for routing: A guided review of recent developments and future directionscited by 7
  3. 2025Molecular Machine Learning in Chemical Process Designcited by 5
  4. 2025Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvementscited by 3
  5. 2025A Graph Signal Processing Framework for Hallucination Detection in Large Language Modelscited by 3
  6. 2025A Design Co-Pilot for Task-Tailored Manipulatorscited by 2
  7. 2026Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphscited by 2
  8. 2025FieldFormer: Locality-Aware Transformers for Spatio-Temporal Modeling on Sparse Sensor Networkscited by 1
  9. 2025Learning what to say and how precisely: Efficient Communication via Differentiable Discrete Communication Learningcited by 1
  10. 2025Understanding the Failure Modes of Transformers through the Lens of Graph Neural Networkscited by 1
  11. 2026MPM-LLM4DSE: Reaching the Pareto Frontier in HLS with Multimodal Learning and LLM-Driven Explorationcited by 1
  12. 2025Latent Graph Learning in Generative Models of Neural Signalscited by 0
17 more
  1. 2025Dynamic Relational Priming Improves Transformer in Multivariate Time Seriescited by 0
  2. 2025Graph Variate Neural Networkscited by 0
  3. 2025Transformers Can Learn Connectivity in Some Graphs but Not Otherscited by 0
  4. 2025The Role of Feature Interactions in Graph-based Tabular Deep Learningcited by 0
  5. 2025Fighter: Unveiling the Graph Convolutional Nature of Transformers in Time Series Modelingcited by 0
  6. 2025Enhancing Graph Representations with Neighborhood-Contextualized Message-Passingcited by 0
  7. 2025Multi-Scale Protein Structure Modelling with Geometric Graph U-Netscited by 0
  8. 2025Solving Oversmoothing in GNNs via Nonlocal Message Passing: Algebraic Smoothing and Depth Scalabilitycited by 0
  9. 2025Iterative Compositional Data Generation for Robot Controlcited by 0
  10. 2025How Smoothing is N-simplicial Attention?cited by 0
  11. 2026Multi-fidelity graph-based neural networks architectures to learn Navier-Stokes solutions on non-parametrized 2D domainscited by 0
  12. 2026On the Runway Cascade of Transformers for Language Modelingcited by 0
  13. 2026Neural Network Quantum Field Theory from Transformer Architecturescited by 0
  14. 2026Transformers are Bayesian Networkscited by 0
  15. 2026Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformercited by 0
  16. 2026Unifying Dynamical Systems and Graph Theory to Mechanistically Understand Computation in Neural Networkscited by 0
  17. 2026GraphReview: Scientific Paper Evaluation via LLM-based Graph Evidence Expansioncited by 0