# Transformers are Graph Neural Networks

arXiv: https://arxiv.org/abs/2506.22084 · HTML: https://mlsanity.com/paper/2506.22084/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Flex Attention: A Programming Model for Generating Optimized Attention Kernels](https://mlsanity.com/paper/2412.05496/) (2024)
- [On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology](https://mlsanity.com/paper/2302.02941/) (2023)
- [ETA Prediction with Graph Neural Networks in Google Maps](https://mlsanity.com/paper/2108.11482/) (2021)

## Built on by (most cited first, 29 of 29)

- [VideoNSA: Native Sparse Attention Scales Video Understanding](https://mlsanity.com/paper/2510.02295/) (2025)
- [Learning for routing: A guided review of recent developments and future directions](https://mlsanity.com/paper/2507.00218/) (2025)
- [Molecular Machine Learning in Chemical Process Design](https://mlsanity.com/paper/2508.20527/) (2025)
- [Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvements](https://mlsanity.com/paper/2507.21040/) (2025)
- [A Graph Signal Processing Framework for Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2510.19117/) (2025)
- [A Design Co-Pilot for Task-Tailored Manipulators](https://mlsanity.com/paper/2509.13077/) (2025)
- [Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs](https://mlsanity.com/paper/2609.10490/) (2026)
- [FieldFormer: Locality-Aware Transformers for Spatio-Temporal Modeling on Sparse Sensor Networks](https://mlsanity.com/paper/2510.03589/) (2025)
- [Learning what to say and how precisely: Efficient Communication via Differentiable Discrete Communication Learning](https://mlsanity.com/paper/2511.01554/) (2025)
- [Understanding the Failure Modes of Transformers through the Lens of Graph Neural Networks](https://mlsanity.com/paper/2512.09182/) (2025)
- [MPM-LLM4DSE: Reaching the Pareto Frontier in HLS with Multimodal Learning and LLM-Driven Exploration](https://mlsanity.com/paper/2601.04801/) (2026)
- [Latent Graph Learning in Generative Models of Neural Signals](https://mlsanity.com/paper/2508.16776/) (2025)
- [Dynamic Relational Priming Improves Transformer in Multivariate Time Series](https://mlsanity.com/paper/2509.12196/) (2025)
- [Graph Variate Neural Networks](https://mlsanity.com/paper/2509.20311/) (2025)
- [Transformers Can Learn Connectivity in Some Graphs but Not Others](https://mlsanity.com/paper/2509.22343/) (2025)
- [The Role of Feature Interactions in Graph-based Tabular Deep Learning](https://mlsanity.com/paper/2510.04543/) (2025)
- [Fighter: Unveiling the Graph Convolutional Nature of Transformers in Time Series Modeling](https://mlsanity.com/paper/2510.17106/) (2025)
- [Enhancing Graph Representations with Neighborhood-Contextualized Message-Passing](https://mlsanity.com/paper/2511.11046/) (2025)
- [Multi-Scale Protein Structure Modelling with Geometric Graph U-Nets](https://mlsanity.com/paper/2512.06752/) (2025)
- [Solving Oversmoothing in GNNs via Nonlocal Message Passing: Algebraic Smoothing and Depth Scalability](https://mlsanity.com/paper/2512.08475/) (2025)
- [Iterative Compositional Data Generation for Robot Control](https://mlsanity.com/paper/2512.10891/) (2025)
- [How Smoothing is N-simplicial Attention?](https://mlsanity.com/paper/2512.15600/) (2025)
- [Multi-fidelity graph-based neural networks architectures to learn Navier-Stokes solutions on non-parametrized 2D domains](https://mlsanity.com/paper/2601.02157/) (2026)
- [On the Runway Cascade of Transformers for Language Modeling](https://mlsanity.com/paper/2601.14522/) (2026)
- [Neural Network Quantum Field Theory from Transformer Architectures](https://mlsanity.com/paper/2602.10209/) (2026)
- [Transformers are Bayesian Networks](https://mlsanity.com/paper/2603.17063/) (2026)
- [Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformer](https://mlsanity.com/paper/2603.22854/) (2026)
- [Unifying Dynamical Systems and Graph Theory to Mechanistically Understand Computation in Neural Networks](https://mlsanity.com/paper/2605.03598/) (2026)
- [GraphReview: Scientific Paper Evaluation via LLM-based Graph Evidence Expansion](https://mlsanity.com/paper/2605.27204/) (2026)

## Abstract (arXiv)

We establish connections between the Transformer architecture, originally introduced for natural language processing, and Graph Neural Networks (GNNs) for representation learning on graphs. We show how Transformers can be viewed as message passing GNNs operating on fully connected graphs of tokens, where the self-attention mechanism capture the relative importance of all tokens w.r.t. each-other, and positional encodings provide hints about sequential ordering or structure. Thus, Transformers are expressive set processing networks that learn relationships among input elements without being constrained by apriori graphs. Despite this mathematical connection to GNNs, Transformers are implemented via dense matrix operations that are significantly more efficient on modern hardware than sparse message passing. This leads to the perspective that Transformers are GNNs currently winning the hardware lottery.