# From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers

arXiv: https://arxiv.org/abs/2607.10677 · HTML: https://mlsanity.com/paper/2607.10677/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 12 Jul 2026.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Rethinking Graph Transformers with Spectral Attention](https://mlsanity.com/paper/2106.03893/) (2021)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs](https://mlsanity.com/paper/2202.04579/) (2022)
- [A Tensorized Transformer for Language Modeling](https://mlsanity.com/paper/1906.09777/) (2019)
- [Sheaf Neural Networks with Connection Laplacians](https://mlsanity.com/paper/2206.08702/) (2022)
- [On the Role of Attention Masks and LayerNorm in Transformers](https://mlsanity.com/paper/2405.18781/) (2024)
- [Tensor Product Attention Is All You Need](https://mlsanity.com/paper/2501.06425/) (2025)
- [Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines](https://mlsanity.com/paper/2106.01506/) (2021)
- [On Learning the Transformer Kernel](https://mlsanity.com/paper/2110.08323/) (2021)
- [Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning](https://mlsanity.com/paper/2503.01329/) (2025)
- [OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization](https://mlsanity.com/paper/2501.18793/) (2025)
- [A Framework for Non-Linear Attention via Modern Hopfield Networks](https://mlsanity.com/paper/2506.11043/) (2025)

## Abstract (arXiv)

Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood. We view a token sequence as a vector field over the token-position graph and identify attention as a connection walk: messages are aggregated by a nonnegative walk matrix while being transported along each edge by a learned linear map. Within this framework, we prove that single-head attention (SHA) is exactly a connection propagation step with constant transport, and that multi-head attention (MHA) is exactly a single edge-dependent connection walk whose effective transport is an attention-gated mixture of headwise transports. We further clarify the conditions under which the corresponding generator reduces to a random-walk connection Laplacian, highlighting the roles of stochasticity, reversibility, and metric-compatible transports. Empirically, we find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory: effective attention graphs converge to stable geometric operators in deeper layers, learned transports self-organize into approximate scaled isometries, and both phenomena strengthen consistently with scale. Overall, the paper provides a precise connection-walk formalism that links self-attention to classical geometric operators, along with a set of operator-level tools for analyzing transformer models from a geometric perspective.