# A Unified Perspective on the Dynamics of Deep Transformers

arXiv: https://arxiv.org/abs/2501.18322 · HTML: https://mlsanity.com/paper/2501.18322/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2025.

## Built on by (most cited first, 30 of 32)

- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Quantitative Clustering in Mean-Field Transformer Models](https://mlsanity.com/paper/2504.14697/) (2025)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Synchronization of mean-field models on the circle](https://mlsanity.com/paper/2507.22857/) (2025)
- [On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers](https://mlsanity.com/paper/2510.20094/) (2025)
- [Learning Enhanced Ensemble Filters](https://mlsanity.com/paper/2504.17836/) (2025)
- [Perceptrons and localization of attention's mean-field landscape](https://mlsanity.com/paper/2601.21366/) (2026)
- [Clustering in Deep Stochastic Transformers](https://mlsanity.com/paper/2601.21942/) (2026)
- [Accelerated Stein Variational Gradient Flow](https://mlsanity.com/paper/2503.23462/) (2025)
- [Transformers through the lens of support-preserving maps between measures](https://mlsanity.com/paper/2509.25611/) (2025)
- [Optimal and Diffusion Transports in Machine Learning](https://mlsanity.com/paper/2512.06797/) (2025)
- [Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime](https://mlsanity.com/paper/2605.10931/) (2026)
- [Token Sample Complexity of Attention](https://mlsanity.com/paper/2512.10656/) (2025)
- [Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective](https://mlsanity.com/paper/2512.11784/) (2025)
- [An Optimal Control Approach To Transformer Training](https://mlsanity.com/paper/2603.09571/) (2026)
- [Propagation of Chaos in Contextual Flow Maps](https://mlsanity.com/paper/2605.16747/) (2026)
- [On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers](https://mlsanity.com/paper/2607.18584/) (2026)
- [Measure-to-measure Regression with Transformers](https://mlsanity.com/paper/2605.28075/) (2026)
- [Preconditioned Regularized Wasserstein Proximal Sampling](https://mlsanity.com/paper/2509.01685/) (2025)
- [Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with L1 Prior](https://mlsanity.com/paper/2510.16356/) (2025)
- [SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds](https://mlsanity.com/paper/2603.16535/) (2026)
- [On the global convergence of gradient flow for wide shallow models beyond homogeneous nonlinearities](https://mlsanity.com/paper/2605.10775/) (2026)
- [Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows](https://mlsanity.com/paper/2605.18870/) (2026)
- [Reachability and asymptotics of Gaussian Transformer dynamics](https://mlsanity.com/paper/2606.07600/) (2026)
- [SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding](https://mlsanity.com/paper/2609.05141/) (2026)
- [Universal Approximation of Measure-to-Measure Operators by Pushforwards](https://mlsanity.com/paper/2609.35483/) (2026)
- [Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Models](https://mlsanity.com/paper/2505.00818/) (2025)
- [Kinetic theory for Transformers and the lost-in-the-middle phenomenon](https://mlsanity.com/paper/2605.09213/) (2026)
- [Uniform Scaling Limits in AdamW-Trained Transformers](https://mlsanity.com/paper/2605.11059/) (2026)
- [Transformer-like Inference from Optimal Control](https://mlsanity.com/paper/2605.15608/) (2026)

## Abstract (arXiv)

Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the attention function, which learns dependencies between tokens and is key to the success of Transformers. However, the iterative application of attention across layers induces complex dynamics that remain to be fully understood. To analyze these dynamics, we identify each input sequence with a probability measure and model its evolution as a Vlasov equation called Transformer PDE, whose velocity field is non-linear in the probability measure. Our first set of contributions focuses on compactly supported initial data. We show the Transformer PDE is well-posed and is the mean-field limit of an interacting particle system, thus generalizing and extending previous analysis to several variants of self-attention: multi-head attention, L2 attention, Sinkhorn attention, Sigmoid attention, and masked attention--leveraging a conditional Wasserstein framework. In a second set of contributions, we are the first to study non-compactly supported initial conditions, by focusing on Gaussian initial data. Again for different types of attention, we show that the Transformer PDE preserves the space of Gaussian measures, which allows us to analyze the Gaussian case theoretically and numerically to identify typical behaviors. This Gaussian analysis captures the evolution of data anisotropy through a deep Transformer. In particular, we highlight a clustering phenomenon that parallels previous results in the non-normalized discrete case.