# Uniform Scaling Limits in AdamW-Trained Transformers

arXiv: https://arxiv.org/abs/2605.11059 · HTML: https://mlsanity.com/paper/2605.11059/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 56 references that are in the map.
- First posted on arXiv 11 May 2026.

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [A Unified Perspective on the Dynamics of Deep Transformers](https://mlsanity.com/paper/2501.18322/) (2025)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivity](https://mlsanity.com/paper/2410.08198/) (2024)
- [The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram](https://mlsanity.com/paper/2509.10167/) (2025)
- [On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime](https://mlsanity.com/paper/2110.02926/) (2021)
- [ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit](https://mlsanity.com/paper/2603.18168/) (2026)
- [Clustering in Deep Stochastic Transformers](https://mlsanity.com/paper/2601.21942/) (2026)
- [Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models](https://mlsanity.com/paper/2604.26898/) (2026)
- [Homogenized Transformers](https://mlsanity.com/paper/2604.01978/) (2026)

## Abstract (arXiv)

We study the large-depth limit of transformers trained with AdamW, by modelling the hidden-state dynamics as an interacting particle system (IPS) coupled through the attention mechanism. Under appropriate scaling of the attention heads, we prove that the joint dynamics of the hidden states and backpropagated variables converge in L2, uniformly over the initial condition, to the solution of a forward--backward system of ODEs at rate mathcal O(L-1+L-1/3H-1/2). Here, L and H denote the depth and number of heads of the transformer, respectively. The limiting system of ODEs can be identified with a McKean--Vlasov ODE (MVODE) when the attention heads do not incorporate causal masking. By using the flow maps associated with this MVODE and applying concentration of measure techniques, we obtain bounds on the difference between the discrete and continuous models that are uniform over compact sets of initial conditions. As this is achieved without resorting to a covering argument, the constants in our bounds are independent of the number of tokens. Furthermore, under a suitable adaptation to AdamW, the bounds become independent of the token embedding dimension.