Phase transitions for the noisy transformer model in arbitrary dimension
Not on the map. Why?
Abstract (arXiv)
We study the McKean--Vlasov free energy on the unit sphere associated with the unnormalized self-attention (USA) model for noisy transformer dynamics. We prove a sharp global-minimizer dichotomy in every dimension d≥2. There is a unique β_*(d)>0 such that beginequation* fracId/2+1(β_*(d))Id/2(β_*(d))=frac1d, endequation* where I_nu is the modified Bessel function of the first kind. For 0<β≤ β_*(d), the uniform density remains the unique global minimizer up to the linear-stability threshold beginequation* K_#(d)(β)=fracβd/22d/2Gamma(d/2)Id/2(β), endequation* and the phase transition is continuous. For β>β_*(d), the uniform density is not globally minimizing at K_#(d)(β), so the critical coupling satisfies Kc<K_#(d)(β) and the transition is discontinuous. This result generalizes the authors' recent d=2 work arXiv:2604.16288 to arbitrary dimension. The proof uses the sharp Beckner--Onofri/logarithmic Hardy-Littlewood-Sobolev (HLS) inequality on the sphere, together with a Funk--Hecke/Bessel coefficient computation and a degree-two quartic obstruction.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 9 of its 25 references that are in the map. First posted on arXiv 3 Jun 2026.
Builds on
9 of 25 references.
References · 9
- 2017Attention Is All You Needcited by 60,361
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2023A mathematical perspective on Transformerscited by 164
- 2023The emergence of clusters in self-attention dynamicscited by 140
- 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
- 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
- 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 31
- 2025On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformerscited by 12
- 2026Phase transitions in Doi-Onsager, Noisy Transformer, and other multimodal modelscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.