Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix
Not on the map. Why?
Abstract (arXiv)
Self-attention layers have become fundamental building blocks of modern deep neural networks, yet their theoretical understanding remains limited, particularly from the perspective of random matrix theory. In this work, we provide a rigorous analysis of the singular value spectrum of the attention matrix and establish the first Gaussian equivalence result for attention. In a natural regime where the inverse temperature remains of constant order, we show that the singular value distribution of the attention matrix is asymptotically characterized by a tractable linear model. We further demonstrate that the distribution of squared singular values deviates from the Marchenko-Pastur law, which has been believed in previous work. Our proof relies on two key ingredients: precise control of fluctuations in the normalization term and a refined linearization that leverages favorable Taylor expansions of the exponential. This analysis also identifies a threshold for linearization and elucidates why attention, despite not being an entrywise operation, admits a rigorous Gaussian equivalence in this regime.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 69 references that are in the map. First posted on arXiv 8 Oct 2025.
Builds on
36 of 69 references.
References · 36
- 2017Attention Is All You Needcited by 60,361
- 2023Qwen Technical Reportcited by 3,344
- 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
- 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
- 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
- 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
- 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
- 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
- 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
- 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
24 more
- 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
- 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
- 2017A Random Matrix Approach to Neural Networkscited by 80
- 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 77
- 2022Universality of empirical risk minimizationcited by 75
- 2018The Emergence of Spectral Universality in Deep Networkscited by 74
- 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
- 2025Scalable-Softmax Is Superior for Attentioncited by 45
- 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
- 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
- 2018Optimality and Sub-optimality of PCA I: Spiked Random Matrix Modelscited by 33
- 2018Fisher Information and Natural Gradient Learning of Random Deep Networkscited by 28
- 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
- 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 26
- 2025Critical attention scaling in long-context transformerscited by 20
- 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 19
- 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 17
- 2022Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networkscited by 13
- 2020The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometrycited by 9
- 2021Analysis of One-Hidden-Layer Neural Networks via the Resolvent Methodcited by 9
- 2024Length-Induced Embedding Collapse in PLM-based Modelscited by 9
- 2021Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Casecited by 5
- 2025Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programscited by 5
- 2024Entrywise application of non-linear functions on orthogonally invariant matricescited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.