mlsanityData to 14 Oct 2026

Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws

arXiv 2509.24914 ↗First posted 29 Sep 2025stat.ML cond-mat.dis-nn cs.ITcited by 6

Not on the map. Why?

Abstract (arXiv)

Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the origin of these phenomena and their implications for generalization remain poorly understood. We study empirical risk minimization in a single-head tied-attention layer trained on synthetic high-dimensional sequence tasks generated from the attention-indexed model. Using tools from random matrix theory, spin-glass theory, and approximate message passing, we obtain an exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and the spectrum of the key and query matrices. Our theory predicts the full singular-value distribution of the trained query-key map, including low-rank structure and isolated spectral outliers, in qualitative agreement with observations in more realistic transformers. Finally, for targets with power-law spectra, we show that learning proceeds through sequential spectral recovery, leading to the emergence of power-law scaling laws.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 58 of its 75 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

58 of 75 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  10. 2021Explaining Neural Scaling Lawscited by 363
  11. 2017Implicit Regularization in Matrix Factorizationcited by 322
  12. 2015Norm-Based Capacity Control in Neural Networkscited by 299
38 more
  1. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  2. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  4. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  5. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  6. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  7. 2022A Solvable Model of Neural Scaling Lawscited by 116
  8. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  9. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 113
  10. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  11. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  12. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  13. 2022Universality of empirical risk minimizationcited by 75
  14. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  15. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  16. 2023Learning time-scales in two-layers neural networkscited by 57
  17. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  18. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
  19. 2022Dimension free ridge regressioncited by 51
  20. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  21. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  22. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 43
  23. 2021Graph-based Approximate Message Passing Iterationscited by 35
  24. 2020Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic Viewcited by 32
  25. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 32
  26. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  27. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  28. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  29. 2024A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimatorcited by 28
  30. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  31. 2024Weight decay induces low-rank attention layerscited by 27
  32. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 25
  33. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 24
  34. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  35. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  36. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 21
  37. 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 19
  38. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 16

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  2. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  3. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 2
  4. 2026Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learningcited by 1
  5. 2026How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Modelscited by 0
  6. 2026How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscited by 0