mlsanityData to 7 Oct 2026

A phase transition between positional and semantic learning in a solvable model of dot-product attention

arXiv 2402.03902 ↗First posted 6 Feb 2024cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Many empirical studies have provided evidence for the emergence of algorithmic mechanisms (abilities) in the learning of language models, that lead to qualitative improvements of the model capabilities. Yet, a theoretical characterization of how such mechanisms emerge remains elusive. In this paper, we take a step in this direction by providing a tight theoretical analysis of the emergence of semantic attention in a solvable model of dot-product attention. More precisely, we consider a non-linear self-attention layer with trainable tied and low-rank query and key matrices. In the asymptotic limit of high-dimensional data and a comparably large number of training samples we provide a tight closed-form characterization of the global minimum of the non-convex empirical loss landscape. We show that this minimum corresponds to either a positional attention mechanism (with tokens attending to each other based on their respective positions) or a semantic attention mechanism (with tokens attending to each other based on their meaning), and evidence an emergent phase transition from the former to the latter with increasing sample complexity. Finally, we compare the dot-product attention layer to a linear positional baseline, and show that it outperforms the latter using the semantic mechanism provided it has access to sufficient data.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 53 of its 74 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

53 of 74 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2018Self-Attention with Relative Position Representationscited by 1,112
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2022Transformers learn in-context by gradient descentcited by 714
  8. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  9. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  10. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  11. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  12. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
38 more
  1. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  2. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  3. 2022Language model compression with weighted low-rank factorizationcited by 192
  4. 2021Thinking Like Transformerscited by 182
  5. 2023A mathematical perspective on Transformerscited by 160
  6. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  7. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  8. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  9. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  10. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  11. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  12. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  13. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  14. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  15. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  16. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  17. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  18. 2022Vision Transformers provably learn spatial structurecited by 92
  19. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  20. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  21. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
  22. 2023Transformers as Support Vector Machinescited by 67
  23. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  24. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  25. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  26. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  27. 2023Transformers learn through gradual rank increasecited by 52
  28. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  29. 2020Phase retrieval in high dimensions: Statistical and computational phase transitionscited by 39
  30. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  31. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  32. 2020Generalization Error of Generalized Linear Models in High Dimensionscited by 25
  33. 2020Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)cited by 23
  34. 2016Phase transitions in Restricted Boltzmann Machines with generic priorscited by 20
  35. 2019Finding the Needle in the Haystack with Convolutions: on the benefits of architectural biascited by 19
  36. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  37. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  38. 2022Learning curves for the multi-class teacher-student perceptroncited by 17

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 13; 2026: 8.

30 papers.

Sort:

By area:

  1. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  2. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  3. 2025A solvable model of learning generative diffusion: theory and insightscited by 14
  4. 2024High-dimensional learning of narrow neural networkscited by 12
  5. 2024Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformerscited by 9
  6. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  7. 2025Physics of Skill Learningcited by 8
  8. 2024Phase Transitions in the Output Distribution of Large Language Modelscited by 7
  9. 2026Perceptrons and localization of attention's mean-field landscapecited by 7
  10. 2024Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyphcited by 6
  11. 2025Statistical Advantage of Softmax Attention: Insights from Single-Location Regressioncited by 6
  12. 2025From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformerscited by 6
18 more
  1. 2024Are queries and keys always relevant? A case study on Transformer wave functionscited by 5
  2. 2024Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layerscited by 5
  3. 2025What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformerscited by 5
  4. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 5
  5. 2024Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokenscited by 3
  6. 2025Bayes optimal learning of attention-indexed modelscited by 3
  7. 2025A statistical physics framework for optimal learningcited by 3
  8. 2026How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretabilitycited by 3
  9. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  10. 2026Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learningcited by 2
  11. 2025Three tiers of computation in transformers and in brain architecturescited by 1
  12. 2025Rethinking Over-Smoothing in Graph Neural Networks: A Perspective from Anderson Localizationcited by 1
  13. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 1
  14. 2025Attention to Order: Transformers Discover Phase Transitions via Learnabilitycited by 0
  15. 2025Minimax Rates for Learning Pairwise Interactions in Attention-Style Modelscited by 0
  16. 2026Scaled Dot-Product Attention implements projection of inputs onto a common surfacecited by 0
  17. 2026Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variablescited by 0
  18. 2026Mean-Field Dynamics of Chain-of-Thought Reasoning in Large Language Modelscited by 0