mlsanityData to 14 Oct 2026

Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks

arXiv 2506.02651 ↗First posted 3 Jun 2025stat.ML cond-mat.dis-nn cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We study the dynamics of stochastic gradient descent (SGD) for a class of sequence models termed Sequence Single-Index (SSI) models, where the target depends on a single direction in input space applied to a sequence of tokens. This setting generalizes classical single-index models to the sequential domain, encompassing simplified one-layer attention architectures. We derive a closed-form expression for the population loss in terms of a pair of sufficient statistics capturing semantic and positional alignment, and characterize the induced high-dimensional SGD dynamics for these coordinates. Our analysis reveals two distinct training phases: escape from uninformative initialization and alignment with the target subspace, and demonstrates how the sequence length and positional encoding influence convergence speed and learning trajectories. These results provide a rigorous and interpretable foundation for understanding how sequential structure in data can be beneficial for learning with attention-based models.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

40 of 49 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  6. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  7. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  8. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  9. 2023Representational Strengths and Limitations of Transformerscited by 133
  10. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  11. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  12. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
28 more
  1. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  2. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  3. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  4. 2023Learning time-scales in two-layers neural networkscited by 57
  5. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 57
  6. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  7. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  8. 2024Computational-Statistical Gaps in Gaussian Single-Index Modelscited by 44
  9. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 43
  10. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  11. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  12. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  13. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  14. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 26
  15. 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networkscited by 25
  16. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 22
  17. 2023Leveraging the two timescale regime to demonstrate convergence of neural networkscited by 19
  18. 2024Attention layers provably solve single-location regressioncited by 17
  19. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 15
  20. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
  21. 2024High-dimensional learning of narrow neural networkscited by 12
  22. 2024On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descentcited by 12
  23. 2025Fundamental limits of learning in sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholdscited by 12
  24. 2024Unraveling the Gradient Descent Dynamics of Transformerscited by 11
  25. 2025Optimal Spectral Transitions in High-Dimensional Multi-Index Modelscited by 7
  26. 2023Symmetric Single Index Learningcited by 5
  27. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 3
  28. 2025Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfittingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Statistical Advantage of Softmax Attention: Insights from Single-Location Regressioncited by 7
  2. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 6
  3. 2025A statistical physics framework for optimal learningcited by 3
  4. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  5. 2026Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learningcited by 2
  6. 2026Specialization of softmax attention heads: insights from the high-dimensional single-location modelcited by 2
  7. 2026Provably Learning Attention with Queriescited by 1
  8. 2026The Power of Power Law: Asymmetry Enables Compositional Reasoningcited by 0
  9. 2026A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attentioncited by 0