mlsanityData to 14 Oct 2026

The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training

arXiv 2605.26489 ↗First posted 26 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language model pre-training typically exhibits a two-phase trajectory: a fast initial loss drop followed by a prolonged slow improvement. We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early, even as parameter matrices continue to evolve. We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures (GPT-2, LLaMA) and settings, including various schedules (Step-wise, WSD, Cosine Decay), weight decays, and optimizers (AdamW, Muon). By analyzing a simplified Transformer, we prove that growing weight norms inevitably precipitate an early SoSD threshold, after which the rate of loss decrease becomes theoretically bounded by the variation in the singular distribution. We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 52 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

44 of 52 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2022In-context Learning and Induction Headscited by 964
  12. 2022Transformers learn in-context by gradient descentcited by 732
32 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  3. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  5. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  8. 2017Implicit Regularization in Matrix Factorizationcited by 322
  9. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  10. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  11. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  12. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  13. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  14. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  15. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  16. 2017Implicit Regularization in Deep Learningcited by 90
  17. 2022Understanding the unstable convergence of gradient descentcited by 86
  18. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 86
  19. 2023Transformers as Support Vector Machinescited by 74
  20. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  21. 2020Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descentcited by 40
  22. 2021Approximating How Single Head Attention Learnscited by 38
  23. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  24. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 32
  25. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
  26. 2025Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisationcited by 17
  27. 2024Learning Associative Memories with Gradient Descentcited by 16
  28. 2024Non-asymptotic Convergence of Training Transformers for Next-token Predictioncited by 13
  29. 2025An Analysis for Reasoning Bias of Language Models with Small Initializationcited by 13
  30. 2025How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Biascited by 11
  31. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 10
  32. 2025Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspectivecited by 5

Built on by

Not cited yet.