mlsanityData to 7 Oct 2026

Universal One-third Time Scaling in Learning Peaked Distributions

arXiv 2602.03685 ↗First posted 3 Feb 2026cs.LG cs.AI stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic analysis of toy models and empirical evaluation of LLMs, we show that this behavior can arise intrinsically from the use of softmax and cross-entropy. When learning peaked probability distributions, e.g., next-token distributions, these components generically yield power-law vanishing losses and gradients, regardless of many microscopic details, creating a fundamental optimization bottleneck. This ultimately leads to power-law time scaling of the loss with a universal exponent of 1/3. Our results provide a mechanistic explanation for observed neural scaling and suggest new directions for improving LLM training efficiency.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

35 of 46 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  9. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2021Explaining Neural Scaling Lawscited by 353
  12. 20242 OLMo 2 Furiouscited by 302
23 more
  1. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  2. 2023The Quantization Model of Neural Scalingcited by 137
  3. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  4. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  5. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  6. 2022A Solvable Model of Neural Scaling Lawscited by 101
  7. 2021Learning Curve Theorycited by 79
  8. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 75
  9. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  10. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  11. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  12. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  13. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  14. 2025Superposition Yields Robust Neural Scalingcited by 30
  15. 2024An exactly solvable model for emergence and scaling laws in the multitask sparse parity problemcited by 21
  16. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  17. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  18. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 14
  19. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
  20. 2025Neural Thermodynamic Laws for Large Language Model Trainingcited by 10
  21. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  22. 2025Physics of Skill Learningcited by 8
  23. 2025FOCUS: First Order Concentrated Updating Schemecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 8
  2. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 4
  3. 2026Power-Law Spectrum of the Random Feature Modelcited by 3
  4. 2026A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classificationcited by 3
  5. 2026Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossovercited by 2
  6. 2026Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucinationcited by 0
  7. 2026Emergent One-Third Scaling Law as Attention Tries to Concentratecited by 0
  8. 2026Optimizer-dependent training dynamics converge to the same one-third optimal data scalingcited by 0