mlsanityData to 7 Oct 2026

Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

arXiv 2609.37745 ↗First posted 29 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a 1/3 exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the 1/3 account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size D. We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to 1/3 across optimizers. In an online teacher-student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents αr and αt. Under SGD, both are close to 1/3, so the data exponent is also 1/3 across different learning rates. Under Adam the two separate: αr simeq 0.48 but αt simeq 0.08. Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: D-independent for SGD but falls with D for Adam. Yet tuned to that optimum, the loss returns to D-1/3 for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, 2αr+ αt = 1, which fixes the optimal data exponent at 1/3. Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with D-1/3 across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 57 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

48 of 57 references.

References · 48

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2019On the Convergence of Adam and Beyondcited by 834
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  7. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2021Explaining Neural Scaling Lawscited by 353
  12. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
36 more
  1. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  2. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  3. 2018An Empirical Model of Large-Batch Trainingcited by 268
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  6. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  7. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  8. 2023The Quantization Model of Neural Scalingcited by 137
  9. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  10. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  11. 2022A Solvable Model of Neural Scaling Lawscited by 101
  12. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  13. 2018Risk and parameter convergence of logistic regressioncited by 81
  14. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  15. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  16. 2024Chinchilla Scaling: A replication attemptcited by 65
  17. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
  18. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  19. 2024Understanding Optimization in Deep Learning with Central Flowscited by 57
  20. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  21. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
  22. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43
  23. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  24. 2024Scaling Optimal LR Across Token Horizonscited by 35
  25. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  26. 2024The Implicit Bias of Adam on Separable Datacited by 32
  27. 2025Superposition Yields Robust Neural Scalingcited by 30
  28. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  29. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  30. 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 11
  31. 2025Neural Thermodynamic Laws for Large Language Model Trainingcited by 10
  32. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 8
  33. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 8
  34. 2025FOCUS: First Order Concentrated Updating Schemecited by 7
  35. 2026A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classificationcited by 3
  36. 2026On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvaturecited by 2

Built on by

Not cited yet.