Optimizer-dependent training dynamics converge to the same one-third optimal data scaling
Not on the map. Why?
Abstract (arXiv)
Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a 1/3 exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the 1/3 account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size D. We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to 1/3 across optimizers. In an online teacher-student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents αr and αt. Under SGD, both are close to 1/3, so the data exponent is also 1/3 across different learning rates. Under Adam the two separate: αr simeq 0.48 but αt simeq 0.08. Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: D-independent for SGD but falls with D for Adam. Yet tuned to that optimum, the loss returns to D-1/3 for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, 2αr+ αt = 1, which fixes the optimal data exponent at 1/3. Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with D-1/3 across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 57 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
48 of 57 references.
References · 48
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2019On the Convergence of Adam and Beyondcited by 834
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
- 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
- 2023Symbolic Discovery of Optimization Algorithmscited by 423
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2021Explaining Neural Scaling Lawscited by 353
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
36 more
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
- 2018An Empirical Model of Large-Batch Trainingcited by 268
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
- 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
- 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
- 2023The Quantization Model of Neural Scalingcited by 137
- 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
- 2024A Dynamical Model of Neural Scaling Lawscited by 105
- 2022A Solvable Model of Neural Scaling Lawscited by 101
- 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
- 2018Risk and parameter convergence of logistic regressioncited by 81
- 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
- 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
- 2024Chinchilla Scaling: A replication attemptcited by 65
- 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 63
- 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
- 2024Understanding Optimization in Deep Learning with Central Flowscited by 57
- 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
- 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
- 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43
- 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
- 2024Scaling Optimal LR Across Token Horizonscited by 35
- 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
- 2024The Implicit Bias of Adam on Separable Datacited by 32
- 2025Superposition Yields Robust Neural Scalingcited by 30
- 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
- 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
- 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 11
- 2025Neural Thermodynamic Laws for Large Language Model Trainingcited by 10
- 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 8
- 2026Inverse Depth Scaling From Most Layers Being Similarcited by 8
- 2025FOCUS: First Order Concentrated Updating Schemecited by 7
- 2026A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classificationcited by 3
- 2026On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvaturecited by 2
Built on by
Not cited yet.