# Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

arXiv: https://arxiv.org/abs/2609.02728 · HTML: https://mlsanity.com/paper/2609.02728/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 74 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Practical Efficiency of Muon for Pretraining](https://mlsanity.com/paper/2505.02222/) (2025)
- [4+3 Phases of Compute-Optimal Neural Scaling Laws](https://mlsanity.com/paper/2405.15074/) (2024)
- [Benchmarking Optimizers for Large Language Model Pretraining](https://mlsanity.com/paper/2509.01440/) (2025)
- [The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training](https://mlsanity.com/paper/2502.19002/) (2025)
- [A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descent](https://mlsanity.com/paper/2310.00692/) (2023)
- [Logarithmic-time Schedules for Scaling Language Models with Momentum](https://mlsanity.com/paper/2602.05298/) (2026)
- [Accelerating SGD with momentum for over-parameterized learning](https://mlsanity.com/paper/1810.13395/) (2018)
- [The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization](https://mlsanity.com/paper/2409.09745/) (2024)
- [Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay](https://mlsanity.com/paper/2602.06797/) (2026)
- [Scaling Law for Stochastic Gradient Descent in Quadratically Parameterized Linear Regression](https://mlsanity.com/paper/2502.09106/) (2025)
- [Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods](https://mlsanity.com/paper/2606.19179/) (2026)
- [Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters](https://mlsanity.com/paper/2506.11904/) (2025)
- [Asymmetric Scaling Laws from Sparse Features](https://mlsanity.com/paper/2605.23591/) (2026)

## Abstract (arXiv)

We study when and how momentum improves large-batch training in the one-pass regime, using power-law kernel regression as a tractable setting. We first characterize risk stability through the critical learning rate, defined as the largest learning rate for stable training, and obtain ηSGDcriteqsim 1, ηPolyakcriteqsim min1,B(1-ρ), and ηNesterovcriteqsim min1,Bβ(1-ρ), where B is the batch size, ρ is the momentum factor, and β>1 is the capacity exponent. Within this admissible region, we derive scaling laws for the full risk dynamics, capturing the progression from an early transient, through power-law decay, to a noise floor. We then minimize the final-step risk over the admissible learning rates and momentum factors under a fixed data budget, yielding a three-regime batch-size phase diagram that reveals how the role of momentum changes with batch size. Notably, Polyak enlarges the critical batch size, the largest batch size preserving the best small-batch data-scaling exponent, thereby enabling greater parallelism without sacrificing data efficiency. In contrast, Nesterov achieves better data efficiency in the large-batch regime because its look-ahead mechanism suppresses noise accumulation. Numerical experiments validate the predicted stability boundaries, risk dynamics, and batch-size phase diagram.