# Dimension-adapted Momentum Outscales SGD

arXiv: https://arxiv.org/abs/2505.16098 · HTML: https://mlsanity.com/paper/2505.16098/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2025.

## Built on by (most cited first, 12 of 12)

- [Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory](https://mlsanity.com/paper/2603.26554/) (2026)
- [A Defense of the Quadratic Model](https://mlsanity.com/paper/2607.21716/) (2026)
- [Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model](https://mlsanity.com/paper/2602.04774/) (2026)
- [Power-Law Spectrum of the Random Feature Model](https://mlsanity.com/paper/2603.14578/) (2026)
- [On the Effectiveness of the z-Transform Method in Quadratic Optimization](https://mlsanity.com/paper/2507.03404/) (2025)
- [Phases of Muon: When Muon Eclipses SignSGD](https://mlsanity.com/paper/2605.09552/) (2026)
- [Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods](https://mlsanity.com/paper/2606.19179/) (2026)
- [High dimensional theory of two-phase optimizers](https://mlsanity.com/paper/2603.26954/) (2026)
- [Optimizer Memory Schedules for Outscaling the Overtraining Axis](https://mlsanity.com/paper/2609.04577/) (2026)
- [What do near-optimal learning rate schedules look like?](https://mlsanity.com/paper/2603.10301/) (2026)
- [Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions](https://mlsanity.com/paper/2605.28961/) (2026)
- [Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models](https://mlsanity.com/paper/2610.13524/) (2026)

## Abstract (arXiv)

We investigate scaling laws for stochastic momentum algorithms with small batch on the power law random features model, parameterized by data complexity, target complexity, and model size. When trained with a stochastic momentum algorithm, our analysis reveals four distinct loss curve shapes determined by varying data-target complexities. While traditional stochastic gradient descent with momentum (SGD-M) yields identical scaling law exponents to SGD, dimension-adapted Nesterov acceleration (DANA) improves these exponents by scaling momentum hyperparameters based on model size and data complexity. This outscaling phenomenon, which also improves compute-optimal scaling behavior, is achieved by DANA across a broad range of data and target complexities, while traditional methods fall short. Extensive experiments on high-dimensional synthetic quadratics validate our theoretical predictions and large-scale text experiments with LSTMs show DANA's improved loss exponents over SGD hold in a practical setting.