# Logarithmic-time Schedules for Scaling Language Models with Momentum

arXiv: https://arxiv.org/abs/2602.05298 · HTML: https://mlsanity.com/paper/2602.05298/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2026.

## Built on by (most cited first, 9 of 9)

- [Deriving Hyperparameter Scaling Laws via Modern Optimization Theory](https://mlsanity.com/paper/2603.15958/) (2026)
- [Power-Law Spectrum of the Random Feature Model](https://mlsanity.com/paper/2603.14578/) (2026)
- [Optimistic Dual Averaging Unifies Modern Optimizers](https://mlsanity.com/paper/2605.11172/) (2026)
- [LeRoPE: Learnable RoPE Frequencies Improve Language Modeling](https://mlsanity.com/paper/2607.10134/) (2026)
- [Optimizer Memory Schedules for Outscaling the Overtraining Axis](https://mlsanity.com/paper/2609.04577/) (2026)
- [Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions](https://mlsanity.com/paper/2605.28961/) (2026)
- [Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency](https://mlsanity.com/paper/2609.02728/) (2026)
- [Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models](https://mlsanity.com/paper/2610.13524/) (2026)
- [When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?](https://mlsanity.com/paper/2610.20413/) (2026)

## Abstract (arXiv)

In practice, the hyperparameters (β1, β2) and weight-decay λ in AdamW are typically kept at fixed values. Is there any reason to do otherwise? We show that for large-scale language model training, the answer is yes: by exploiting the power-law structure of language data, one can design time-varying schedules for (β1, β2, λ) that deliver substantial performance gains. We study logarithmic-time scheduling, in which the optimizer's gradient memory horizon grows with training time. Although naive variants of this are unstable, we show that suitable damping mechanisms restore stability while preserving the benefits of longer memory. Based on this, we present ADANA, an AdamW-like optimizer that couples log-time schedules with explicit damping to balance stability and performance. We empirically evaluate ADANA across transformer scalings (45M to 2.6B parameters), comparing against AdamW, Muon, and AdEMAMix. When properly tuned, ADANA achieves up to 40% compute efficiency relative to a tuned AdamW, with gains that persist--and even improve--as model scale increases. We further show that similar benefits arise when applying logarithmic-time scheduling to AdEMAMix, and that logarithmic-time weight-decay alone can yield significant improvements. Finally, we present variants of ADANA that mitigate potential failure modes and improve robustness.