# Early Memory Selection for Balanced Adam

arXiv: https://arxiv.org/abs/2610.08624 · HTML: https://mlsanity.com/paper/2610.08624/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 28 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 3 of 3)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise](https://mlsanity.com/paper/2402.01567/) (2024)
- [Refresh-Scaling the Memory of Balanced Adam](https://mlsanity.com/paper/2605.10119/) (2026)

## Abstract (arXiv)

We propose a method for choosing the shared memory parameter β1=β2=β in Adam from a short pilot training. The selected β remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared β=0.95. The mean gap is also 32.3% lower than that of the best constant β chosen across all eleven workloads.