# Refresh-Scaling the Memory of Balanced Adam

arXiv: https://arxiv.org/abs/2605.10119 · HTML: https://mlsanity.com/paper/2605.10119/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 11 May 2026.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [In Search of Adam's Secret Sauce](https://mlsanity.com/paper/2505.21829/) (2025)
- [A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms](https://mlsanity.com/paper/2009.06125/) (2020)
- [Why β1 = β2 Is Dynamically Special in Adam](https://mlsanity.com/paper/2601.21739/) (2026)

## Built on by (most cited first, 1 of 1)

- [Early Memory Selection for Balanced Adam](https://mlsanity.com/paper/2610.08624/) (2026)

## Abstract (arXiv)

Recent evidence suggests that Adam performs robustly when its momentum parameters are tied, β1=β2, reducing the optimizer to a single remaining parameter. However, how this parameter should be set remains poorly understood. We argue that, in balanced Adam, β should not be treated as a dimensionless constant: it defines a statistical memory horizon Hβ=(1-β)-1. In terms of the effective learning horizon TES, estimated from the validation trajectory, we study the refresh count Rβ=(1-β)TES, which measures how many times Adam renews its internal statistics during the useful phase of training. Across 11 vision and language experiments, we find that choosing β so that Rβ≈1000 selects different β values depending on the training scale, yet improves robustness over the best fixed-beta baseline. Compared with the strongest fixed choice β=0.944, the refresh rule improves worst-case robustness, reducing the maximum relative gap in validation loss by 33.4%, while bringing all 11 runs within 1% of their validation oracle. These results suggest that the remaining hyperparameter of balanced Adam is more naturally viewed as a memory-scale variable than as a fixed constant. This provides a simple budget-aware perspective on optimizer scaling and opens a path toward treating Adam's momentum as part of the learning dynamics rather than as a static default.