# Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate

arXiv: https://arxiv.org/abs/1805.07557 · HTML: https://mlsanity.com/paper/1805.07557/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2018.

## Built on by (most cited first, 16 of 16)

- [High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize](https://mlsanity.com/paper/2204.02833/) (2022)
- [Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam](https://mlsanity.com/paper/2202.06009/) (2022)
- [Adam revisited: a weighted past gradients perspective](https://mlsanity.com/paper/2101.00238/) (2021)
- [AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs](https://mlsanity.com/paper/2210.06364/) (2022)
- [Momentum Centering and Asynchronous Update for Adaptive Gradient Methods](https://mlsanity.com/paper/2110.05454/) (2021)
- [Analysis of Dual-Based PID Controllers through Convolutional Mirror Descent](https://mlsanity.com/paper/2202.06152/) (2022)
- [AdaInject: Injection Based Adaptive Gradient Descent Optimizers for Convolutional Neural Networks](https://mlsanity.com/paper/2109.12504/) (2021)
- [Angle based dynamic learning rate for gradient descent](https://mlsanity.com/paper/2304.10457/) (2023)
- [Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning](https://mlsanity.com/paper/2101.09192/) (2021)
- [Deterministic Adam-Inspired Methods with Accelerated Convergence Rate](https://mlsanity.com/paper/2604.08742/) (2026)
- [Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization](https://mlsanity.com/paper/2605.12878/) (2026)
- [On the Convergence of Adam, Revisited](https://mlsanity.com/paper/2607.03519/) (2026)
- [AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations](https://mlsanity.com/paper/2107.01525/) (2021)
- [Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks](https://mlsanity.com/paper/2207.09066/) (2022)
- [A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm](https://mlsanity.com/paper/2605.29273/) (2026)
- [Adaptive Forgetting for Nonstationary Optimization: Towards Robust EEG Decoding](https://mlsanity.com/paper/2609.24233/) (2026)

## Abstract (arXiv)

First-order optimization algorithms have been proven prominent in deep learning. In particular, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of ``long-term memory" in Adam-like algorithms, which could hamper their performance and lead to divergence. In our study, we observe that there are benefits of weighting more of the past gradients when designing the adaptive learning rate. We therefore propose an algorithm called the Nostalgic Adam (NosAdam) with theoretically guaranteed convergence at the best known convergence rate. NosAdam can be regarded as a fix to the non-convergence issue of Adam in alternative to the recent work of [Reddi et al., 2018]. Our preliminary numerical experiments show that NosAdam is a promising alternative algorithm to Adam. The proofs, code and other supplementary materials can be found in an anonymously shared link.