# Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM

arXiv: https://arxiv.org/abs/1908.00700 · HTML: https://mlsanity.com/paper/1908.00700/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2019.

## Built on by (most cited first, 3 of 3)

- [On the Algorithmic Stability and Generalization of Adaptive Optimization Methods](https://mlsanity.com/paper/2211.03970/) (2022)
- [Convolutional Neural Network for Early Pulmonary Embolism Detection via Computed Tomography Pulmonary Angiography](https://mlsanity.com/paper/2204.03204/) (2022)
- [Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution](https://mlsanity.com/paper/2404.06679/) (2024)

## Abstract (arXiv)

Adaptive gradient methods (AGMs) have become popular in optimizing the nonconvex problems in deep learning area. We revisit AGMs and identify that the adaptive learning rate (A-LR) used by AGMs varies significantly across the dimensions of the problem over epochs (i.e., anisotropic scale), which may lead to issues in convergence and generalization. All existing modified AGMs actually represent efforts in revising the A-LR. Theoretically, we provide a new way to analyze the convergence of AGMs and prove that the convergence rate of textscAdam also depends on its hyper-parameter ε, which has been overlooked previously. Based on these two facts, we propose a new AGM by calibrating the A-LR with an activation (em softplus) function, resulting in the textscSadam and textscSAMSGrad methods footnoteCode is available at https://github.com/neilliang90/Sadam.git.. We further prove that these algorithms enjoy better convergence speed under nonconvex, non-strongly convex, and Polyak-Lojasiewicz conditions compared with textscAdam. Empirical studies support our observation of the anisotropic A-LR and show that the proposed methods outperform existing AGMs and generalize even better than S-Momentum in multiple deep learning tasks.