# AdaX: Adaptive Gradient Descent with Exponential Long Term Memory

arXiv: https://arxiv.org/abs/2004.09740 · HTML: https://mlsanity.com/paper/2004.09740/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2020.

## Built on by (most cited first, 10 of 10)

- [Context-aware Cross-level Fusion Network for Camouflaged Object Detection](https://mlsanity.com/paper/2105.12555/) (2021)
- [Camouflaged Object Detection via Context-aware Cross-level Fusion](https://mlsanity.com/paper/2207.13362/) (2022)
- [Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers](https://mlsanity.com/paper/2307.00631/) (2023)
- [Momentum Centering and Asynchronous Update for Adaptive Gradient Methods](https://mlsanity.com/paper/2110.05454/) (2021)
- [AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimate](https://mlsanity.com/paper/2511.13465/) (2025)
- [AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations](https://mlsanity.com/paper/2107.01525/) (2021)
- [a novel attention-based network for fast salient object detection](https://mlsanity.com/paper/2112.10481/) (2021)
- [ODC-SA Net: Orthogonal Direction Enhancement and Scale Aware Network for Polyp Segmentation](https://mlsanity.com/paper/2405.06191/) (2024)
- [Adaptive Moment Estimation Optimization Algorithm Using Projection Gradient for Deep Learning](https://mlsanity.com/paper/2503.10005/) (2025)
- [Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimates](https://mlsanity.com/paper/2605.20533/) (2026)

## Abstract (arXiv)

Although adaptive optimization algorithms such as Adam show fast convergence in many machine learning tasks, this paper identifies a problem of Adam by analyzing its performance in a simple non-convex synthetic problem, showing that Adam's fast convergence would possibly lead the algorithm to local minimums. To address this problem, we improve Adam by proposing a novel adaptive gradient descent algorithm named AdaX. Unlike Adam that ignores the past gradients, AdaX exponentially accumulates the long-term gradient information in the past during training, to adaptively tune the learning rate. We thoroughly prove the convergence of AdaX in both the convex and non-convex settings. Extensive experiments show that AdaX outperforms Adam in various tasks of computer vision and natural language processing and can catch up with Stochastic Gradient Descent.