# AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimate

arXiv: https://arxiv.org/abs/2511.13465 · HTML: https://mlsanity.com/paper/2511.13465/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Builds on (most cited first, 20 of 20)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://mlsanity.com/paper/2305.14342/) (2023)
- [Improving Generalization Performance by Switching from Adam to SGD](https://mlsanity.com/paper/1712.07628/) (2017)
- [Understanding the Role of Momentum in Stochastic Gradient Methods](https://mlsanity.com/paper/1910.13962/) (2019)
- [Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum](https://mlsanity.com/paper/2006.15815/) (2020)
- [On the Generalization of Stochastic Gradient Descent with Momentum](https://mlsanity.com/paper/1809.04564/) (2018)
- [AdaX: Adaptive Gradient Descent with Exponential Long Term Memory](https://mlsanity.com/paper/2004.09740/) (2020)
- [Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training](https://mlsanity.com/paper/2007.13985/) (2020)

## Built on by (most cited first, 2 of 2)

- [Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations](https://mlsanity.com/paper/2604.12968/) (2026)
- [Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimates](https://mlsanity.com/paper/2605.20533/) (2026)

## Abstract (arXiv)

This paper studies the exponential decay mechanism of the second-moment estimate in Adam. We propose AdamNX and a time-varying second-moment decay rate that gradually weakens the correction applied to the update scale. Under the assumptions used in our analysis, this mechanism makes the updates approach momentum-SGD-like behavior during the training plateau phase. We report results on the image-classification, object-detection, and semantic-segmentation tasks, configurations, and comparators included in this paper. These results do not establish multi-seed statistical effects, flatness, or generalization beyond the reported tasks. Our code is open-sourced at https://github.com/mengzhu0308/AdamNX.