AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimate
Not on the map. Why?
Abstract (arXiv)
This paper studies the exponential decay mechanism of the second-moment estimate in Adam. We propose AdamNX and a time-varying second-moment decay rate that gradually weakens the correction applied to the update scale. Under the assumptions used in our analysis, this mechanism makes the updates approach momentum-SGD-like behavior during the training plateau phase. We report results on the image-classification, object-detection, and semantic-segmentation tasks, configurations, and comparators included in this paper. These results do not establish multi-seed statistical effects, flatness, or generalization beyond the reported tasks. Our code is open-sourced at https://github.com/mengzhu0308/AdamNX.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 17 Nov 2025.
Builds on
20 of 36 references.
References · 20
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
- 2017Decoupled Weight Decay Regularizationcited by 1,722
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
- 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
- 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
- 2020Designing Network Design Spacescited by 954
- 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
- 2019On the Convergence of Adam and Beyondcited by 834
- 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
8 more
- 2023Symbolic Discovery of Optimization Algorithmscited by 423
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
- 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
- 2019Understanding the Role of Momentum in Stochastic Gradient Methodscited by 42
- 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
- 2018On the Generalization of Stochastic Gradient Descent with Momentumcited by 20
- 2020AdaX: Adaptive Gradient Descent with Exponential Long Term Memorycited by 10
- 2020Stochastic Normalized Gradient Descent with Momentum for Large-Batch Trainingcited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.