# Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks

arXiv: https://arxiv.org/abs/2002.09647 · HTML: https://mlsanity.com/paper/2002.09647/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2020.

## Built on by (most cited first, 16 of 16)

- [A Review of Deep Transfer Learning and Recent Advancements](https://mlsanity.com/paper/2201.09679/) (2022)
- [AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2303.00565/) (2023)
- [Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule](https://mlsanity.com/paper/2201.11989/) (2022)
- [A Learning Convolutional Neural Network Approach for Network Robustness Prediction](https://mlsanity.com/paper/2203.10552/) (2022)
- [Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness](https://mlsanity.com/paper/2206.13290/) (2022)
- [Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization](https://mlsanity.com/paper/2311.08745/) (2023)
- [The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size](https://mlsanity.com/paper/2108.11713/) (2021)
- [Critical Bach Size Minimizes Stochastic First-Order Oracle Complexity of Deep Learning Optimizer using Hyperparameters Close to One](https://mlsanity.com/paper/2208.09814/) (2022)
- [Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks](https://mlsanity.com/paper/2412.11400/) (2024)
- [Edge Attention Module for Object Classification](https://mlsanity.com/paper/2502.03103/) (2025)
- [Minimization of Stochastic First-order Oracle Complexity of Adaptive Methods for Nonconvex Optimization](https://mlsanity.com/paper/2112.07163/) (2021)
- [Applications of Reinforcement Learning in Deregulated Power Market: A Comprehensive Review](https://mlsanity.com/paper/2205.08369/) (2022)
- [EXPANSE: A Deep Continual / Progressive Learning System for Deep Transfer Learning](https://mlsanity.com/paper/2205.10356/) (2022)
- [Imbalanced Aircraft Data Anomaly Detection](https://mlsanity.com/paper/2305.10082/) (2023)
- [Improving the Accuracy and Interpretability of Neural Networks for Wind Power Forecasting](https://mlsanity.com/paper/2312.15741/) (2023)
- [ADP-VRSGP: Decentralized Learning with Adaptive Differential Privacy via Variance-Reduced Stochastic Gradient Push](https://mlsanity.com/paper/2510.20157/) (2025)

## Abstract (arXiv)

This paper deals with nonconvex stochastic optimization problems in deep learning and provides appropriate learning rates with which adaptive learning rate optimization algorithms, such as Adam and AMSGrad, can approximate a stationary point of the problem. In particular, constant and diminishing learning rates are provided to approximate a stationary point of the problem. Our results also guarantee that the adaptive learning rate optimization algorithms can approximate global minimizers of convex stochastic optimization problems. The adaptive learning rate optimization algorithms are examined in numerical experiments on text and image classification. The experiments show that the algorithms with constant learning rates perform better than ones with diminishing learning rates.