# Exploring the Optimized Value of Each Hyperparameter in Various Gradient Descent Algorithms

arXiv: https://arxiv.org/abs/2212.12279 · HTML: https://mlsanity.com/paper/2212.12279/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 23 Dec 2022.

## Builds on (most cited first, 6 of 6)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [A Survey of Optimization Methods from a Machine Learning Perspective](https://mlsanity.com/paper/1906.06821/) (2019)
- [A Comparison of Optimization Algorithms for Deep Learning](https://mlsanity.com/paper/2007.14166/) (2020)
- [The Effects of Hyperparameters on SGD Training of Neural Networks](https://mlsanity.com/paper/1508.02788/) (2015)

## Abstract (arXiv)

In the recent years, various gradient descent algorithms including the methods of gradient descent, gradient descent with momentum, adaptive gradient (AdaGrad), root-mean-square propagation (RMSProp) and adaptive moment estimation (Adam) have been applied to the parameter optimization of several deep learning models with higher accuracies or lower errors. These optimization algorithms may need to set the values of several hyperparameters which include a learning rate, momentum coefficients, etc. Furthermore, the convergence speed and solution accuracy may be influenced by the values of hyperparameters. Therefore, this study proposes an analytical framework to use mathematical models for analyzing the mean error of each objective function based on various gradient descent algorithms. Moreover, the suitable value of each hyperparameter could be determined by minimizing the mean error. The principles of hyperparameter value setting have been generalized based on analysis results for model optimization. The experimental results show that higher efficiency convergences and lower errors can be obtained by the proposed method.