# Gradient Descent based Optimization Algorithms for Deep Learning Models Training

arXiv: https://arxiv.org/abs/1903.03614 · HTML: https://mlsanity.com/paper/1903.03614/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2019.

## Built on by (most cited first, 6 of 6)

- [Newton Method-based Subspace Support Vector Data Description](https://mlsanity.com/paper/2309.13960/) (2023)
- [Quantum-Enhanced Weight Optimization for Neural Networks Using Grover's Algorithm](https://mlsanity.com/paper/2504.14568/) (2025)
- [Efficient Low-Latency Dynamic Licensing for Deep Neural Network Deployment on Edge Devices](https://mlsanity.com/paper/2102.12165/) (2021)
- [Integrate-and-Fire Neurons for Low-Powered Pattern Recognition](https://mlsanity.com/paper/2106.14596/) (2021)
- [Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networks](https://mlsanity.com/paper/2303.18083/) (2023)
- [Optimal Pricing for Data-Augmented AutoML Marketplaces](https://mlsanity.com/paper/2310.17843/) (2023)

## Abstract (arXiv)

In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back propagation algorithm actually. In back propagation, the model variables will be updated iteratively until convergence with gradient descent based optimization algorithms. Besides the conventional vanilla gradient descent algorithm, many gradient descent variants have also been proposed in recent years to improve the learning performance, including Momentum, Adagrad, Adam, Gadam, etc., which will all be introduced in this paper respectively.