A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning
Not on the map. Why?
Abstract (arXiv)
One of the most important parts of Artificial Neural Networks is minimizing the loss functions which tells us how good or bad our model is. To minimize these losses we need to tune the weights and biases. Also to calculate the minimum value of a function we need gradient. And to update our weights we need gradient descent. But there are some problems with regular gradient descent ie. it is quite slow and not that accurate. This article aims to give an introduction to optimization strategies to gradient descent. In addition, we shall also discuss the architecture of these algorithms and further optimization of Neural Networks in general
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 41 references that are in the map. First posted on arXiv 7 Jan 2021.
Builds on
20 of 41 references.
References · 20
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2019On the Convergence of Adam and Beyondcited by 841
- 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
- 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
- 2014The Loss Surfaces of Multilayer Networkscited by 280
- 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
- 2016Deep Learning without Poor Local Minimacited by 234
- 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
- 2016Matrix Completion has No Spurious Local Minimumcited by 125
- 2014Stochastic Optimization with Importance Samplingcited by 91
- 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
- 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 90
8 more
- 2017Shake-Shake regularizationcited by 87
- 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
- 2017Natasha 2: Faster Non-Convex Optimization Than SGDcited by 49
- 2016Efficient approaches for escaping higher order saddle points in non-convex optimizationcited by 23
- 2018Adding One Neuron Can Eliminate All Bad Local Minimacited by 21
- 2015Complete Dictionary Recovery over the Spherecited by 19
- 2019Elimination of All Bad Local Minima in Deep Learningcited by 10
- 2020Stochastic gradient descent with random learning ratecited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.