New logarithmic step size for stochastic gradient descent
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an O(frac1√T) convergence rate for the SGD. We conduct a comprehensive implementation to demonstrate the efficiency of the newly proposed step size on the ~FashionMinst,~ CIFAR10, and CIFAR100 datasets. Moreover, we compare our results with nine other existing approaches and demonstrate that the new logarithmic step size improves test accuracy by 0.9% for the CIFAR100 dataset when we utilize a convolutional neural network (CNN) model.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 31 references that are in the map. First posted on arXiv 1 Apr 2024.
Builds on
13 of 31 references.
References · 13
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2016YOLO9000: Better, Faster, Strongercited by 1,429
- 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
- 2015Cyclical Learning Rates for Training Neural Networkscited by 627
- 2019SGD: General Analysis and Improved Ratescited by 206
- 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
- 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
- 2020A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performancecited by 26
- 2021On the Convergence of Step Decay Step-Size for Stochastic Optimizationcited by 21
1 more
Built on by
Not cited yet.