Modified Step Size for Enhanced Stochastic Gradient Descent: Convergence and Experiments

This paper introduces a novel approach to enhance the performance of the stochastic gradient descent (SGD) algorithm by incorporating a modified decay step size based on frac1√t. The proposed step size integrates a logarithmic term, leading to the selection of smaller values in the final iterations. Our analysis establishes a convergence rate of O(fracln T√T) for smooth non-convex functions without the Polyak-Lojasiewicz condition. To evaluate the effectiveness of our approach, we conducted numerical experiments on image classification tasks using the FashionMNIST, and CIFAR10 datasets, and the results demonstrate significant improvements in accuracy, with enhancements of 0.5% and 1.4% observed, respectively, compared to the traditional frac1√t step size. The source code can be found at urlhttps://github.com/Shamaeem/LNSQRTStepSize.