Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size
Not on the map. Why?
Abstract (arXiv)
We theoretically analyzed the convergence behavior of Riemannian stochastic gradient descent (RSGD) and found that using an increasing batch size leads to faster convergence than using a constant batch size, not only with a constant learning rate but also with a decaying learning rate, such as cosine annealing decay and polynomial decay. The convergence rate improves from O(T-1+C) with a constant batch size to O(T-1) with an increasing batch size, where T denotes the total number of iterations and C is a constant. Using principal component analysis and low-rank matrix completion, we investigated, both theoretically and numerically, how an increasing batch size affects computational time as quantified by stochastic first-order oracle (SFO) complexity. An increasing batch size was found to reduce the SFO complexity of RSGD. Furthermore, an increasing batch size was found to offer the advantages of both small and large constant batch sizes.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 30 Jan 2025.
Builds on
28 of 49 references.
References · 28
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2019Hyperbolic Graph Convolutional Neural Networkscited by 402
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
- 2019Hyperbolic Graph Neural Networkscited by 244
- 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
- 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
- 2019Orthogonal Convolutional Neural Networkscited by 123
- 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
16 more
- 2016First-order Methods for Geodesically Convex Optimizationcited by 93
- 2020Graph Geometry Interaction Learningcited by 55
- 2020Projection Robust Wasserstein Distance and Riemannian Optimizationcited by 45
- 2019Low-Rank Matrix Completion: A Contemporary Surveycited by 43
- 2016Deep Learning on Lie Groups for Skeleton-based Action Recognitioncited by 42
- 2018Averaging Stochastic Gradient Descent on Riemannian Manifoldscited by 39
- 2017Riemannian Optimization via Frank-Wolfe Methodscited by 21
- 2017An Alternative to EM for Gaussian Mixture Models: Batch and Stochastic Riemannian Optimizationcited by 20
- 2016Low-rank tensor completion: a Riemannian manifold preconditioning approachcited by 19
- 2019Simple algorithms for optimization on Riemannian manifolds with constraintscited by 14
- 2014A Lower Bound for the Optimization of Finite Sumscited by 13
- 2021On Riemannian Stochastic Approximation Schemes with Fixed Step-Sizecited by 12
- 2024Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descentcited by 11
- 2022Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rulecited by 8
- 2024A general framework of Riemannian adaptive optimization methods with a convergence analysiscited by 5
- 2023Convergence of Riemannian Stochastic Gradient Descent on Hadamard Manifoldcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.