Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization
Not on the map. Why?
Abstract (arXiv)
signSGD is attractive in nonconvex optimization because it communicates sign-valued rather than full-precision gradients. Several standard analyses assume independent stochastic-gradient samples, whereas a common finite-sum implementation reshuffles the data and processes them sequentially. We study this variant, signSGD with random reshuffling (SignRR), and show that reshuffling does not in general repair the bias created by discarding gradient magnitudes. In particular, on a one-dimensional two-component strongly convex quadratic, the expected gradient norm at every SignRR inner iterate equals 1/2. We complement this impossibility result with an alignment-explicit finite-time bound O(log(nT)/√nT+varepsilonalign), where varepsilonalign measures the averaged loss of descent caused by component-sign misalignment. A horizon-tuned constant stepsize improves the vanishing term to O(1/√nT), and a remaining-set alignment condition yields a residual-free O(1/√nT) guarantee. The alignment term is upper bounded by twice the averaged mean absolute gradient error and, in turn, by twice an averaged coordinatewise conditional root-mean-square error. As a variance-reduced alternative, we analyze SignRVR, which signs an SVRG estimator anchored at the beginning of every epoch. A pathwise argument gives a residual-free guarantee with an O(√d/T) averaged ℓ1-stationarity bound.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 41 of its 68 references that are in the map. First posted on arXiv 24 Oct 2023.
Builds on
41 of 68 references.
References · 41
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
- 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
- 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
- 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
- 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
- 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
- 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
- 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
- 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
- 2020Momentum Improves Normalized SGDcited by 136
- 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
29 more
- 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
- 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 112
- 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
- 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
- 2015Why Random Reshuffling Beats Stochastic Gradient Descentcited by 78
- 2016Variance Reduction for Faster Non-Convex Optimizationcited by 72
- 2019Optimization for deep learning: theory and algorithmscited by 70
- 2018Random Shuffling Beats SGD after Finite Epochscited by 66
- 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
- 2019How Good is SGD with Random Shuffling?cited by 63
- 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
- 2019SGD without Replacement: Sharper Rates for General Smooth Convex Functionscited by 59
- 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 58
- 2020A Unified Convergence Analysis for Shuffling-Type Gradient Methodscited by 57
- 2020SGD with shuffling: optimal rates without component convexity and large epoch requirementscited by 57
- 2020Closing the convergence gap of SGD without replacementcited by 56
- 2018Distributed learning with compressed gradientscited by 53
- 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
- 2020Stochastic-Sign SGD for Federated Learning with Theoretical Guaranteescited by 34
- 2019Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithmscited by 29
- 2017Convergence Analysis of Distributed Stochastic Gradient Descent with Shufflingcited by 23
- 2020SMG: A Shuffling Gradient-Based Method with Momentumcited by 22
- 2017Stochastic Recursive Gradient Algorithm for Nonconvex Optimizationcited by 20
- 2021Random Reshuffling with Variance Reduction: New Analysis and Better Ratescited by 19
- 2021Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned Problemscited by 16
- 2021Convergence of Random Reshuffling Under The Kurdyka-Lojasiewicz Inequalitycited by 16
- 2019Incremental Methods for Weakly Convex Optimizationcited by 7
- 2023SignSVRG: fixing SignSGD via variance reductioncited by 7
- 2023Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneitycited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.