Enhancing Stochastic Optimization for Statistical Efficiency Using ROOT-SGD with Diminishing Stepsize
Not on the map. Why?
Abstract (arXiv)
In this paper, we revisit textsfROOT-SGD, an innovative method for stochastic optimization to bridge the gap between stochastic optimization and statistical efficiency. The proposed method enhances the performance and reliability of textsfROOT-SGD by integrating a carefully designed diminishing stepsize strategy. This approach addresses key challenges in optimization, providing robust theoretical guarantees and practical benefits. Our analysis demonstrates that textsfROOT-SGD with diminishing achieves optimal convergence rates while maintaining computational efficiency. By dynamically adjusting the learning rate, textsfROOT-SGD ensures improved stability and precision throughout the optimization process. The findings of this study offer valuable insights for developing advanced optimization algorithms that are both efficient and statistically robust.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 78 references that are in the map. First posted on arXiv 15 Jul 2024.
Builds on
50 of 78 references.
References · 50
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
- 2014Convex Optimization: Algorithms and Complexitycited by 569
- 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
- 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
- 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
- 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 165
- 2016High-dimensional Bayesian inference via the Unadjusted Langevin Algorithmcited by 144
- 2020Momentum Improves Normalized SGDcited by 136
- 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
- 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 84
- 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
- 2020On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentrationcited by 65
38 more
- 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
- 2020Least Squares Regression with Markovian Data: Fundamental Limits and Algorithmscited by 56
- 2020Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noisecited by 52
- 2019Non-asymptotic Analysis of Biased Stochastic Approximation Schemecited by 45
- 2020An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Biascited by 45
- 2020Is Temporal Difference Learning Optimal? An Instance-Dependent Analysiscited by 41
- 2018Averaging Stochastic Gradient Descent on Riemannian Manifoldscited by 39
- 2016Asymptotic Optimality in Stochastic Optimizationcited by 38
- 2020Online Covariance Matrix Estimation in Stochastic Gradient Descentcited by 37
- 2013Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regressioncited by 32
- 2021Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scalingcited by 28
- 2015From Averaging to Acceleration, There is Only a Step-sizecited by 24
- 2017Statistical inference using SGDcited by 21
- 2014Competing with the Empirical Risk Minimizer in a Single Passcited by 20
- 2023Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stabilitycited by 20
- 2020Finite-Time Analysis of Stochastic Gradient Descent under Markov Randomnesscited by 18
- 2023Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Datacited by 18
- 2020Statistical Inference for Online Decision Making via Stochastic Gradient Descentcited by 17
- 2018First-order Newton-type Estimator for Distributed Estimation and Inferencecited by 16
- 2023High-probability sample complexities for policy evaluation with linear function approximationcited by 16
- 2021Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methodscited by 14
- 2022Optimal variance-reduced stochastic approximation in Banach spacescited by 14
- 2018HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximationcited by 13
- 2019On Constructing Confidence Region for Model Parameters in Stochastic Gradient Descent via Batch Meanscited by 13
- 2019Reducing the variance in online optimization by transporting past gradientscited by 12
- 2015Towards stability and optimality in stochastic gradient descentcited by 11
- 2017Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradientscited by 9
- 2024High Confidence Level Inference is Almost Free using Parallel Stochastic Optimizationcited by 9
- 2023Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimalitycited by 7
- 2023Online covariance estimation for stochastic gradient descent under Markovian samplingcited by 7
- 2016Local Minimax Complexity of Stochastic Convex Optimizationcited by 6
- 2020ROOT-SGD: Sharp Nonasymptotics and Near-Optimal Asymptotics in a Single Algorithmcited by 6
- 2021Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithmcited by 6
- 2022Instance-Dependent Confidence and Early Stopping for Reinforcement Learningcited by 5
- 2022Covariance Estimators for the ROOT-SGD Algorithm in Online Learningcited by 5
- 2023Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inferencecited by 5
- 2022Tuning Stochastic Gradient Algorithms for Statistical Inference via Large-Sample Asymptoticscited by 4
- 2023Convergence and Inference of Stream SGD, with Applications to Queueing Systems and Inventory Controlcited by 2
Built on by
Not cited yet.