mlsanityData to 7 Oct 2026

Stochastic Variance Reduction for Nonconvex Optimization

arXiv 1603.06160 ↗First posted 19 Mar 2016math.OC cs.LG cs.NEcited by 143

Not on the map. Why?

Abstract (arXiv)

We study nonconvex finite-sum problems and analyze stochastic variance reduced gradient (SVRG) methods for them. SVRG and related methods have recently surged into prominence for convex optimization given their edge over stochastic gradient descent (SGD); but their theoretical analysis almost exclusively assumes convexity. In contrast, we prove non-asymptotic rates of convergence (to stationary points) of SVRG for nonconvex optimization, and show that it is provably faster than SGD and gradient descent. We also analyze a subclass of nonconvex problems on which SVRG attains linear convergence to the global optimum. We extend our analysis to mini-batch variants of SVRG, showing (theoretical) linear speedup due to mini-batching in parallel settings.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 22; 2023: 25; 2024: 24; 2025: 17; 2026: 13.

143 papers.

Sort:

By area:

  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  2. 2025Learning to Reason under Off-Policy Guidancecited by 266
  3. 2021Decentralized Federated Averagingcited by 114
  4. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  5. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  6. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  7. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  8. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  9. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
  10. 2021Stochastic Variance Reduction for Variational Inequality Methodscited by 46
  11. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  12. 2021Non-convex Distributionally Robust Optimization: Non-asymptotic Analysiscited by 39
38 more
  1. 2021On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learningcited by 35
  2. 2023Deep networks for system identification: a Surveycited by 35
  3. 2023Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditionscited by 31
  4. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 30
  5. 2021ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computationcited by 28
  6. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 23
  7. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  8. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  9. 2021Fast Distributionally Robust Learning with Variance Reduced Min-Max Optimizationcited by 19
  10. 2021Breaking the Linear Iteration Cost Barrier for Some Well-known Conditional Gradient Methods Using MaxIP Data-structurescited by 19
  11. 2021A Biased Graph Neural Network Sampler with Near-Optimal Regretcited by 17
  12. 2021Distributed Learning Systems with First-order Methodscited by 17
  13. 2021Global Convergence and Stability of Stochastic Gradient Descentcited by 17
  14. 2022Stochastic Halpern Iteration with Variance Reduction for Stochastic Monotone Inclusionscited by 17
  15. 2021A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guaranteecited by 16
  16. 2021DESTRESS: Computation-Optimal and Communication-Efficient Decentralized Nonconvex Finite-Sum Optimizationcited by 16
  17. 2022Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentationcited by 16
  18. 2022Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimizationcited by 15
  19. 2023Accelerating Hybrid Federated Learning Convergence under Partial Participationcited by 15
  20. 2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
  21. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  22. 2021Optimizing Large-Scale Hyperparameters via Automated Learning Algorithmcited by 13
  23. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  24. 2023How to escape sharp minima with random perturbationscited by 13
  25. 2024Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reductioncited by 13
  26. 2022Optimization with Access to Auxiliary Informationcited by 12
  27. 2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
  28. 2024Efficient Algorithms for Empirical Group Distributionally Robust Optimization and Beyondcited by 11
  29. 2022Zeroth-Order Alternating Gradient Descent Ascent Algorithms for a Class of Nonconvex-Nonconcave Minimax Problemscited by 10
  30. 2024Accelerating Heterogeneous Federated Learning with Closed-form Classifierscited by 10
  31. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  32. 2021On the Last Iterate Convergence of Momentum Methodscited by 9
  33. 2021ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Methodcited by 9
  34. 2022Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimizationcited by 9
  35. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  36. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  37. 2021GT-STORM: Taming Sample, Communication, and Memory Complexities in Decentralized Non-Convex Learningcited by 8
  38. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8

50 most cited of 143 All 143 (JSON)