mlsanityData to 7 Oct 2026

Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives

arXiv 1506.01972 ↗First posted 5 Jun 2015cs.LG cs.DS math.OCcited by 38

Not on the map. Why?

Abstract (arXiv)

Many classical algorithms are found until several years later to outlive the confines in which they were conceived, and continue to be relevant in unforeseen settings. In this paper, we show that SVRG is one such method: being originally designed for strongly convex objectives, it is also very robust in non-strongly convex or sum-of-non-convex settings. More precisely, we provide new analysis to improve the state-of-the-art running times in both settings by either applying SVRG or its novel variant. Since non-strongly convex objectives include important examples such as Lasso or logistic regression, and sum-of-non-convex objectives include famous examples such as stochastic PCA and is even believed to be related to training deep neural nets, our results also imply better performances in these applications.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 11; 2023: 6; 2024: 6; 2025: 5; 2026: 2.

38 papers.

Sort:

  1. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 88
  2. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  3. 2021Stochastic Variance Reduction for Variational Inequality Methodscited by 46
  4. 2021Fast Distributionally Robust Learning with Variance Reduced Min-Max Optimizationcited by 19
  5. 2021Distributed Learning Systems with First-order Methodscited by 17
  6. 2022Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimizationcited by 15
  7. 2022Hessian Averaging in Stochastic Newton Methods Achieves Superlinear Convergencecited by 14
  8. 2025Towards Weaker Variance Assumptions for Stochastic Optimizationcited by 13
  9. 2024Efficient Algorithms for Empirical Group Distributionally Robust Optimization and Beyondcited by 11
  10. 2024BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learningcited by 10
  11. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  12. 2021ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Methodcited by 9
26 more
  1. 2022Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batchescited by 9
  2. 2021Random-reshuffled SARAH does not need a full gradient computationscited by 7
  3. 2022Efficiency Ordering of Stochastic Gradient Descentcited by 6
  4. 2022Trade-off Between Efficiency and Consistency for Removal-based Explanationscited by 6
  5. 2023Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Featurescited by 6
  6. 2021Memory Augmented Optimizers for Deep Learningcited by 5
  7. 2021Practical Schemes for Finding Near-Stationary Points of Convex Finite-Sumscited by 4
  8. 2022Adaptive Accelerated (Extra-)Gradient Methods with Variance Reductioncited by 4
  9. 2023Faster Stochastic Algorithms for Minimax Optimization under Polyak--Lojasiewicz Conditionscited by 4
  10. 2023Variance reduction techniques for stochastic proximal point algorithmscited by 3
  11. 2024Quantum Algorithms and Lower Bounds for Finite-Sum Optimizationcited by 3
  12. 2026PPI-SVRG: Unifying Prediction-Powered Inference and Variance Reduction for Semi-Supervised Optimizationcited by 3
  13. 2021Delayed Projection Techniques for Linearly Constrained Problems: Convergence Rates, Acceleration, and Applicationscited by 2
  14. 2023EMO: Episodic Memory Optimization for Few-Shot Meta-Learningcited by 2
  15. 2024Variance-Reduced Fast Krasnoselkii-Mann Methods for Finite-Sum Root-Finding Problemscited by 2
  16. 2022SADAM: Stochastic Adam, A Stochastic Operator for First-Order Gradient-based Optimizercited by 1
  17. 2022Theoretical Analysis of Primal-Dual Algorithm for Non-Convex Stochastic Decentralized Optimizationcited by 1
  18. 2022Decomposable Non-Smooth Convex Optimization with Nearly-Linear Gradient Oracle Complexitycited by 0
  19. 2022SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalitiescited by 0
  20. 2022Adaptive Stochastic Optimisation of Nonconvex Composite Objectivescited by 0
  21. 2023Computing Approximate ℓp Sensitivitiescited by 0
  22. 2024Efficient Continual Finite-Sum Minimizationcited by 0
  23. 2025Permutation Randomization on Nonsmooth Nonconvex Optimization: A Theoretical and Experimental Studycited by 0
  24. 2025VAMO: Efficient Zeroth-Order Variance Reduction for SGD with Faster Convergencecited by 0
  25. 2025Adjusted Shuffling SARAH: Advancing Complexity Analysis via Dynamic Gradient Weightingcited by 0
  26. 2026Rennala MVR: Improved Time Complexity for Parallel Stochastic Optimization via Momentum-Based Variance Reductioncited by 0