mlsanityData to 7 Oct 2026

Stochastic Recursive Gradient Algorithm for Nonconvex Optimization

arXiv 1705.07261 ↗First posted 20 May 2017stat.ML cs.LG math.OCcited by 20

Not on the map. Why?

Abstract (arXiv)

In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses. We provide a sublinear convergence rate (to stationary points) for general nonconvex functions and a linear convergence rate for gradient dominated functions, both of which have some advantages compared to other modern stochastic gradient algorithms for nonconvex losses.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 5; 2023: 4; 2024: 4; 2025: 1; 2026: 2.

20 papers.

Sort:

  1. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  2. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  3. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  4. 2022Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimizationcited by 16
  5. 2022Zeroth-Order Alternating Gradient Descent Ascent Algorithms for a Class of Nonconvex-Nonconcave Minimax Problemscited by 10
  6. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  7. 2024Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGradcited by 8
  8. 2021Random-reshuffled SARAH does not need a full gradient computationscited by 7
  9. 2023Improving Accelerated Federated Learning with Compression and Importance Samplingcited by 7
  10. 2023Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Featurescited by 6
  11. 2023Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimizationcited by 2
  12. 2024Probabilistic Guarantees of Stochastic Recursive Gradient in Non-Convex Finite Sum Problemscited by 2
8 more
  1. 2021Faster Perturbed Stochastic Gradient Methods for Finding Local Minimacited by 1
  2. 2022Riemannian Stochastic Gradient Method for Nested Composition Optimizationcited by 1
  3. 2024Zeroth-Order Stochastic Mirror Descent Algorithms for Minimax Excess Risk Optimizationcited by 1
  4. 2026AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learningcited by 1
  5. 2026OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimalitycited by 1
  6. 2021Scalable Projection-Free Optimizationcited by 0
  7. 2022Versatile Single-Loop Method for Gradient Estimator: First and Second Order Optimality, and its Application to Federated Learningcited by 0
  8. 2022SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalitiescited by 0