mlsanityData to 7 Oct 2026

SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient

arXiv 1703.00102 ↗First posted 1 Mar 2017stat.ML cs.LG math.OCcited by 218

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a StochAstic Recursive grAdient algoritHm (SARAH), as well as its practical variant SARAH+, as a novel approach to the finite-sum minimization problems. Different from the vanilla SGD and other modern stochastic methods such as SVRG, S2GD, SAG and SAGA, SARAH admits a simple recursive framework for updating stochastic gradient estimates; when comparing to SAG/SAGA, SARAH does not require a storage of past gradients. The linear convergence rate of SARAH is proven under strong convexity assumption. We also prove a linear convergence rate (in the strongly convex case) for an inner loop of SARAH, the property that SVRG does not possess. Numerical experiments demonstrate the efficiency of our algorithm.

Key facts. Cited by 218 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 49; 2023: 37; 2024: 38; 2025: 28; 2026: 28.

218 papers.

Sort:

  1. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  2. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  3. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  4. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  5. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  6. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
  7. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  8. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  9. 2021Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learningcited by 51
  10. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  11. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  12. 2021Randomized Stochastic Variance-Reduced Methods for Multi-Task Stochastic Bilevel Optimizationcited by 32
38 more
  1. 2023Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditionscited by 31
  2. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 30
  3. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
  4. 2021ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computationcited by 28
  5. 2021Enhanced Bilevel Optimization via Bregman Distancecited by 28
  6. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 23
  7. 2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
  8. 2022Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimizationcited by 22
  9. 2022Can 5th Generation Local Training Methods Support Client Sampling? Yes!cited by 22
  10. 2021A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex Optimizationcited by 21
  11. 2022AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimationcited by 21
  12. 2021FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learningcited by 19
  13. 2022Faster federated optimization under second-order similaritycited by 18
  14. 2021Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guaranteescited by 17
  15. 2022Optimal Algorithms for Stochastic Multi-Level Compositional Optimizationcited by 17
  16. 2022Stochastic Halpern Iteration with Variance Reduction for Stochastic Monotone Inclusionscited by 17
  17. 2021Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Informationcited by 16
  18. 2021DESTRESS: Computation-Optimal and Communication-Efficient Decentralized Nonconvex Finite-Sum Optimizationcited by 16
  19. 2022Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimizationcited by 16
  20. 2022Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimizationcited by 16
  21. 2023Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimizationcited by 16
  22. 2023A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimizationcited by 16
  23. 2021AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methodscited by 15
  24. 2022PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimationcited by 15
  25. 2022Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimizationcited by 15
  26. 2021Better SGD using Second-order Momentumcited by 14
  27. 2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
  28. 2022Optimal variance-reduced stochastic approximation in Banach spacescited by 14
  29. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  30. 2023On Momentum-Based Gradient Methods for Bilevel Optimization with Nonconvex Lower-Levelcited by 14
  31. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  32. 2024Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reductioncited by 13
  33. 2021FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learningcited by 12
  34. 2022Optimization with Access to Auxiliary Informationcited by 12
  35. 2023Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Ratescited by 12
  36. 2022Momentum-Based Policy Gradient with Second-Order Informationcited by 11
  37. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11
  38. 2025Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noisecited by 11

50 most cited of 218 All 218 (JSON)