mlsanityData to 13 Oct 2026

AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods

arXiv 2102.09700 ↗First posted 19 Feb 2021cs.LG math.OCcited by 15

Not on the map. Why?

Abstract (arXiv)

We present AI-SARAH, a practical variant of SARAH. As a variant of SARAH, this algorithm employs the stochastic recursive gradient yet adjusts step-size based on local geometry. AI-SARAH implicitly computes step-size and efficiently estimates local Lipschitz smoothness of stochastic functions. It is fully adaptive, tune-free, straightforward to implement, and computationally efficient. We provide technical insight and intuitive illustrations on its design and convergence. We conduct extensive empirical analysis and demonstrate its strong performance compared with its classical counterparts and other state-of-the-art first-order methods in solving convex machine learning problems.

Key facts. Cited by 15 papers in the map (data through 13 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

30 of 47 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,142
  3. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 769
  4. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 718
  5. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  6. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 303
  7. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  8. 2019SGD: General Analysis and Improved Ratescited by 206
  9. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  10. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  11. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  12. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
18 more
  1. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 133
  2. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  3. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
  4. 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 66
  5. 2017Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methodscited by 49
  6. 2015Equilibrated adaptive learning rates for non-convex optimizationcited by 42
  7. 2020Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimizationcited by 36
  8. 2018Stochastic Quasi-Gradient Methods: Variance Reduction via Jacobian Sketchingcited by 34
  9. 2016Barzilai-Borwein Step Size for Stochastic Gradient Descentcited by 33
  10. 2020Stochastic Hamiltonian Gradient Methods for Smooth Gamescited by 32
  11. 2016A Simple Practical Accelerated Method for Finite Sumscited by 18
  12. 2020Adaptivity of Stochastic Gradient Methods for Nonconvex Optimizationcited by 16
  13. 2017Linearly convergent stochastic heavy ball method for minimizing generalization errorcited by 14
  14. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  15. 2015Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Settingcited by 10
  16. 2019Almost Tune-Free Variance Reductioncited by 7
  17. 2019Accelerating Mini-batch SARAH by Step Size Rulescited by 4
  18. 2019Adaptive Step Sizes in Variance Reduction via Regularizationcited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 3; 2024: 2; 2025: 2; 2026: 4.

15 papers.

Sort:

  1. 2024Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performancecited by 13
  2. 2021A Fairness Analysis on Private Aggregation of Teacher Ensemblescited by 11
  3. 2024Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGradcited by 8
  4. 2023Stochastic Gradient Descent with Preconditioned Polyak Step-sizecited by 7
  5. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  6. 2023On the Fairness Impacts of Private Ensembles Modelscited by 4
  7. 2022FLECS-CGD: A Federated Learning Second-Order Framework via Compression and Sketching with Compressed Gradient Differencescited by 2
  8. 2025Glocal Smoothness: Line search and adaptive step sizes can help in theory too!cited by 2
  9. 2025Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradientscited by 2
  10. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  11. 2026Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Schedulercited by 1
  12. 2021Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimizationcited by 0
3 more
  1. 2022An Adaptive Incremental Gradient Method With Support for Non-Euclidean Normscited by 0
  2. 2026Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Sizecited by 0
  3. 2026Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothnesscited by 0