mlsanityData to 7 Oct 2026

Variance-Reduced Methods for Machine Learning

arXiv 2010.00892 ↗First posted 2 Oct 2020cs.LG math.OC stat.MLcited by 79

Not on the map. Why?

Abstract (arXiv)

Stochastic optimization lies at the heart of machine learning, and its cornerstone is stochastic gradient descent (SGD), a method introduced over 60 years ago. The last 8 years have seen an exciting new development: variance reduction (VR) for stochastic optimization methods. These VR methods excel in settings where more than one pass through the training data is allowed, achieving a faster convergence than SGD in theory as well as practice. These speedups underline the surge of interest in VR methods and the fast-growing body of work on this topic. This review covers the key principles and main developments behind VR methods for optimization with finite data sets and is aimed at non-expert readers. We focus mainly on the convex setting, and leave pointers to readers interested in extensions for minimizing non-convex functions.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 6; 2023: 19; 2024: 19; 2025: 13; 2026: 7.

79 papers.

Sort:

By area:

  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  2. 2021Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learningcited by 51
  3. 2021Stochastic Variance Reduction for Variational Inequality Methodscited by 46
  4. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  5. 2021Query Complexity of Least Absolute Deviation Regression via Robust Uniform Convergencecited by 24
  6. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  7. 2021Random Reshuffling with Variance Reduction: New Analysis and Better Ratescited by 19
  8. 2022EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimizationcited by 17
  9. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 15
  10. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  11. 2021MURANA: A Generic Framework for Stochastic Variance-Reduced Optimizationcited by 13
  12. 2022CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimizationcited by 11
38 more
  1. 2024LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compressioncited by 11
  2. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  3. 2023Linear Convergence of Black-Box Variational Inference: Should We Stick the Landing?cited by 10
  4. 2023Variance Reduced Halpern Iteration for Finite-Sum Monotone Inclusionscited by 10
  5. 2024Exact Gauss-Newton Optimization for Training Deep Neural Networkscited by 10
  6. 2021IntSGD: Adaptive Floatless Compression of Stochastic Gradientscited by 8
  7. 2022RECAPP: Crafting a More Efficient Catalyst for Convex Optimizationcited by 8
  8. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  9. 2024Gap-Dependent Bounds for Q-Learning using Reference-Advantage Decompositioncited by 8
  10. 2023A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learningcited by 7
  11. 2023Adaptive Federated Learning with Auto-Tuned Clientscited by 7
  12. 2023Balancing Act: Constraining Disparate Impact in Sparse Modelscited by 7
  13. 2023A Coefficient Makes SVRG Effectivecited by 7
  14. 2021EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedbackcited by 6
  15. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  16. 2024Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networkscited by 6
  17. 2024Efficient Optimization Algorithms for Linear Adversarial Trainingcited by 6
  18. 2023Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correctioncited by 5
  19. 2023Coreset Markov Chain Monte Carlocited by 5
  20. 2021First-Order Methods for Convex Optimizationcited by 4
  21. 2023A One-Sample Decentralized Proximal Algorithm for Non-Convex Stochastic Composite Optimizationcited by 4
  22. 2024A Unified Theory of Stochastic Proximal Point Methods without Smoothnesscited by 4
  23. 2024Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodscited by 4
  24. 2023TAMUNA: Doubly Accelerated Distributed Optimization under Partial Participationcited by 3
  25. 2023Byzantine-Robust Loopless Stochastic Variance-Reduced Gradientcited by 3
  26. 2023Sketch-and-Project Meets Newton Method: Global mathcal O(k-2) Convergence with Low-Rank Updatescited by 3
  27. 2023Critical Influence of Overparameterization on Sharpness-aware Minimizationcited by 3
  28. 2024On the Convergence of FedProx with Extrapolation and Inexact Proxcited by 3
  29. 2025The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applicationscited by 3
  30. 2025Variance Matters: Improving Domain Adaptation via Stratified Samplingcited by 3
  31. 2021Delayed Projection Techniques for Linearly Constrained Problems: Convergence Rates, Acceleration, and Applicationscited by 2
  32. 2021LoSAC: An Efficient Local Stochastic Average Control Method for Federated Optimizationcited by 2
  33. 2023A Large Deviations Perspective on Policy Gradient Algorithmscited by 2
  34. 2024Demystifying SGD with Doubly Stochastic Gradientscited by 2
  35. 2024An Effective Dynamic Gradient Calibration Method for Continual Learningcited by 2
  36. 2025Variance Reduction Methods Do Not Need to Compute Full Gradients: Improved Efficiency through Shufflingcited by 2
  37. 2025GC-Fed: Gradient Centralized Federated Learning with Partial Client Participationcited by 2
  38. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2

50 most cited of 79 All 79 (JSON)