mlsanityData to 7 Oct 2026

On the Ineffectiveness of Variance Reduced Optimization for Deep Learning

arXiv 1812.04529 ↗First posted 11 Dec 2018cs.LG stat.MLcited by 48

Not on the map. Why?

Abstract (arXiv)

The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 4; 2023: 10; 2024: 8; 2025: 7; 2026: 4.

48 papers.

Sort:

By area:

  1. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  2. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  3. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  4. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  5. 2021Anarchic Federated Learningcited by 42
  6. 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 41
  7. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  8. 2021Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspectivecited by 15
  9. 2021Better SGD using Second-order Momentumcited by 14
  10. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  11. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  12. 2021Variance Reduced Training with Stratified Sampling for Forecasting Modelscited by 13
36 more
  1. 2021Secure Distributed Training at Scalecited by 11
  2. 2024Federated Optimization with Doubly Regularized Drift Correctioncited by 10
  3. 2024Exact Gauss-Newton Optimization for Training Deep Neural Networkscited by 10
  4. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  5. 2021Coordinating Momenta for Cross-silo Federated Learningcited by 9
  6. 2023Statistically Optimal Force Aggregation for Coarse-Graining Molecular Dynamicscited by 9
  7. 2022Partial Model Averaging in Federated Learning: Performance Guarantees and Benefitscited by 8
  8. 2022On the effectiveness of partial variance reduction in federated learning with heterogeneous datacited by 8
  9. 2023Handling Data Heterogeneity via Architectural Design for Federated Visual Recognitioncited by 8
  10. 2021Stochastic Reweighted Gradient Descentcited by 7
  11. 2021Training Structured Neural Networks Through Manifold Identification and Variance Reductioncited by 7
  12. 2023Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learningcited by 7
  13. 2023A Coefficient Makes SVRG Effectivecited by 7
  14. 2021Zeroth and First Order Stochastic Frank-Wolfe Algorithms for Constrained Optimizationcited by 6
  15. 2023Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentumcited by 3
  16. 2025Bant: Byzantine Antidote via Trial Function and Trust Scorescited by 3
  17. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  18. 2021Variance Reduction in Deep Learning: More Momentum is All You Needcited by 2
  19. 2022Tackling benign nonconvexity with smoothing and stochastic gradientscited by 2
  20. 2024Convergence Conditions for Stochastic Line Search Based Optimization of Over-parametrized Modelscited by 2
  21. 2025Gradient Prediction with Control Variates in the Cheap-Forward Regimecited by 2
  22. 2023Balancing Privacy and Performance for Private Federated Learning Algorithmscited by 1
  23. 2024Deep Companion Learning: Enhancing Generalization Through Historical Consistencycited by 1
  24. 2024Effectively Leveraging Momentum Terms in Stochastic Line Search Frameworks for Fast Optimization of Finite-Sum Problemscited by 1
  25. 2025Faster Gradient Methods for Highly-Smooth Stochastic Bilevel Optimizationcited by 1
  26. 2021Scalable Projection-Free Optimizationcited by 0
  27. 2021Training With Data Dependent Dynamic Learning Ratescited by 0
  28. 2023Ordering for Non-Replacement SGDcited by 0
  29. 2024Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolationcited by 0
  30. 2024Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learningcited by 0
  31. 2025Convergence Analysis of alpha-SVRG under Strong Convexitycited by 0
  32. 2025SVRG and Beyond via Posterior Correctioncited by 0
  33. 2026Accelerated Methods with Complexity Separation Under Data Similarity for Federated Learning Problemscited by 0
  34. 2026Rennala MVR: Improved Time Complexity for Parallel Stochastic Optimization via Momentum-Based Variance Reductioncited by 0
  35. 2026Accelerated and Stable Convergence with Anchored Generalized Optimistic Methodcited by 0
  36. 2026Optimizers for Diffusion Models: A Controlled Benchmarkcited by 0