mlsanityData to 14 Oct 2026

A Coefficient Makes SVRG Effective

arXiv 2311.05589 ↗First posted 9 Nov 2023cs.LG math.OC stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Stochastic Variance Reduced Gradient (SVRG), introduced by Johnson & Zhang (2013), is a theoretically compelling optimization method. However, as Defazio & Bottou (2019) highlight, its effectiveness in deep learning is yet to be proven. In this work, we demonstrate the potential of SVRG in optimizing real-world neural networks. Our empirical analysis finds that, for deeper neural networks, the strength of the variance reduction term in SVRG should be smaller and decrease as training progresses. Inspired by this, we introduce a multiplicative coefficient α to control the strength and adjust it through a linear decay schedule. We name our method α-SVRG. Our results show α-SVRG better optimizes models, consistently reducing training loss compared to the baseline and standard SVRG across various model architectures and multiple image classification datasets. We hope our findings encourage further exploration into variance reduction techniques in deep learning. Code is available at github.com/davidyyd/alpha-SVRG.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 47 of its 68 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

47 of 68 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  12. 2017Random Erasing Data Augmentationcited by 1,239
35 more
  1. 2016Deep Networks with Stochastic Depthcited by 979
  2. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  3. 2021Going deeper with Image Transformerscited by 764
  4. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  5. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  6. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  7. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  8. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  9. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  10. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  11. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  12. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  13. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
  14. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
  15. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  16. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  17. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
  18. 2020Variance-Reduced Methods for Machine Learningcited by 79
  19. 2014A Proximal Stochastic Gradient Method with Progressive Variance Reductioncited by 78
  20. 2016Variance Reduction for Faster Non-Convex Optimizationcited by 72
  21. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  22. 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
  23. 2015A Universal Catalyst for First-Order Optimizationcited by 54
  24. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  25. 2017Catalyst Acceleration for First-order Convex Optimization: from Theory to Practicecited by 39
  26. 2014Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learningcited by 35
  27. 2018A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimizationcited by 33
  28. 2021ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computationcited by 29
  29. 2023Dropout Reduces Underfittingcited by 21
  30. 2016A Simple Practical Accelerated Method for Finite Sumscited by 18
  31. 2022Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimizationcited by 15
  32. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  33. 2014Finito: A Faster, Permutable Incremental Gradient Method for Big Data Problemscited by 10
  34. 2022Divergence Results and Convergence of a Variance Reduced Version of ADAMcited by 8
  35. 2020Variance Reduction with Sparse Gradientscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  2. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  3. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  4. 2025Non-Convex Federated Optimization under Cost-Aware Client Selectioncited by 1
  5. 2025Convergence Analysis of alpha-SVRG under Strong Convexitycited by 0
  6. 2025SVRG and Beyond via Posterior Correctioncited by 0
  7. 2026Personalized Federated Learning for Gradient Alignmentcited by 0