mlsanityData to 14 Oct 2026

SVRG and Beyond via Posterior Correction

arXiv 2512.01930 ↗First posted 1 Dec 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections. Originally proposed over a decade ago, these methods have never been connected to any Bayesian method at a fundamental level. Here, we fill this gap and derive surprising new connections of SVRG to a recently proposed Bayesian method called `posterior correction'. Our main contribution is to show that SVRG can be recovered as a special case of posterior-correction over isotropic-Gaussian posteriors. Novel extensions of SVRG are automatically obtained by using more flexible exponential-family posteriors. We derive two new such extensions by using Gaussian families: a Newton-like variant with novel Hessian corrections, and an Adam-like extension that scales to large problems. Our work is the first to connect SVRG to Bayes and use it to speed-up training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

42 of 54 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  12. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
30 more
  1. 2013Black Box Variational Inferencecited by 319
  2. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  3. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  4. 2017Advances in Variational Inferencecited by 236
  5. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  6. 2013Expectation Propagation for approximate Bayesian inferencecited by 174
  7. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  8. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  9. 2021The Bayesian Learning Rulecited by 99
  10. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  11. 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
  12. 2018Quasi-hyperbolic momentum and Adam for deep learningcited by 48
  13. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  14. 2018Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Modelscited by 38
  15. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  16. 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 23
  17. 2015Stop Wasting My Gradients: Practical SVRGcited by 14
  18. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  19. 2019Reducing the variance in online optimization by transporting past gradientscited by 12
  20. 2023Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methodscited by 12
  21. 2023Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learningcited by 10
  22. 2022Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batchescited by 9
  23. 2023A Coefficient Makes SVRG Effectivecited by 7
  24. 2025Knowledge Adaptation as Posterior Correctioncited by 6
  25. 2017Tracking the gradients using the Hessian: A new look at variance reducing stochastic methodscited by 5
  26. 2022Stochastic Gradient Methods with Preconditioned Updatescited by 5
  27. 2025SAPPHIRE: Preconditioned Stochastic Variance Reduction for Faster Large-Scale Statistical Learningcited by 4
  28. 2024Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradientscited by 3
  29. 2014Smoothed Gradients for Stochastic Variational Inferencecited by 2
  30. 2021Variance Reduction in Deep Learning: More Momentum is All You Needcited by 2

Built on by

Not cited yet.