mlsanityData to 14 Oct 2026

Large-scale empirical tuning and comparison of default optimizers for variational inference

arXiv 2606.07841 ↗First posted 5 Jun 2026stat.CO cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Black-box variational inference (BBVI) is a methodology for posterior approximation that relies on stochastic optimization. In practice, the stochastic optimizers underpinning BBVI generally require extensive problem-specific tuning, which undermines its promise as a truly "black box" inference algorithm. However, over the past decade, many new adaptive stochastic optimization algorithms have been developed that reduce or remove entirely the need for tuning. In this work, we investigate this new collection of adaptive methods in the context of BBVI, with the goal of establishing the current state of the art in tuning-free optimization-based inference. In particular, we present a large-scale empirical evaluation of 56 stochastic gradient-based optimization algorithms applied to 1092 Bayesian inference optimization problems, involving over 550,000 individual optimization runs and 15 core-years of compute. The optimization algorithms we evaluate are chosen to represent a wide spectrum of recent approaches and the benchmark problems are chosen to span a range of difficulty, with posterior target dimension 1-10^4, condition number 1-10^8, and a range of variational families. Our results show that no single method dominates, but running a selection of 5 algorithms suffices to reliably get close to the best-possible observed performance. We thus provide a strong baseline for applications where expert tuning is not possible and for comparison when developing new stochastic optimization algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 71 of its 114 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

71 of 114 references.

References · 71

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Layer Normalizationcited by 5,669
  4. 2015Variational Inference with Normalizing Flowscited by 2,170
  5. 2016Variational Inference: A Review for Statisticianscited by 1,472
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  7. 2019On the Convergence of Adam and Beyondcited by 841
  8. 2015Importance Weighted Autoencoderscited by 456
  9. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  10. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  11. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  12. 2013Black Box Variational Inferencecited by 319
38 more
  1. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  2. 2013Expectation Propagation for approximate Bayesian inferencecited by 174
  3. 2016Automatic Differentiation Variational Inferencecited by 159
  4. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  5. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  6. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  7. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
  8. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  9. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  10. 2016R'enyi Divergence Variational Inferencecited by 104
  11. 2021The Bayesian Learning Rulecited by 99
  12. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 98
  13. 2019Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMCcited by 93
  14. 2015Hierarchical Variational Modelscited by 89
  15. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 88
  16. 2022Variational inference via Wasserstein gradient flowscited by 85
  17. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  18. 2018Importance Weighting and Variational Inferencecited by 55
  19. 2019Revisiting the Polyak step sizecited by 55
  20. 2018Yes, but Did It Work?: Evaluating Variational Inferencecited by 53
  21. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  22. 2022Making SGD Parameter-Freecited by 50
  23. 2019Training Neural Networks for and by Interpolationcited by 45
  24. 2017Variational Inference using Implicit Distributionscited by 39
  25. 2021MCMC Variational Inference via Uncorrected Hamiltonian Annealingcited by 39
  26. 2015Black-box α-divergence Minimizationcited by 37
  27. 2021Differentiable Annealed Importance Sampling and the Perils of Gradient Noisecited by 36
  28. 2023Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Spacecited by 36
  29. 2021Challenges and Opportunities in High-dimensional Variational Inferencecited by 35
  30. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  31. 2013Automated Variational Inference in Probabilistic Programmingcited by 34
  32. 2018Unbiased Implicit Variational Inferencecited by 33
  33. 2019Provable Smoothness Guarantees for Black-Box Variational Inferencecited by 29
  34. 2023Provable convergence guarantees for black-box variational inferencecited by 29
  35. 2020Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradientscited by 27
  36. 2021Parameter-free Stochastic Optimization of Variationally Coherent Functionscited by 27
  37. 2019Validated Variational Inference via Practical Posterior Error Boundscited by 26
  38. 2018Quasi-Monte Carlo Variational Inferencecited by 24

50 most cited of 71 All 71 (JSON)

Built on by

Not cited yet.