# Adaptive variational Bayes: Optimality, computation and applications

arXiv: https://arxiv.org/abs/2109.03204 · HTML: https://mlsanity.com/paper/2109.03204/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 23 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 6 of 6)

- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Variational Bayes for high-dimensional linear regression with sparse priors](https://mlsanity.com/paper/1904.07150/) (2019)
- [Mixture models with a prior on the number of components](https://mlsanity.com/paper/1502.06241/) (2015)
- [Variable Selection Consistency of Gaussian Process Regression](https://mlsanity.com/paper/1912.05738/) (2019)
- [Probabilistic community detection with unknown number of communities](https://mlsanity.com/paper/1602.08062/) (2016)
- [Gibbs posterior concentration rates under sub-exponential type losses](https://mlsanity.com/paper/2012.04505/) (2020)

## Built on by (most cited first, 17 of 17)

- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Posterior and variational inference for deep neural networks with heavy-tailed weights](https://mlsanity.com/paper/2406.03369/) (2024)
- [Deep Horseshoe Gaussian Processes](https://mlsanity.com/paper/2403.01737/) (2024)
- [Machine Learning and the Future of Bayesian Computation](https://mlsanity.com/paper/2304.11251/) (2023)
- [Adaptive sparse variational approximations for Gaussian process regression](https://mlsanity.com/paper/2504.03321/) (2025)
- [Masked Bayesian Neural Networks : Computation and Optimality](https://mlsanity.com/paper/2206.00853/) (2022)
- [Convergence of Statistical Estimators via Mutual Information Bounds](https://mlsanity.com/paper/2412.18539/) (2024)
- [Understanding the Trade-offs in Accuracy and Uncertainty Quantification: Architecture and Inference Choices in Bayesian Neural Networks](https://mlsanity.com/paper/2503.11808/) (2025)
- [Robust and Scalable Variational Bayes](https://mlsanity.com/paper/2504.12528/) (2025)
- [Scalable and adaptive variational Bayes methods for Hawkes processes](https://mlsanity.com/paper/2212.00293/) (2022)
- [Variational Bayesian Bow tie Neural Networks with Shrinkage](https://mlsanity.com/paper/2411.11132/) (2024)
- [Variational empirical Bayes variable selection in high-dimensional logistic regression](https://mlsanity.com/paper/2502.10532/) (2025)
- [Variational bagging: a robust approach for Bayesian uncertainty quantification](https://mlsanity.com/paper/2511.20594/) (2025)
- [Beyond Expected Information Gain: Stable Bayesian Optimal Experimental Design with Integral Probability Metrics and Plug-and-Play Extensions](https://mlsanity.com/paper/2604.21849/) (2026)
- [Posterior Concentration of Bayesian Physics-Informed Neural Networks for Elliptic PDEs](https://mlsanity.com/paper/2605.08672/) (2026)
- [Rate-optimal neural boundary detection from unlabeled noisy images](https://mlsanity.com/paper/2606.00715/) (2026)
- [Fast rates in Bayesian online learning with approximate posteriors](https://mlsanity.com/paper/2608.25706/) (2026)

## Abstract (arXiv)

In this paper, we explore adaptive inference based on variational Bayes. Although several studies have been conducted to analyze the contraction properties of variational posteriors, there is still a lack of a general and computationally tractable variational Bayes method that performs adaptive inference. To fill this gap, we propose a novel adaptive variational Bayes framework, which can operate on a collection of models. The proposed framework first computes a variational posterior over each individual model separately and then combines them with certain weights to produce a variational posterior over the entire model. It turns out that this combined variational posterior is the closest member to the posterior over the entire model in a predefined family of approximating distributions. We show that the adaptive variational Bayes attains optimal contraction rates adaptively under very general conditions. We also provide a methodology to maintain the tractability and adaptive optimality of the adaptive variational Bayes even in the presence of an enormous number of individual models, such as sparse models. We apply the general results to several examples, including deep learning and sparse factor models, and derive new and adaptive inference results. In addition, we characterize an implicit regularization effect of variational Bayes and show that the adaptive variational posterior can utilize this.