# Covariances, Robustness, and Variational Bayes

arXiv: https://arxiv.org/abs/1709.02536 · HTML: https://mlsanity.com/paper/1709.02536/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2017.

## Built on by (most cited first, 30 of 40)

- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [On the Convergence of Black-Box Variational Inference](https://mlsanity.com/paper/2305.15349/) (2023)
- [Sensitivity-Aware Amortized Bayesian Inference](https://mlsanity.com/paper/2310.11122/) (2023)
- [Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Box](https://mlsanity.com/paper/2304.05527/) (2023)
- [The Memory Perturbation Equation: Understanding Model's Sensitivity to Data](https://mlsanity.com/paper/2310.19273/) (2023)
- [Bayesian Influence Functions for Hessian-Free Data Attribution](https://mlsanity.com/paper/2509.26544/) (2025)
- [Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs](https://mlsanity.com/paper/2403.13748/) (2024)
- [The Shrinkage-Delinkage Trade-off: An Analysis of Factorized Gaussian Approximations for Variational Inference](https://mlsanity.com/paper/2302.09163/) (2023)
- [Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrix](https://mlsanity.com/paper/2410.11067/) (2024)
- [The Loss Kernel: A Geometric Probe for Deep Learning Interpretability](https://mlsanity.com/paper/2509.26537/) (2025)
- [Toward a Taxonomy of Trust for Probabilistic Machine Learning](https://mlsanity.com/paper/2112.03270/) (2021)
- [Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection](https://mlsanity.com/paper/2207.11208/) (2022)
- [Influence Dynamics and Stagewise Data Attribution](https://mlsanity.com/paper/2510.12071/) (2025)
- [Sample Average Approximation for Black-Box VI](https://mlsanity.com/paper/2304.06803/) (2023)
- [Evaluating Sensitivity to the Stick-Breaking Prior in Bayesian Nonparametrics](https://mlsanity.com/paper/2107.03584/) (2021)
- [Provably Scalable Black-Box Variational Inference with Structured Variational Families](https://mlsanity.com/paper/2401.10989/) (2024)
- [Stability of Mean-Field Variational Inference](https://mlsanity.com/paper/2506.07856/) (2025)
- [Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry](https://mlsanity.com/paper/2511.01064/) (2025)
- [Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning](https://mlsanity.com/paper/2605.07980/) (2026)
- [VFDS: Variational Foresight Dynamic Selection in Bayesian Neural Networks for Efficient Human Activity Recognition](https://mlsanity.com/paper/2204.00130/) (2022)
- [An accelerated first-order regularized momentum descent ascent algorithm for stochastic nonconvex-concave minimax problems](https://mlsanity.com/paper/2310.15448/) (2023)
- [VISA: Variational Inference with Sequential Sample-Average Approximations](https://mlsanity.com/paper/2403.09429/) (2024)
- [Logistic Variational Bayes Revisited](https://mlsanity.com/paper/2406.00713/) (2024)
- [Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models](https://mlsanity.com/paper/2407.08843/) (2024)
- [Robust and Scalable Variational Bayes](https://mlsanity.com/paper/2504.12528/) (2025)
- [Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference](https://mlsanity.com/paper/2505.21721/) (2025)
- [A Bayesian Information-Theoretic Approach to Data Attribution](https://mlsanity.com/paper/2604.03858/) (2026)
- [Symmetry Guarantees Statistic Recovery in Variational Inference](https://mlsanity.com/paper/2604.18310/) (2026)
- [Even More Guarantees for Variational Inference in the Presence of Symmetries](https://mlsanity.com/paper/2604.21407/) (2026)
- [Linear Response Estimators for Singular Statistical Models](https://mlsanity.com/paper/2605.07970/) (2026)

## Abstract (arXiv)

Mean-field Variational Bayes (MFVB) is an approximate Bayesian posterior inference technique that is increasingly popular due to its fast runtimes on large-scale datasets. However, even when MFVB provides accurate posterior means for certain parameters, it often mis-estimates variances and covariances. Furthermore, prior robustness measures have remained undeveloped for MFVB. By deriving a simple formula for the effect of infinitesimal model perturbations on MFVB posterior means, we provide both improved covariance estimates and local robustness measures for MFVB, thus greatly expanding the practical usefulness of MFVB posterior approximations. The estimates for MFVB posterior covariances rely on a result from the classical Bayesian robustness literature relating derivatives of posterior expectations to posterior covariances and include the Laplace approximation as a special case. Our key condition is that the MFVB approximation provides good estimates of a select subset of posterior means---an assumption that has been shown to hold in many practical settings. In our experiments, we demonstrate that our methods are simple, general, and fast, providing accurate posterior uncertainty estimates and robustness measures with runtimes that can be an order of magnitude faster than MCMC.