# Gaussian variational approximation with a factor covariance structure

arXiv: https://arxiv.org/abs/1701.03208 · HTML: https://mlsanity.com/paper/1701.03208/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jan 2017.

## Built on by (most cited first, 22 of 22)

- [BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models](https://mlsanity.com/paper/2406.11675/) (2024)
- [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021)
- [Low-rank extended Kalman filtering for online learning of neural networks from streaming data](https://mlsanity.com/paper/2305.19535/) (2023)
- [A practical tutorial on Variational Bayes](https://mlsanity.com/paper/2103.01327/) (2021)
- [Flat Seeking Bayesian Neural Networks](https://mlsanity.com/paper/2302.02713/) (2023)
- [Batch, match, and patch: low-rank approximations for score-based variational inference](https://mlsanity.com/paper/2410.22292/) (2024)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [Federated Variational Inference Methods for Structured Latent Variable Models](https://mlsanity.com/paper/2302.03314/) (2023)
- [Provably Scalable Black-Box Variational Inference with Structured Variational Families](https://mlsanity.com/paper/2401.10989/) (2024)
- [Marginally calibrated response distributions for end-to-end learning in autonomous driving](https://mlsanity.com/paper/2110.01050/) (2021)
- [Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning](https://mlsanity.com/paper/2407.02721/) (2024)
- [Low Rank Field-Weighted Factorization Machines for Low Latency Item Recommendation](https://mlsanity.com/paper/2408.00801/) (2024)
- [Large Skew-t Copula Models and Asymmetric Dependence in Intraday Equity Returns](https://mlsanity.com/paper/2308.05564/) (2023)
- [Distributed Variational Inference for Online Supervised Learning](https://mlsanity.com/paper/2309.02606/) (2023)
- [Vector Copula Variational Inference and Dependent Block Posterior Approximations](https://mlsanity.com/paper/2503.01072/) (2025)
- [Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference](https://mlsanity.com/paper/2505.21721/) (2025)
- [Boltzmann Tuning of Generative Models](https://mlsanity.com/paper/2104.05252/) (2021)
- [Bayesian Neural Network Versus Ex-Post Calibration For Prediction Uncertainty](https://mlsanity.com/paper/2209.14594/) (2022)
- [Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Models](https://mlsanity.com/paper/2302.13536/) (2023)
- [Variational Approximated Restricted Maximum Likelihood Estimation for Spatial Data](https://mlsanity.com/paper/2604.07635/) (2026)
- [Variational predictive resampling](https://mlsanity.com/paper/2605.11168/) (2026)
- [Large-scale empirical tuning and comparison of default optimizers for variational inference](https://mlsanity.com/paper/2606.07841/) (2026)

## Abstract (arXiv)

Variational approximation methods have proven to be useful for scaling Bayesian computations to large data sets and highly parametrized models. Applying variational methods involves solving an optimization problem, and recent research in this area has focused on stochastic gradient ascent methods as a general approach to implementation. Here variational approximation is considered for a posterior distribution in high dimensions using a Gaussian approximating family. Gaussian variational approximation with an unrestricted covariance matrix can be computationally burdensome in many problems because the number of elements in the covariance matrix increases quadratically with the dimension of the model parameter. To circumvent this problem, low-dimensional factor covariance structures are considered. General stochastic gradient approaches to efficiently perform the optimization are described, with gradient estimates obtained using the so-called "reparametrization trick". The end result is a flexible and efficient approach to high-dimensional Gaussian variational approximation, which we illustrate using eight real datasets.