# Partially factorized variational inference for high-dimensional mixed models

arXiv: https://arxiv.org/abs/2312.13148 · HTML: https://mlsanity.com/paper/2312.13148/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 15 of 15)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Pathfinder: Parallel quasi-Newton variational inference](https://mlsanity.com/paper/2108.03782/) (2021)
- [On the Convergence of Coordinate Ascent Variational Inference](https://mlsanity.com/paper/2306.01122/) (2023)
- [Gaussian variational approximation with sparse precision matrices](https://mlsanity.com/paper/1605.05622/) (2016)
- [Scalable and Accurate Variational Bayes for High-Dimensional Binary Regression Models](https://mlsanity.com/paper/1911.06743/) (2019)
- [Scalable Bayesian computation for crossed and nested hierarchical models](https://mlsanity.com/paper/2103.10875/) (2021)
- [Scalable inference for crossed random effects models](https://mlsanity.com/paper/1803.09460/) (2018)
- [Fast and Accurate Variational Inference for Models with Many Latent Variables](https://mlsanity.com/paper/2005.07430/) (2020)
- [Fast Moment-Based Estimation for Hierarchical Models](https://mlsanity.com/paper/1504.04941/) (2015)
- [Efficient moment calculations for variance components in large unbalanced crossed random effects models](https://mlsanity.com/paper/1602.00346/) (2016)
- [Streamlined Variational Inference for Linear Mixed Models with Crossed Random Effects](https://mlsanity.com/paper/1910.01799/) (2019)
- [Fast and Accurate Estimation of Non-Nested Binomial Hierarchical Models Using Variational Inference](https://mlsanity.com/paper/2007.12300/) (2020)
- [Variational Bayes in State Space Models: Inferential and Predictive Accuracy](https://mlsanity.com/paper/2106.12262/) (2021)
- [Streamlined Variational Inference for Higher Level Group-Specific Curve Models](https://mlsanity.com/paper/1903.04043/) (2019)

## Built on by (most cited first, 10 of 10)

- [Convergence rate of random scan Coordinate Ascent Variational Inference under log-concavity](https://mlsanity.com/paper/2406.07292/) (2024)
- [Linear-cost unbiased posterior estimates for crossed effects and matrix factorization models via couplings](https://mlsanity.com/paper/2410.08939/) (2024)
- [Conjugate gradient methods for high-dimensional GLMMs](https://mlsanity.com/paper/2411.04729/) (2024)
- [Mixing times of data-augmentation Gibbs samplers for high-dimensional probit regression](https://mlsanity.com/paper/2505.14343/) (2025)
- [Vector Copula Variational Inference and Dependent Block Posterior Approximations](https://mlsanity.com/paper/2503.01072/) (2025)
- [Exponential Convergence of CAVI for Bayesian PCA](https://mlsanity.com/paper/2505.16145/) (2025)
- [Wasserstein Contraction of Coordinate Ascent Variational Inference](https://mlsanity.com/paper/2605.30253/) (2026)
- [Scalable Krylov Subspace Methods for Generalized Mixed-Effects Models with Crossed Random Effects](https://mlsanity.com/paper/2505.09552/) (2025)
- [Approximate Bayesian inference for cumulative probit regression models](https://mlsanity.com/paper/2511.06967/) (2025)
- [Variational Approximated Restricted Maximum Likelihood Estimation for Spatial Data](https://mlsanity.com/paper/2604.07635/) (2026)

## Abstract (arXiv)

While generalized linear mixed models are a fundamental tool in applied statistics, many specifications, such as those involving categorical factors with many levels or interaction terms, can be computationally challenging to estimate due to the need to compute or approximate high-dimensional integrals. Variational inference is a popular way to perform such computations, especially in the Bayesian context. However, naive use of such methods can provide unreliable uncertainty quantification. We show that this is indeed the case for mixed models, proving that standard mean-field variational inference dramatically underestimates posterior uncertainty in high-dimensions. We then show how appropriately relaxing the mean-field assumption leads to methods whose uncertainty quantification does not deteriorate in high-dimensions, and whose total computational cost scales linearly with the number of parameters and observations. Our theoretical and numerical results focus on mixed models with Gaussian or binomial likelihoods, and rely on connections to random graph theory to obtain sharp high-dimensional asymptotic analysis. We also provide generic results, which are of independent interest, relating the accuracy of variational inference to the convergence rate of the corresponding coordinate ascent algorithm that is used to find it. Our proposed methodology is implemented in the R package, see https://github.com/mgoplerud/vglmer . Numerical results with simulated and real data examples illustrate the favourable computation cost versus accuracy trade-off of our approach compared to various alternatives.