# Scalable and Accurate Variational Bayes for High-Dimensional Binary Regression Models

arXiv: https://arxiv.org/abs/1911.06743 · HTML: https://mlsanity.com/paper/1911.06743/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2019.

## Built on by (most cited first, 12 of 12)

- [Variational Inference in high-dimensional linear regression](https://mlsanity.com/paper/2104.12232/) (2021)
- [Partially factorized variational inference for high-dimensional mixed models](https://mlsanity.com/paper/2312.13148/) (2023)
- [Easy Variational Inference for Categorical Models via an Independent Binary Approximation](https://mlsanity.com/paper/2206.00093/) (2022)
- [Mixing times of data-augmentation Gibbs samplers for high-dimensional probit regression](https://mlsanity.com/paper/2505.14343/) (2025)
- [Bayesian Inference for the Multinomial Probit Model under Gaussian Prior Distribution](https://mlsanity.com/paper/2206.00720/) (2022)
- [Robust leave-one-out cross-validation for high-dimensional Bayesian models](https://mlsanity.com/paper/2209.09190/) (2022)
- [Efficient expectation propagation for posterior approximation in high-dimensional probit models](https://mlsanity.com/paper/2309.01619/) (2023)
- [Efficient computation of predictive probabilities in probit models via expectation propagation](https://mlsanity.com/paper/2309.01630/) (2023)
- [Expectation propagation for the smoothing distribution in dynamic probit](https://mlsanity.com/paper/2309.01641/) (2023)
- [Approximate Bayesian inference for cumulative probit regression models](https://mlsanity.com/paper/2511.06967/) (2025)
- [BaGGLS: A Bayesian Shrinkage Framework for Interpretable Modeling of Interactions in High-Dimensional Biological Data](https://mlsanity.com/paper/2511.15330/) (2025)
- [Mean-field Variational Bayes for Sparse Probit Regression](https://mlsanity.com/paper/2601.21765/) (2026)

## Abstract (arXiv)

Modern methods for Bayesian regression beyond the Gaussian response setting are often computationally impractical or inaccurate in high dimensions. In fact, as discussed in recent literature, bypassing such a trade-off is still an open problem even in routine binary regression models, and there is limited theory on the quality of variational approximations in high-dimensional settings. To address this gap, we study the approximation accuracy of routinely-used mean-field variational Bayes solutions in high-dimensional probit regression with Gaussian priors, obtaining novel and practically relevant results on the pathological behavior of such strategies in uncertainty quantification, point estimation and prediction. Motivated by these results, we further develop a new partially-factorized variational approximation for the posterior of the probit coefficients which leverages a representation with global and local variables but, unlike for classical mean-field assumptions, it avoids a fully factorized approximation, and instead assumes a factorization only for the local variables. We prove that the resulting approximation belongs to a tractable class of unified skew-normal densities that crucially incorporates skewness and, unlike for state-of-the-art mean-field solutions, converges to the exact posterior density as p goes to infinity. To solve the variational optimization problem, we derive a tractable CAVI algorithm that easily scales to p in the tens of thousands, and provably requires a number of iterations converging to 1 as p goes to infinity. Such findings are also illustrated in extensive empirical studies where our novel solution is shown to improve the approximation accuracy of mean-field variational Bayes for any n and p, with the magnitude of these gains being remarkable in those high-dimensional p>n settings where state-of-the-art methods are computationally impractical.