# Variational Inference in high-dimensional linear regression

arXiv: https://arxiv.org/abs/2104.12232 · HTML: https://mlsanity.com/paper/2104.12232/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 72 references that are in the map.
- First posted on arXiv 25 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Benign overfitting in ridge regression](https://mlsanity.com/paper/2009.14286/) (2020)
- [On the properties of variational approximations of Gibbs posteriors](https://mlsanity.com/paper/1506.04091/) (2015)
- [Concentration of tempered posteriors and of their variational approximations](https://mlsanity.com/paper/1706.09293/) (2017)
- [Frequentist Consistency of Variational Bayes](https://mlsanity.com/paper/1705.03439/) (2017)
- [Variational Bayes for high-dimensional linear regression with sparse priors](https://mlsanity.com/paper/1904.07150/) (2019)
- [Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression](https://mlsanity.com/paper/1804.02605/) (2018)
- [Bayesian linear regression with sparse priors](https://mlsanity.com/paper/1403.0735/) (2014)
- [Variational Bayes under Model Misspecification](https://mlsanity.com/paper/1905.10859/) (2019)
- [Theoretical and Computational Guarantees of Mean Field Variational Inference for Community Detection](https://mlsanity.com/paper/1710.11268/) (2017)
- [Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures](https://mlsanity.com/paper/1805.05054/) (2018)
- [Spike and slab variational Bayes for high dimensional logistic regression](https://mlsanity.com/paper/2010.11665/) (2020)
- [Scalable and Accurate Variational Bayes for High-Dimensional Binary Regression Models](https://mlsanity.com/paper/1911.06743/) (2019)
- [An Instability in Variational Inference for Topic Models](https://mlsanity.com/paper/1802.00568/) (2018)
- [Statistical Inference in Mean-Field Variational Bayes](https://mlsanity.com/paper/1911.01525/) (2019)
- [The Mean-Field Approximation: Information Inequalities, Algorithms, and Complexity](https://mlsanity.com/paper/1802.06126/) (2018)
- [Mean-field approximation, convex hierarchies, and the optimality of correlation rounding: a unified perspective](https://mlsanity.com/paper/1808.07226/) (2018)
- [Tail-adaptive Bayesian shrinkage](https://mlsanity.com/paper/2007.02192/) (2020)

## Built on by (most cited first, 10 of 10)

- [Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models](https://mlsanity.com/paper/2309.11420/) (2023)
- [Local convexity of the TAP free energy and AMP convergence for Z2-synchronization](https://mlsanity.com/paper/2106.11428/) (2021)
- [The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?](https://mlsanity.com/paper/2205.10009/) (2022)
- [Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection](https://mlsanity.com/paper/2207.11208/) (2022)
- [Performance of Bayesian linear regression in a model with mismatch](https://mlsanity.com/paper/2107.06936/) (2021)
- [A Mean Field Approach to Empirical Bayes Estimation in High-dimensional Linear Regression](https://mlsanity.com/paper/2309.16843/) (2023)
- [The TAP free energy for high-dimensional linear regression](https://mlsanity.com/paper/2203.07539/) (2022)
- [A Particle Algorithm for Mean-Field Variational Inference](https://mlsanity.com/paper/2412.20385/) (2024)
- [Stability of Mean-Field Variational Inference](https://mlsanity.com/paper/2506.07856/) (2025)
- [Unrolled denoising networks provably learn optimal Bayesian inference](https://mlsanity.com/paper/2409.12947/) (2024)

## Abstract (arXiv)

We study high-dimensional Bayesian linear regression with product priors. Using the nascent theory of non-linear large deviations (Chatterjee and Dembo,2016), we derive sufficient conditions for the leading-order correctness of the naive mean-field approximation to the log-normalizing constant of the posterior distribution. Subsequently, assuming a true linear model for the observed data, we derive a limiting infinite dimensional variational formula for the log normalizing constant of the posterior. Furthermore, we establish that under an additional "separation" condition, the variational problem has a unique optimizer, and this optimizer governs the probabilistic properties of the posterior distribution. We provide intuitive sufficient conditions for the validity of this "separation" condition. Finally, we illustrate our results on concrete examples with specific design matrices.