mlsanityData to 7 Oct 2026

Variational Inference in high-dimensional linear regression

arXiv 2104.12232 ↗First posted 25 Apr 2021math.ST math.PR stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

We study high-dimensional Bayesian linear regression with product priors. Using the nascent theory of non-linear large deviations (Chatterjee and Dembo,2016), we derive sufficient conditions for the leading-order correctness of the naive mean-field approximation to the log-normalizing constant of the posterior distribution. Subsequently, assuming a true linear model for the observed data, we derive a limiting infinite dimensional variational formula for the log normalizing constant of the posterior. Furthermore, we establish that under an additional "separation" condition, the variational problem has a unique optimizer, and this optimizer governs the probabilistic properties of the posterior distribution. We provide intuitive sufficient conditions for the validity of this "separation" condition. Finally, we illustrate our results on concrete examples with specific design matrices.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 20 of its 72 references that are in the map. First posted on arXiv 25 Apr 2021.

Builds on

20 of 72 references.

References · 20

  1. 2016Variational Inference: A Review for Statisticianscited by 1,466
  2. 2019Benign Overfitting in Linear Regressioncited by 544
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2020Benign overfitting in ridge regressioncited by 152
  5. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  6. 2017Concentration of tempered posteriors and of their variational approximationscited by 66
  7. 2017Frequentist Consistency of Variational Bayescited by 60
  8. 2019Variational Bayes for high-dimensional linear regression with sparse priorscited by 44
  9. 2018Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regressioncited by 42
  10. 2014Bayesian linear regression with sparse priorscited by 40
  11. 2019Variational Bayes under Model Misspecificationcited by 22
  12. 2017Theoretical and Computational Guarantees of Mean Field Variational Inference for Community Detectioncited by 19
8 more
  1. 2018Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixturescited by 17
  2. 2020Spike and slab variational Bayes for high dimensional logistic regressioncited by 16
  3. 2019Scalable and Accurate Variational Bayes for High-Dimensional Binary Regression Modelscited by 12
  4. 2018An Instability in Variational Inference for Topic Modelscited by 11
  5. 2019Statistical Inference in Mean-Field Variational Bayescited by 10
  6. 2018The Mean-Field Approximation: Information Inequalities, Algorithms, and Complexitycited by 7
  7. 2018Mean-field approximation, convex hierarchies, and the optimality of correlation rounding: a unified perspectivecited by 6
  8. 2020Tail-adaptive Bayesian shrinkagecited by 1

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 2; 2024: 2; 2025: 1.

10 papers.

  1. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Modelscited by 39
  2. 2021Local convexity of the TAP free energy and AMP convergence for Z2-synchronizationcited by 15
  3. 2022The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?cited by 9
  4. 2022Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selectioncited by 7
  5. 2021Performance of Bayesian linear regression in a model with mismatchcited by 6
  6. 2023A Mean Field Approach to Empirical Bayes Estimation in High-dimensional Linear Regressioncited by 4
  7. 2022The TAP free energy for high-dimensional linear regressioncited by 3
  8. 2024A Particle Algorithm for Mean-Field Variational Inferencecited by 3
  9. 2025Stability of Mean-Field Variational Inferencecited by 3
  10. 2024Unrolled denoising networks provably learn optimal Bayesian inferencecited by 1