# A Mean Field Approach to Empirical Bayes Estimation in High-dimensional Linear Regression

arXiv: https://arxiv.org/abs/2309.16843 · HTML: https://mlsanity.com/paper/2309.16843/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 137 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [On the properties of variational approximations of Gibbs posteriors](https://mlsanity.com/paper/1506.04091/) (2015)
- [Variational inference via Wasserstein gradient flows](https://mlsanity.com/paper/2205.15902/) (2022)
- [Frequentist Consistency of Variational Bayes](https://mlsanity.com/paper/1705.03439/) (2017)
- [Convergence Rates of Variational Posterior Distributions](https://mlsanity.com/paper/1712.02519/) (2017)
- [Variational Bayes for high-dimensional linear regression with sparse priors](https://mlsanity.com/paper/1904.07150/) (2019)
- [Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Space](https://mlsanity.com/paper/2304.05398/) (2023)
- [Variational Bayes under Model Misspecification](https://mlsanity.com/paper/1905.10859/) (2019)
- [Theoretical and Computational Guarantees of Mean Field Variational Inference for Community Detection](https://mlsanity.com/paper/1710.11268/) (2017)
- [On the nonparametric maximum likelihood estimator for Gaussian location mixture densities with application to Gaussian denoising](https://mlsanity.com/paper/1712.02009/) (2017)
- [Mean-field Variational Inference via Wasserstein Gradient Flow](https://mlsanity.com/paper/2207.08074/) (2022)
- [Self-regularizing Property of Nonparametric Maximum Likelihood Estimator in Mixture Models](https://mlsanity.com/paper/2008.08244/) (2020)
- [Two Modeling Strategies for Empirical Bayes Estimation](https://mlsanity.com/paper/1409.2677/) (2014)
- [Spike and slab variational Bayes for high dimensional logistic regression](https://mlsanity.com/paper/2010.11665/) (2020)
- [Local convexity of the TAP free energy and AMP convergence for Z2-synchronization](https://mlsanity.com/paper/2106.11428/) (2021)
- [An Instability in Variational Inference for Topic Models](https://mlsanity.com/paper/1802.00568/) (2018)
- [Empirical Bayes PCA in high dimensions](https://mlsanity.com/paper/2012.11676/) (2020)
- [Statistical Inference in Mean-Field Variational Bayes](https://mlsanity.com/paper/1911.01525/) (2019)
- [Variational Inference in high-dimensional linear regression](https://mlsanity.com/paper/2104.12232/) (2021)
- [Sharp regret bounds for empirical Bayes and compound decision problems](https://mlsanity.com/paper/2109.03943/) (2021)
- [Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energy](https://mlsanity.com/paper/2208.09550/) (2022)
- [A flexible empirical Bayes approach to multiple linear regression and connections with penalized regression](https://mlsanity.com/paper/2208.10910/) (2022)
- [Optimal empirical Bayes estimation for the Poisson model via minimum-distance methods](https://mlsanity.com/paper/2209.01328/) (2022)
- [Group-Linear Empirical Bayes Estimates for a Heteroscedastic Normal Mean](https://mlsanity.com/paper/1503.08503/) (2015)
- [A fast algorithm for maximum likelihood estimation of mixture proportions using sequential quadratic programming](https://mlsanity.com/paper/1806.01412/) (2018)
- [Two-component Mixture Model in the Presence of Covariates](https://mlsanity.com/paper/1810.07897/) (2018)
- [Convergence Rates of Empirical Bayes Posterior Distributions: A Variational Perspective](https://mlsanity.com/paper/2009.03969/) (2020)
- [The TAP free energy for high-dimensional linear regression](https://mlsanity.com/paper/2203.07539/) (2022)
- [Nonparametric Empirical Bayes Estimation on Heterogeneous Data](https://mlsanity.com/paper/2002.12586/) (2020)

## Built on by (most cited first, 4 of 4)

- [Solving Empirical Bayes via Transformers](https://mlsanity.com/paper/2502.09844/) (2025)
- [Nonparametric MLE for Gaussian Location Mixtures: Certified Computation and Generic Behavior](https://mlsanity.com/paper/2503.20193/) (2025)
- [Stability of Mean-Field Variational Inference](https://mlsanity.com/paper/2506.07856/) (2025)
- [Empirical Bayes for correlated Gaussian sequence model](https://mlsanity.com/paper/2607.03596/) (2026)

## Abstract (arXiv)

We study empirical Bayes estimation in high-dimensional linear regression. To facilitate computationally efficient estimation of the underlying prior, we adopt a variational empirical Bayes approach, introduced originally in Carbonetto and Stephens (2012) and Kim et al. (2022). We establish asymptotic consistency of the nonparametric maximum likelihood estimator (NPMLE) and its (computable) naive mean field variational surrogate under mild assumptions on the design and the prior. Assuming, in addition, that the naive mean field approximation has a dominant optimizer, we develop a computationally efficient approximation to the oracle posterior distribution, and establish its accuracy under the 1-Wasserstein metric. This enables computationally feasible Bayesian inference; e.g., construction of posterior credible intervals with an average coverage guarantee, Bayes optimal estimation for the regression coefficients, estimation of the proportion of non-nulls, etc. Our analysis covers both deterministic and random designs, and accommodates correlations among the features. To the best of our knowledge, this provides the first rigorous nonparametric empirical Bayes method in a high-dimensional regression setting without sparsity.