# A flexible empirical Bayes approach to multiple linear regression and connections with penalized regression

arXiv: https://arxiv.org/abs/2208.10910 · HTML: https://mlsanity.com/paper/2208.10910/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 90 references that are in the map.
- First posted on arXiv 23 Aug 2022.

## Builds on (most cited first, 10 of 10)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Fast Best Subset Selection: Coordinate Descent and Local Combinatorial Optimization Algorithms](https://mlsanity.com/paper/1803.01454/) (2018)
- [Variational Bayes for high-dimensional linear regression with sparse priors](https://mlsanity.com/paper/1904.07150/) (2019)
- [De-biasing the Lasso: Optimal Sample Size for Gaussian Designs](https://mlsanity.com/paper/1508.02757/) (2015)
- [False Discoveries Occur Early on the Lasso Path](https://mlsanity.com/paper/1511.01957/) (2015)
- [Empirical Bayes Matrix Factorization](https://mlsanity.com/paper/1802.06931/) (2018)
- [Variational Bayesian inference for linear and logistic regression](https://mlsanity.com/paper/1310.5438/) (2013)
- [A fast algorithm for maximum likelihood estimation of mixture proportions using sequential quadratic programming](https://mlsanity.com/paper/1806.01412/) (2018)
- [Empirical Bayes Shrinkage and False Discovery Rate Estimation, Allowing For Unwanted Variation](https://mlsanity.com/paper/1709.10066/) (2017)
- [Solving the Empirical Bayes Normal Means Problem with Correlated Noise](https://mlsanity.com/paper/1812.07488/) (2018)

## Built on by (most cited first, 8 of 8)

- [Solving Empirical Bayes via Transformers](https://mlsanity.com/paper/2502.09844/) (2025)
- [A Mean Field Approach to Empirical Bayes Estimation in High-dimensional Linear Regression](https://mlsanity.com/paper/2309.16843/) (2023)
- [Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm](https://mlsanity.com/paper/2209.08139/) (2022)
- [Distribution-dependent Generalization Bounds for Tuning Linear Regression Across Tasks](https://mlsanity.com/paper/2507.05084/) (2025)
- [Empirical Bayes Covariance Decomposition, and a Solution to the Multiple Tuning Problem in Sparse PCA](https://mlsanity.com/paper/2312.03274/) (2023)
- [Nash: Neural Adaptive Shrinkage for Structured High-Dimensional Regression](https://mlsanity.com/paper/2505.11143/) (2025)
- [A Flexible Empirical Bayes Approach to Generalized Linear Models, with Applications to Sparse Logistic Regression](https://mlsanity.com/paper/2601.21217/) (2026)
- [Empirical Bayes for correlated Gaussian sequence model](https://mlsanity.com/paper/2607.03596/) (2026)

## Abstract (arXiv)

We introduce a new empirical Bayes approach for large-scale multiple linear regression. Our approach combines two key ideas: (i) the use of flexible "adaptive shrinkage" priors, which approximate the nonparametric family of scale mixture of normal distributions by a finite mixture of normal distributions; and (ii) the use of variational approximations to efficiently estimate prior hyperparameters and compute approximate posteriors. Combining these two ideas results in fast and flexible methods, with computational speed comparable to fast penalized regression methods such as the Lasso, and with competitive prediction accuracy across a wide range of scenarios. Further, we provide new results that establish conceptual connections between our empirical Bayes methods and penalized methods. Specifically, we show that the posterior mean from our method solves a penalized regression problem, with the form of the penalty function being learned from the data by directly solving an optimization problem (rather than being tuned by cross-validation). Our methods are implemented in an R package, mr.ash.alpha, available from https://github.com/stephenslab/mr.ash.alpha.