# Variational Bayes for high-dimensional linear regression with sparse priors

arXiv: https://arxiv.org/abs/1904.07150 · HTML: https://mlsanity.com/paper/1904.07150/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2019.

## Built on by (most cited first, 30 of 44)

- [On the Convergence of Coordinate Ascent Variational Inference](https://mlsanity.com/paper/2306.01122/) (2023)
- [Adaptive variational Bayes: Optimality, computation and applications](https://mlsanity.com/paper/2109.03204/) (2021)
- [Thompson Sampling for High-Dimensional Sparse Linear Contextual Bandits](https://mlsanity.com/paper/2211.05964/) (2022)
- [Local convexity of the TAP free energy and AMP convergence for Z2-synchronization](https://mlsanity.com/paper/2106.11428/) (2021)
- [Variational Inference in high-dimensional linear regression](https://mlsanity.com/paper/2104.12232/) (2021)
- [The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?](https://mlsanity.com/paper/2205.10009/) (2022)
- [A flexible empirical Bayes approach to multiple linear regression and connections with penalized regression](https://mlsanity.com/paper/2208.10910/) (2022)
- [On the inability of Gaussian process regression to optimally learn compositional functions](https://mlsanity.com/paper/2205.07764/) (2022)
- [Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection](https://mlsanity.com/paper/2207.11208/) (2022)
- [Performance of Bayesian linear regression in a model with mismatch](https://mlsanity.com/paper/2107.06936/) (2021)
- [Additive Multi-Index Gaussian process modeling, with application to multi-physics surrogate modeling of the quark-gluon plasma](https://mlsanity.com/paper/2306.07299/) (2023)
- [A sparse PAC-Bayesian approach for high-dimensional quantile prediction](https://mlsanity.com/paper/2409.01687/) (2024)
- [Scalable Spike-and-Slab](https://mlsanity.com/paper/2204.01668/) (2022)
- [Machine Learning and the Future of Bayesian Computation](https://mlsanity.com/paper/2304.11251/) (2023)
- [Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networks](https://mlsanity.com/paper/2308.09104/) (2023)
- [A Mean Field Approach to Empirical Bayes Estimation in High-dimensional Linear Regression](https://mlsanity.com/paper/2309.16843/) (2023)
- [Spike and slab Bayesian sparse principal component analysis](https://mlsanity.com/paper/2102.00305/) (2021)
- [A fast asynchronous MCMC sampler for sparse Bayesian inference](https://mlsanity.com/paper/2108.06446/) (2021)
- [Variational Bayes for high-dimensional proportional hazards models with applications within gene expression](https://mlsanity.com/paper/2112.10270/) (2021)
- [Sparse online variational Bayesian regression](https://mlsanity.com/paper/2102.12261/) (2021)
- [Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm](https://mlsanity.com/paper/2209.08139/) (2022)
- [Mixed Variational Flows for Discrete Variables](https://mlsanity.com/paper/2308.15613/) (2023)
- [Variational Gaussian Processes For Linear Inverse Problems](https://mlsanity.com/paper/2311.00663/) (2023)
- [Covariate-Elaborated Robust Partial Information Transfer with Conditional Spike-and-Slab Prior](https://mlsanity.com/paper/2404.03764/) (2024)
- [Contraction rates for conjugate gradient and Lanczos approximate posteriors in Gaussian process regression](https://mlsanity.com/paper/2406.12678/) (2024)
- [Adaptive sparse variational approximations for Gaussian process regression](https://mlsanity.com/paper/2504.03321/) (2025)
- [Pointwise uncertainty quantification for sparse variational Gaussian process regression with a Brownian motion prior](https://mlsanity.com/paper/2310.00097/) (2023)
- [Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance](https://mlsanity.com/paper/2405.19016/) (2024)
- [Robust and Scalable Variational Bayes](https://mlsanity.com/paper/2504.12528/) (2025)
- [Stability of Sequential and Parallel Coordinate Ascent Variational Inference](https://mlsanity.com/paper/2603.20929/) (2026)

## Abstract (arXiv)

We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse truth at the optimal rate and gives optimal prediction of the response vector. The empirical performance of our algorithm is studied, showing that it works comparably well as other state-of-the-art Bayesian variable selection methods. We also numerically demonstrate that the widely used coordinate-ascent variational inference (CAVI) algorithm can be highly sensitive to the parameter updating order, leading to potentially poor performance. To mitigate this, we propose a novel prioritized updating scheme that uses a data-driven updating order and performs better in simulations. The variational algorithm is implemented in the R package 'sparsevb'.