# Sparse online variational Bayesian regression

arXiv: https://arxiv.org/abs/2102.12261 · HTML: https://mlsanity.com/paper/2102.12261/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 89 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 12 of 12)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [SGD: General Analysis and Improved Rates](https://mlsanity.com/paper/1901.09401/) (2019)
- [Proximal Markov chain Monte Carlo algorithms](https://mlsanity.com/paper/1306.0187/) (2013)
- [Variational Bayes for high-dimensional linear regression with sparse priors](https://mlsanity.com/paper/1904.07150/) (2019)
- [Bayesian linear regression with sparse priors](https://mlsanity.com/paper/1403.0735/) (2014)
- [Automatic Variational Inference in Stan](https://mlsanity.com/paper/1506.03431/) (2015)
- [Variational Relevance Vector Machines](https://mlsanity.com/paper/1301.3838/) (2013)
- [Maximum-a-posteriori estimation with Bayesian confidence regions](https://mlsanity.com/paper/1602.08590/) (2016)
- [Spike-and-Slab Meets LASSO: A Review of the Spike-and-Slab LASSO](https://mlsanity.com/paper/2010.06451/) (2020)
- [Cluster, Classify, Regress: A General Method For Learning Discountinous Functions](https://mlsanity.com/paper/1905.06220/) (2019)
- [Variational Bayesian inference for linear and logistic regression](https://mlsanity.com/paper/1310.5438/) (2013)
- [Variance prior forms for high-dimensional Bayesian variable selection](https://mlsanity.com/paper/1801.03019/) (2018)

## Built on by (most cited first, 2 of 2)

- [Mixtures of Gaussian Process Experts with SMC2](https://mlsanity.com/paper/2208.12830/) (2022)
- [Variational Bayesian Inference for Tensor Robust Principal Component Analysis](https://mlsanity.com/paper/2412.18717/) (2024)

## Abstract (arXiv)

This work considers variational Bayesian inference as an inexpensive and scalable alternative to a fully Bayesian approach in the context of sparsity-promoting priors. In particular, the priors considered arise from scale mixtures of Normal distributions with a generalized inverse Gaussian mixing distribution. This includes the variational Bayesian LASSO as an inexpensive and scalable alternative to the Bayesian LASSO introduced in [65]. It also includes a family of priors which more strongly promote sparsity. For linear models the method requires only the iterative solution of deterministic least squares problems. Furthermore, for p unknown covariates the method can be implemented exactly online with a cost of O(p3) in computation and O(p2) in memory per iteration -- in other words, the cost per iteration is independent of n, and in principle infinite data can be considered. For large p an approximation is able to achieve promising results for a cost of O(p) per iteration, in both computation and memory. Strategies for hyper-parameter tuning are also considered. The method is implemented for real and simulated data. It is shown that the performance in terms of variable selection and uncertainty quantification of the variational Bayesian LASSO can be comparable to the Bayesian LASSO for problems which are tractable with that method, and for a fraction of the cost. The present method comfortably handles n = 65536, p = 131073 on a laptop in less than 30 minutes, and n = 105, p = 2.1 × 106 overnight.