# Learning from a lot: Empirical Bayes in high-dimensional prediction settings

arXiv: https://arxiv.org/abs/1709.04192 · HTML: https://mlsanity.com/paper/1709.04192/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2017.

## Built on by (most cited first, 1 of 1)

- [Guiding adaptive shrinkage by co-data to improve regression-based prediction and feature selection](https://mlsanity.com/paper/2405.04917/) (2024)

## Abstract (arXiv)

Empirical Bayes is a versatile approach to `learn from a lot' in two ways: first, from a large number of variables and second, from a potentially large amount of prior information, e.g. stored in public repositories. We review applications of a variety of empirical Bayes methods to several well-known model-based prediction methods including penalized regression, linear discriminant analysis, and Bayesian models with sparse or dense priors. We discuss `formal' empirical Bayes methods which maximize the marginal likelihood, but also more informal approaches based on other data summaries. We contrast empirical Bayes to cross-validation and full Bayes, and discuss hybrid approaches. To study the relation between the quality of an empirical Bayes estimator and p, the number of variables, we consider a simple empirical Bayes estimator in a linear model setting. We argue that empirical Bayes is particularly useful when the prior contains multiple parameters which model a priori information on variables, termed `co-data'. In particular, we present two novel examples that allow for co-data. First, a Bayesian spike-and-slab setting that facilitates inclusion of multiple co-data sources and types; second, a hybrid empirical Bayes-full Bayes ridge regression approach for estimation of the posterior predictive interval.