# For high-dimensional hierarchical models, consider exchangeability of effects across covariates instead of across datasets

arXiv: https://arxiv.org/abs/2107.06428 · HTML: https://mlsanity.com/paper/2107.06428/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 73 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 6 of 6)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Recasting Gradient-Based Meta-Learning as Hierarchical Bayes](https://mlsanity.com/paper/1801.08930/) (2018)
- [Lecture notes on ridge regression](https://mlsanity.com/paper/1509.09169/) (2015)
- [Simultaneous Variable and Covariance Selection with the Multivariate Spike-and-Slab Lasso](https://mlsanity.com/paper/1708.08911/) (2017)
- [Weighted Meta-Learning](https://mlsanity.com/paper/2003.09465/) (2020)

## Built on by (most cited first, 2 of 2)

- [Training Flexible Models of Genetic Variant Effects from Functional Annotations using Accelerated Linear Algebra](https://mlsanity.com/paper/2506.19598/) (2025)
- [Inferring Effects of Major Events through Discontinuity Forecasting of Population Anxiety](https://mlsanity.com/paper/2508.21722/) (2025)

## Abstract (arXiv)

Hierarchical Bayesian methods enable information sharing across multiple related regression problems. While standard practice is to model regression parameters (effects) as (1) exchangeable across datasets and (2) correlated to differing degrees across covariates, we show that this approach exhibits poor statistical performance when the number of covariates exceeds the number of datasets. For instance, in statistical genetics, we might regress dozens of traits (defining datasets) for thousands of individuals (responses) on up to millions of genetic variants (covariates). When an analyst has more covariates than datasets, we argue that it is often more natural to instead model effects as (1) exchangeable across covariates and (2) correlated to differing degrees across datasets. To this end, we propose a hierarchical model expressing our alternative perspective. We devise an empirical Bayes estimator for learning the degree of correlation between datasets. We develop theory that demonstrates that our method outperforms the classic approach when the number of covariates dominates the number of datasets, and corroborate this result empirically on several high-dimensional multiple regression and classification problems.