# Bayesian subset selection and variable importance for interpretable prediction and classification

arXiv: https://arxiv.org/abs/2104.10150 · HTML: https://mlsanity.com/paper/2104.10150/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 51 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 14 of 14)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Best Subset Selection via a Modern Optimization Lens](https://mlsanity.com/paper/1507.03133/) (2015)
- [Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models](https://mlsanity.com/paper/1901.03209/) (2019)
- [Cross-Validation with Confidence](https://mlsanity.com/paper/1703.07904/) (2017)
- [Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach](https://mlsanity.com/paper/1809.09505/) (2018)
- [Forward Stability and Model Path Selection](https://mlsanity.com/paper/2103.03462/) (2021)
- [Decoupling shrinkage and selection in Bayesian linear models: a posterior summary perspective](https://mlsanity.com/paper/1408.0464/) (2014)
- [In Defense of the Indefensible: A Very Naive Approach to High-Dimensional Inference](https://mlsanity.com/paper/1705.05543/) (2017)
- [Projective Inference in High-dimensional Problems: Prediction and Feature Selection](https://mlsanity.com/paper/1810.02406/) (2018)
- [Model interpretation through lower-dimensional posterior summarization](https://mlsanity.com/paper/1905.07103/) (2019)
- [Variable Selection in Seemingly Unrelated Regressions with Random Predictors](https://mlsanity.com/paper/1605.08963/) (2016)
- [Efficient sampling for Gaussian linear regression with arbitrary priors](https://mlsanity.com/paper/1806.05738/) (2018)
- [Bayesian Function-on-Scalars Regression for High Dimensional Data](https://mlsanity.com/paper/1808.06689/) (2018)

## Built on by (most cited first, 6 of 6)

- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Subset selection for linear mixed models](https://mlsanity.com/paper/2107.12890/) (2021)
- [Soft Random Sampling: A Theoretical and Empirical Analysis](https://mlsanity.com/paper/2311.12727/) (2023)
- [Scalable Subset Selection in Linear Mixed Models](https://mlsanity.com/paper/2506.20425/) (2025)
- [Bayesian adaptive and interpretable functional regression for exposure profiles](https://mlsanity.com/paper/2203.00784/) (2022)
- [A Mathematical Optimization Approach for Expert-Informed Bayesian Best Subset Selection](https://mlsanity.com/paper/2606.29516/) (2026)

## Abstract (arXiv)

Subset selection is a valuable tool for interpretable learning, scientific discovery, and data compression. However, classical subset selection is often avoided due to selection instability, lack of regularization, and difficulties with post-selection inference. We address these challenges from a Bayesian perspective. Given any Bayesian predictive model M, we extract a family of near-optimal subsets of variables for linear prediction or classification. This strategy deemphasizes the role of a single "best" subset and instead advances the broader perspective that often many subsets are highly competitive. The acceptable family of subsets offers a new pathway for model interpretation and is neatly summarized by key members such as the smallest acceptable subset, along with new (co-) variable importance metrics based on whether variables (co-) appear in all, some, or no acceptable subsets. More broadly, we apply Bayesian decision analysis to derive the optimal linear coefficients for any subset of variables. These coefficients inherit both regularization and predictive uncertainty quantification via M. For both simulated and real data, the proposed approach exhibits better prediction, interval estimation, and variable selection than competing Bayesian and frequentist selection methods. These tools are applied to a large education dataset with highly correlated covariates. Our analysis provides unique insights into the combination of environmental, socioeconomic, and demographic factors that predict educational outcomes, and identifies over 200 distinct subsets of variables that offer near-optimal out-of-sample predictive accuracy.