# Projective Inference in High-dimensional Problems: Prediction and Feature Selection

arXiv: https://arxiv.org/abs/1810.02406 · HTML: https://mlsanity.com/paper/1810.02406/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2018.

## Built on by (most cited first, 8 of 8)

- [Bayesian subset selection and variable importance for interpretable prediction and classification](https://mlsanity.com/paper/2104.10150/) (2021)
- [A fully Bayesian sparse polynomial chaos expansion approach with joint priors on the coefficients and global selection of terms](https://mlsanity.com/paper/2204.06043/) (2022)
- [Meta-Uncertainty in Bayesian Model Comparison](https://mlsanity.com/paper/2210.07278/) (2022)
- [Uncertainty Quantification and Propagation in Surrogate-based Bayesian Inference](https://mlsanity.com/paper/2312.05153/) (2023)
- [Decoupling Shrinkage and Selection for the Bayesian Quantile Regression](https://mlsanity.com/paper/2107.08498/) (2021)
- [Scalable mixed-domain Gaussian process modeling and model reduction for longitudinal data](https://mlsanity.com/paper/2111.02019/) (2021)
- [Variational Bayesian Bow tie Neural Networks with Shrinkage](https://mlsanity.com/paper/2411.11132/) (2024)
- [Bayesian Model Averaging under Predictor Redundancy via Density-Ratio Posterior Compression](https://mlsanity.com/paper/2606.21080/) (2026)

## Abstract (arXiv)

This paper discusses predictive inference and feature selection for generalized linear models with scarce but high-dimensional data. We argue that in many cases one can benefit from a decision theoretically justified two-stage approach: first, construct a possibly non-sparse model that predicts well, and then find a minimal subset of features that characterize the predictions. The model built in the first step is referred to as the reference model and the operation during the latter step as predictive projection. The key characteristic of this approach is that it finds an excellent tradeoff between sparsity and predictive accuracy, and the gain comes from utilizing all available information including prior and that coming from the left out features. We review several methods that follow this principle and provide novel methodological contributions. We present a new projection technique that unifies two existing techniques and is both accurate and fast to compute. We also propose a way of evaluating the feature selection process using fast leave-one-out cross-validation that allows for easy and intuitive model size selection. Furthermore, we prove a theorem that helps to understand the conditions under which the projective approach could be beneficial. The benefits are illustrated via several simulated and real world examples.