# Loss-guided Stability Selection

arXiv: https://arxiv.org/abs/2202.04956 · HTML: https://mlsanity.com/paper/2202.04956/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 64 references that are in the map.
- First posted on arXiv 10 Feb 2022.

## Builds on (most cited first, 9 of 9)

- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [Stabilizing Variable Selection and Regression](https://mlsanity.com/paper/1911.01850/) (2019)
- [High-Dimensional Inference: Confidence Intervals, p-Values and R-Software hdi](https://mlsanity.com/paper/1408.4026/) (2014)
- [Stability selection for component-wise gradient boosting in multiple dimensions](https://mlsanity.com/paper/1611.10171/) (2016)
- [Estimation Stability with Cross Validation (ESCV)](https://mlsanity.com/paper/1303.3128/) (2013)
- [Controlling false discoveries in high-dimensional situations: Boosting with stability selection](https://mlsanity.com/paper/1411.1285/) (2014)
- [Adaptive sparse group LASSO in quantile regression](https://mlsanity.com/paper/1911.01081/) (2019)
- [High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking](https://mlsanity.com/paper/1808.00723/) (2018)
- [Trimming Stability Selection increases variable selection robustness](https://mlsanity.com/paper/2111.11818/) (2021)

## Built on by (most cited first, 3 of 3)

- [Convergence of uncertainty estimates in Ensemble and Bayesian sparse model discovery](https://mlsanity.com/paper/2301.12649/) (2023)
- [Integrated path stability selection](https://mlsanity.com/paper/2403.15877/) (2024)
- [Robust variable selection for spatial point processes observed with noise](https://mlsanity.com/paper/2510.25550/) (2025)

## Abstract (arXiv)

In modern data analysis, sparse model selection becomes inevitable once the number of predictors variables is very high. It is well-known that model selection procedures like the Lasso or Boosting tend to overfit on real data. The celebrated Stability Selection overcomes these weaknesses by aggregating models, based on subsamples of the training data, followed by choosing a stable predictor set which is usually much sparser than the predictor sets from the raw models. The standard Stability Selection is based on a global criterion, namely the per-family error rate, while additionally requiring expert knowledge to suitably configure the hyperparameters. Since model selection depends on the loss function, i.e., predictor sets selected w.r.t. some particular loss function differ from those selected w.r.t. some other loss function, we propose a Stability Selection variant which respects the chosen loss function via an additional validation step based on out-of-sample validation data, optionally enhanced with an exhaustive search strategy. Our Stability Selection variants are widely applicable and user-friendly. Moreover, our Stability Selection variants can avoid the issue of severe underfitting which affects the original Stability Selection for noisy high-dimensional data, so our priority is not to avoid false positives at all costs but to result in a sparse stable model with which one can make predictions. Experiments where we consider both regression and binary classification and where we use Boosting as model selection algorithm reveal a significant precision improvement compared to raw Boosting models while not suffering from any of the mentioned issues of the original Stability Selection.