# Stabilizing Variable Selection and Regression

arXiv: https://arxiv.org/abs/1911.01850 · HTML: https://mlsanity.com/paper/1911.01850/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2019.

## Built on by (most cited first, 30 of 31)

- [Towards a Theoretical Framework of Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.04496/) (2021)
- [Invariant Causal Mechanisms through Distribution Matching](https://mlsanity.com/paper/2206.11646/) (2022)
- [Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena](https://mlsanity.com/paper/2206.05487/) (2022)
- [Improvement-Focused Causal Recourse (ICR)](https://mlsanity.com/paper/2210.15709/) (2022)
- [A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization](https://mlsanity.com/paper/2111.02355/) (2021)
- [Exploiting Independent Instruments: Identification and Distribution Generalization](https://mlsanity.com/paper/2202.01864/) (2022)
- [Invariant Policy Learning: A Causal Perspective](https://mlsanity.com/paper/2106.00808/) (2021)
- [Environment Invariant Linear Least Squares](https://mlsanity.com/paper/2303.03092/) (2023)
- [Characterization and Greedy Learning of Gaussian Structural Causal Models under Unknown Interventions](https://mlsanity.com/paper/2211.14897/) (2022)
- [A Causal Perspective on Meaningful and Robust Algorithmic Recourse](https://mlsanity.com/paper/2107.07853/) (2021)
- [Performative Validity of Recourse Explanations](https://mlsanity.com/paper/2506.15366/) (2025)
- [Causality Pursuit from Heterogeneous Environments via Neural Adversarial Invariance Learning](https://mlsanity.com/paper/2405.04715/) (2024)
- [Causality-oriented robustness: exploiting general noise interventions](https://mlsanity.com/paper/2307.10299/) (2023)
- [Learning Invariant Representations under General Interventions on the Response](https://mlsanity.com/paper/2208.10027/) (2022)
- [Scalable Causal Domain Adaptation](https://mlsanity.com/paper/2103.00139/) (2021)
- [Loss-guided Stability Selection](https://mlsanity.com/paper/2202.04956/) (2022)
- [An Invariant Matching Property for Distribution Generalization under Intervened Response](https://mlsanity.com/paper/2205.09162/) (2022)
- [Causality-Inspired Robustness for Nonlinear Models via Representation Learning](https://mlsanity.com/paper/2505.12868/) (2025)
- [Generalized Invariant Matching Property via LASSO](https://mlsanity.com/paper/2301.05975/) (2023)
- [Multi-dimensional domain generalization with low-rank structures](https://mlsanity.com/paper/2309.09555/) (2023)
- [Invariant Subspace Decomposition](https://mlsanity.com/paper/2404.09962/) (2024)
- [Causal Invariance Learning via Efficient Nonconvex Optimization](https://mlsanity.com/paper/2412.11850/) (2024)
- [Stability Selection via Variable Decorrelation](https://mlsanity.com/paper/2505.20864/) (2025)
- [Stable Blanket with Hidden Variables and Cycles](https://mlsanity.com/paper/2605.01856/) (2026)
- [FAIRM: Learning invariant representations for algorithmic fairness and domain generalization with minimax optimality](https://mlsanity.com/paper/2404.01608/) (2024)
- [Mining Invariance from Nonlinear Multi-Environment Data: Binary Classification](https://mlsanity.com/paper/2404.15245/) (2024)
- [Considerations for Distribution Shift Robustness of Diagnostic Models in Healthcare](https://mlsanity.com/paper/2410.19575/) (2024)
- [Anti-causal domain generalization: Leveraging unlabeled data](https://mlsanity.com/paper/2602.17187/) (2026)
- [2D Stability Selection: Design Jittering for Doubly Stable Feature Selection](https://mlsanity.com/paper/2605.02205/) (2026)
- [Prediction-Intervention Games and Invariant Sets](https://mlsanity.com/paper/2605.16828/) (2026)

## Abstract (arXiv)

We consider regression in which one predicts a response Y with a set of predictors X across different experiments or environments. This is a common setup in many data-driven scientific fields and we argue that statistical inference can benefit from an analysis that takes into account the distributional changes across environments. In particular, it is useful to distinguish between stable and unstable predictors, i.e., predictors which have a fixed or a changing functional dependence on the response, respectively. We introduce stabilized regression which explicitly enforces stability and thus improves generalization performance to previously unseen environments. Our work is motivated by an application in systems biology. Using multiomic data, we demonstrate how hypothesis generation about gene function can benefit from stabilized regression. We believe that a similar line of arguments for exploiting heterogeneity in data can be powerful for many other applications as well. We draw a theoretical connection between multi-environment regression and causal models, which allows to graphically characterize stable versus unstable functional dependence on the response. Formally, we introduce the notion of a stable blanket which is a subset of the predictors that lies between the direct causal predictors and the Markov blanket. We prove that this set is optimal in the sense that a regression based on these predictors minimizes the mean squared prediction error given that the resulting regression generalizes to unseen new environments.