# Comparing Model-agnostic Feature Selection Methods through Relative Efficiency

arXiv: https://arxiv.org/abs/2508.14268 · HTML: https://mlsanity.com/paper/2508.14268/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 55 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 9 of 9)

- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection](https://mlsanity.com/paper/1610.02351/) (2016)
- [Controlling the false discovery rate via knockoffs](https://mlsanity.com/paper/1404.5609/) (2014)
- [A general framework for inference on algorithm-agnostic variable importance](https://mlsanity.com/paper/2004.03683/) (2020)
- [Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process](https://mlsanity.com/paper/2109.01433/) (2021)
- [Dropout Feature Ranking for Deep Learning Models](https://mlsanity.com/paper/1712.08645/) (2017)
- [Lazy Estimation of Variable Importance for Large Neural Networks](https://mlsanity.com/paper/2207.09097/) (2022)
- [Reliable and scalable variable importance estimation via warm-start and early stopping](https://mlsanity.com/paper/2412.01120/) (2024)

## Abstract (arXiv)

Feature selection and importance estimation in a model-agnostic setting is an ongoing challenge of significant interest. Wrapper methods are commonly used because they are typically model-agnostic. In this paper, we develop a general comparison framework for model-agnostic feature selection methods based on relative efficiency, using relative variability σ/μ to account for different statistics having different means. In particular we focus on state-of-the-art feature selection methods, the Generalized Covariance Measure (GCM) and Leave-One-Covariate-Out (LOCO) estimation. In particular, we present a theoretical comparison under three model settings: linear models, non-linear additive models, and single index models that mimic a single-layer neural network. We complement this with simulations and real data examples for the above models and mis-specified models. Our theoretical results, along with empirical findings, demonstrate that GCM-related methods generally out-perform LOCO under suitable regularity conditions defined by a suitably defined correlation quantity which quantifies the asymptotic relative efficiency of these approaches. Our simulations and real data analysis include widely used machine learning methods such as neural networks and gradient boosting trees.