# All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously

arXiv: https://arxiv.org/abs/1801.01489 · HTML: https://mlsanity.com/paper/1801.01489/

## Key facts

- Cited by 373 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jan 2018.

## Built on by (most cited first, 30 of 373)

- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Exploring the Whole Rashomon Set of Sparse Decision Trees](https://mlsanity.com/paper/2209.08040/) (2022)
- [Characterizing Fairness Over the Set of Good Models Under Selective Labels](https://mlsanity.com/paper/2101.00352/) (2021)
- [Diversify and Disambiguate: Learning From Underspecified Data](https://mlsanity.com/paper/2202.03418/) (2022)
- [Amazing Things Come From Having Many Good Models](https://mlsanity.com/paper/2407.04846/) (2024)
- [Directional Bias Amplification](https://mlsanity.com/paper/2102.12594/) (2021)
- [Adversarial attacks and defenses in explainable artificial intelligence: A survey](https://mlsanity.com/paper/2306.06123/) (2023)
- [Rashomon Capacity: A Metric for Predictive Multiplicity in Classification](https://mlsanity.com/paper/2206.01295/) (2022)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [Gender Artifacts in Visual Datasets](https://mlsanity.com/paper/2206.09191/) (2022)
- [Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis](https://mlsanity.com/paper/2202.10553/) (2022)
- [A Path to Simpler Models Starts With Noise](https://mlsanity.com/paper/2310.19726/) (2023)
- [Exploring and Interacting with the Set of Good Sparse Generalized Additive Models](https://mlsanity.com/paper/2303.16047/) (2023)
- [The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance](https://mlsanity.com/paper/2309.13775/) (2023)
- [Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?](https://mlsanity.com/paper/2401.13544/) (2024)
- [Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning](https://mlsanity.com/paper/2207.02598/) (2022)
- [An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learning](https://mlsanity.com/paper/2306.15786/) (2023)
- [Optimal ablation for interpretability](https://mlsanity.com/paper/2409.09951/) (2024)
- [Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process](https://mlsanity.com/paper/2109.01433/) (2021)
- [SurvSHAP(t): Time-dependent explanations of machine learning survival models](https://mlsanity.com/paper/2208.11080/) (2022)
- [Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena](https://mlsanity.com/paper/2206.05487/) (2022)
- [Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?](https://mlsanity.com/paper/2203.06487/) (2022)
- [Position: Explain to Question not to Justify](https://mlsanity.com/paper/2402.13914/) (2024)
- [Revealing the Barriers of Language Agents in Planning](https://mlsanity.com/paper/2410.12409/) (2024)
- [Sibyl: Understanding and Addressing the Usability Challenges of Machine Learning In High-Stakes Decision Making](https://mlsanity.com/paper/2103.02071/) (2021)
- [Arbitrary Decisions are a Hidden Cost of Differentially Private Training](https://mlsanity.com/paper/2302.14517/) (2023)
- [Loss-Aversively Fair Classification](https://mlsanity.com/paper/2105.04273/) (2021)
- [Rational Shapley Values](https://mlsanity.com/paper/2106.10191/) (2021)
- [Scrutinizing XAI using linear ground-truth data with suppressor variables](https://mlsanity.com/paper/2111.07473/) (2021)

## Abstract (arXiv)

Variable importance (VI) tools describe how much covariates contribute to a prediction model's accuracy. However, important variables for one well-performing model (for example, a linear model f(x)=xTβ with a fixed coefficient vector β) may be unimportant for another model. In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class. Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. In the process of deriving MCR, we show several informative results for permutation-based VI estimates, based on the VI measures used in Random Forests. Specifically, we derive connections between permutation importance estimates for a single prediction model, U-statistics, conditional variable importance, conditional causal effects, and linear model coefficients. We then give probabilistic bounds for MCR, using a novel, generalizable technique. We apply MCR to a public data set of Broward County criminal records to study the reliance of recidivism prediction models on sex and race. In this application, MCR can be used to help inform VI for unknown, proprietary models.