# Amazing Things Come From Having Many Good Models

arXiv: https://arxiv.org/abs/2407.04846 · HTML: https://mlsanity.com/paper/2407.04846/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 30 of 32)

- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [Generalized and Scalable Optimal Sparse Decision Trees](https://mlsanity.com/paper/2006.08690/) (2020)
- [On the Existence of Simpler Machine Learning Models](https://mlsanity.com/paper/1908.01755/) (2019)
- [Exploring the Whole Rashomon Set of Sparse Decision Trees](https://mlsanity.com/paper/2209.08040/) (2022)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [Characterizing Fairness Over the Set of Good Models Under Selective Labels](https://mlsanity.com/paper/2101.00352/) (2021)
- [The age of secrecy and unfairness in recidivism prediction](https://mlsanity.com/paper/1811.00731/) (2018)
- [Rashomon Capacity: A Metric for Predictive Multiplicity in Classification](https://mlsanity.com/paper/2206.01295/) (2022)
- [Interpretable Classification Models for Recidivism Prediction](https://mlsanity.com/paper/1503.07810/) (2015)
- [Empirical observation of negligible fairness-accuracy trade-offs in machine learning for public policy](https://mlsanity.com/paper/2012.02972/) (2020)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction](https://mlsanity.com/paper/2005.04176/) (2020)
- [Fast Sparse Decision Tree Optimization via Reference Ensembles](https://mlsanity.com/paper/2112.00798/) (2021)
- [A Path to Simpler Models Starts With Noise](https://mlsanity.com/paper/2310.19726/) (2023)
- [Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification](https://mlsanity.com/paper/2301.11562/) (2023)
- [Exploring and Interacting with the Set of Good Sparse Generalized Additive Models](https://mlsanity.com/paper/2303.16047/) (2023)
- [The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance](https://mlsanity.com/paper/2309.13775/) (2023)
- [FasterRisk: Fast and Accurate Interpretable Risk Scores](https://mlsanity.com/paper/2210.05846/) (2022)
- [Fast Sparse Classification for Generalized Linear and Additive Models](https://mlsanity.com/paper/2202.11389/) (2022)
- [GAM Changer: Editing Generalized Additive Models with Interactive Visualization](https://mlsanity.com/paper/2112.03245/) (2021)
- [A Holistic Approach to Interpretability in Financial Lending: Models, Visualizations, and Summary-Explanations](https://mlsanity.com/paper/2106.02605/) (2021)
- [A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results](https://mlsanity.com/paper/1804.08646/) (2018)
- [TimberTrek: Exploring and Curating Sparse Decision Trees with Interactive Visualization](https://mlsanity.com/paper/2209.09227/) (2022)
- [Computing the Collection of Good Models for Rule Lists](https://mlsanity.com/paper/2204.11285/) (2022)
- [OKRidge: Scalable Optimal k-Sparse Ridge Regression](https://mlsanity.com/paper/2304.06686/) (2023)

## Built on by (most cited first, 30 of 55)

- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Rashomon Sets for Prototypical-Part Networks: Editing Interpretable Models in Real-Time](https://mlsanity.com/paper/2503.01087/) (2025)
- [Be Intentional About Fairness!: Fairness, Size, and Multiplicity in the Rashomon Set](https://mlsanity.com/paper/2501.15634/) (2025)
- [What Constitutes a Less Discriminatory Algorithm?](https://mlsanity.com/paper/2412.18138/) (2024)
- [Near Optimal Decision Trees in a SPLIT Second](https://mlsanity.com/paper/2502.15988/) (2025)
- [Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road of Building Responsible AI Systems](https://mlsanity.com/paper/2505.18139/) (2025)
- [SORTeD Rashomon Sets of Sparse Decision Trees: Anytime Enumeration](https://mlsanity.com/paper/2511.03344/) (2025)
- [The Role of Hyperparameters in Predictive Multiplicity](https://mlsanity.com/paper/2503.13506/) (2025)
- [Leveraging Predictive Equivalence in Decision Trees](https://mlsanity.com/paper/2506.14143/) (2025)
- [PCS Workflow for Veridical Data Science in the Age of AI](https://mlsanity.com/paper/2508.00835/) (2025)
- [Unsupervised Machine Learning for Scientific Discovery: Workflow and Best Practices](https://mlsanity.com/paper/2506.04553/) (2025)
- [Doctor Rashomon and the UNIVERSE of Madness: Variable Importance with Unobserved Confounding and the Rashomon Effect](https://mlsanity.com/paper/2510.12734/) (2025)
- [All Emulators are Wrong, Many are Useful, and Some are More Useful Than Others: A Reproducible Comparison of Computer Model Surrogates](https://mlsanity.com/paper/2512.09060/) (2025)
- [Influence-based Attributions can be Manipulated](https://mlsanity.com/paper/2409.05208/) (2024)
- [Rashomon perspective for measuring uncertainty in the survival predictive maintenance models](https://mlsanity.com/paper/2502.15772/) (2025)
- [Uncertainty Quantification as a Principled Foundation for Explainable Artificial Intelligence: A Case Study of Counterfactual Explanations](https://mlsanity.com/paper/2502.17007/) (2025)
- [Unique Rashomon Sets for Robust Active Learning](https://mlsanity.com/paper/2503.06770/) (2025)
- [ElliCE: Efficient and Provably Robust Algorithmic Recourse via the Rashomon Sets](https://mlsanity.com/paper/2602.07674/) (2026)
- [Using predictive multiplicity to measure individual performance within the AI Act](https://mlsanity.com/paper/2602.11944/) (2026)
- [DPA: A one-stop metric to measure bias amplification in classification datasets](https://mlsanity.com/paper/2412.11060/) (2024)
- [Predictive Multiplicity in Survival Models: A Method for Quantifying Model Uncertainty in Predictive Maintenance Applications](https://mlsanity.com/paper/2504.12156/) (2025)
- [Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity](https://mlsanity.com/paper/2505.16638/) (2025)
- [Leo Breiman, the Rashomon Effect, and the Occam Dilemma](https://mlsanity.com/paper/2507.03884/) (2025)
- [On Arbitrary Predictions from Equally Valid Models](https://mlsanity.com/paper/2507.19408/) (2025)
- [Towards the Formalization of a Trustworthy AI for Mining Interpretable Models explOiting Sophisticated Algorithms](https://mlsanity.com/paper/2510.20621/) (2025)
- [Complete Evidence Extraction with Model Ensembles: A Case Study on Medical Coding](https://mlsanity.com/paper/2511.07055/) (2025)
- [Parameter-Efficient Construction of the Rashomon Slice for Concept Bottleneck Models](https://mlsanity.com/paper/2511.19636/) (2025)
- [Why are there many equally good models? An Anatomy of the Rashomon Effect](https://mlsanity.com/paper/2601.06730/) (2026)
- [Resolving Predictive Multiplicity for the Rashomon Set](https://mlsanity.com/paper/2601.09071/) (2026)
- [CASHomon Sets: Efficient Rashomon Sets Across Multiple Model Classes and their Hyperparameters](https://mlsanity.com/paper/2603.15321/) (2026)

## Abstract (arXiv)

The Rashomon Effect, coined by Leo Breiman, describes the phenomenon that there exist many equally good predictive models for the same dataset. This phenomenon happens for many real datasets and when it does, it sparks both magic and consternation, but mostly magic. In light of the Rashomon Effect, this perspective piece proposes reshaping the way we think about machine learning, particularly for tabular data problems in the nondeterministic (noisy) setting. We address how the Rashomon Effect impacts (1) the existence of simple-yet-accurate models, (2) flexibility to address user preferences, such as fairness and monotonicity, without losing performance, (3) uncertainty in predictions, fairness, and explanations, (4) reliable variable importance, (5) algorithm choice, specifically, providing advanced knowledge of which algorithms might be suitable for a given problem, and (6) public policy. We also discuss a theory of when the Rashomon Effect occurs and why. Our goal is to illustrate how the Rashomon Effect can have a massive impact on the use of machine learning for complex problems in society.