# An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learning

arXiv: https://arxiv.org/abs/2306.15786 · HTML: https://mlsanity.com/paper/2306.15786/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 26 of 26)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [Explainable Machine Learning for Scientific Insights and Discoveries](https://mlsanity.com/paper/1905.08883/) (2019)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022)
- [Making deep neural networks right for the right scientific reasons by interacting with their explanations](https://mlsanity.com/paper/2001.05371/) (2020)
- [On the Existence of Simpler Machine Learning Models](https://mlsanity.com/paper/1908.01755/) (2019)
- [Exploring the Whole Rashomon Set of Sparse Decision Trees](https://mlsanity.com/paper/2209.08040/) (2022)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [Incorporating Priors with Feature Attribution on Text Classification](https://mlsanity.com/paper/1906.08286/) (2019)
- [Order in the Court: Explainable AI Methods Prone to Disagreement](https://mlsanity.com/paper/2105.03287/) (2021)
- [Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanations](https://mlsanity.com/paper/2105.06791/) (2021)
- [Comparing Explanation Methods for Traditional Machine Learning Models Part 1: An Overview of Current Methods and Quantifying Their Disagreement](https://mlsanity.com/paper/2211.08943/) (2022)
- [On The Stability of Interpretable Models](https://mlsanity.com/paper/1810.09352/) (2018)
- [Saliency Diversified Deep Ensemble for Robustness to Adversaries](https://mlsanity.com/paper/2112.03615/) (2021)

## Built on by (most cited first, 25 of 25)

- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML](https://mlsanity.com/paper/2507.14744/) (2025)
- [The Role of Hyperparameters in Predictive Multiplicity](https://mlsanity.com/paper/2503.13506/) (2025)
- [The Anatomy of Evidence: An Investigation Into Explainable ICD Coding](https://mlsanity.com/paper/2507.01802/) (2025)
- [Rashomon perspective for measuring uncertainty in the survival predictive maintenance models](https://mlsanity.com/paper/2502.15772/) (2025)
- [Rashomon in the Streets: Explanation Ambiguity in Scene Understanding](https://mlsanity.com/paper/2509.03169/) (2025)
- [ElliCE: Efficient and Provably Robust Algorithmic Recourse via the Rashomon Sets](https://mlsanity.com/paper/2602.07674/) (2026)
- [Dynamic Interpretability for Model Comparison via Decision Rules](https://mlsanity.com/paper/2309.17095/) (2023)
- [Local Universal Explainer (LUX) -- a rule-based explainer with factual, counterfactual and visual explanations](https://mlsanity.com/paper/2310.14894/) (2023)
- [DPA: A one-stop metric to measure bias amplification in classification datasets](https://mlsanity.com/paper/2412.11060/) (2024)
- [CFIRE: A General Method for Combining Local Explanations](https://mlsanity.com/paper/2504.00930/) (2025)
- [Towards the Formalization of a Trustworthy AI for Mining Interpretable Models explOiting Sophisticated Algorithms](https://mlsanity.com/paper/2510.20621/) (2025)
- [ProtoTSNet: Interpretable Multivariate Time Series Classification With Prototypical Parts](https://mlsanity.com/paper/2511.02152/) (2025)
- [Complete Evidence Extraction with Model Ensembles: A Case Study on Medical Coding](https://mlsanity.com/paper/2511.07055/) (2025)
- [CASHomon Sets: Efficient Rashomon Sets Across Multiple Model Classes and their Hyperparameters](https://mlsanity.com/paper/2603.15321/) (2026)
- [Is Machine Learning Unsafe and Irresponsible in Social Sciences? Paradoxes and Reconsidering from Recidivism Prediction Tasks](https://mlsanity.com/paper/2311.06537/) (2023)
- [A Look into How Machine Learning is Reshaping Engineering Models: the Rise of Analysis Paralysis, Optimal yet Infeasible Solutions, and the Inevitable Rashomon Paradox](https://mlsanity.com/paper/2501.04894/) (2025)
- [The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learning](https://mlsanity.com/paper/2511.21799/) (2025)
- [Translating the Rashomon Effect to Sequential Decision-Making Tasks](https://mlsanity.com/paper/2512.17470/) (2025)
- [Sensivity of LLMs' Explanations to the Training Randomness:Context, Class & Task Dependencies](https://mlsanity.com/paper/2603.08241/) (2026)
- [Non-identifiability of Explanations from Model Behavior in Deep Networks of Image Authenticity Judgments](https://mlsanity.com/paper/2604.07254/) (2026)
- [Auditing Near-Optimal Policies Can Be Exponentially Hard: Conditional Query Lower Bounds via Occupancy Rashomon Capacity](https://mlsanity.com/paper/2606.00414/) (2026)
- [Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning](https://mlsanity.com/paper/2606.06056/) (2026)
- [The Impact of Stochasticity on the Rashomon Effect in Machine Learning](https://mlsanity.com/paper/2609.32934/) (2026)
- [Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packages](https://mlsanity.com/paper/2609.36970/) (2026)

## Abstract (arXiv)

The Rashomon Effect describes the following phenomenon: for a given dataset there may exist many models with equally good performance but with different solution strategies. The Rashomon Effect has implications for Explainable Machine Learning, especially for the comparability of explanations. We provide a unified view on three different comparison scenarios and conduct a quantitative evaluation across different datasets, models, attribution methods, and metrics. We find that hyperparameter-tuning plays a role and that metric selection matters. Our results provide empirical support for previously anecdotal evidence and exhibit challenges for both scientists and practitioners.