mlsanityData to 7 Oct 2026

An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learning

arXiv 2306.15786 ↗First posted 27 Jun 2023cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

The Rashomon Effect describes the following phenomenon: for a given dataset there may exist many models with equally good performance but with different solution strategies. The Rashomon Effect has implications for Explainable Machine Learning, especially for the comparability of explanations. We provide a unified view on three different comparison scenarios and conduct a quantitative evaluation across different datasets, models, attribution methods, and metrics. We find that hyperparameter-tuning plays a role and that metric selection matters. Our results provide empirical support for previously anecdotal evidence and exhibit challenges for both scientists and practitioners.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

26 of 41 references.

References · 26

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  7. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  8. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  9. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 364
  10. 2018On the Robustness of Interpretability Methodscited by 310
  11. 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
  12. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
14 more
  1. 2019Explainable Machine Learning for Scientific Insights and Discoveriescited by 165
  2. 2019Predictive Multiplicity in Classificationcited by 162
  3. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  4. 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 156
  5. 2020Making deep neural networks right for the right scientific reasons by interacting with their explanationscited by 141
  6. 2019On the Existence of Simpler Machine Learning Modelscited by 87
  7. 2022Exploring the Whole Rashomon Set of Sparse Decision Treescited by 80
  8. 2022Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanationscited by 77
  9. 2019Incorporating Priors with Feature Attribution on Text Classificationcited by 68
  10. 2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
  11. 2021Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanationscited by 15
  12. 2022Comparing Explanation Methods for Traditional Machine Learning Models Part 1: An Overview of Current Methods and Quantifying Their Disagreementcited by 14
  13. 2018On The Stability of Interpretable Modelscited by 2
  14. 2021Saliency Diversified Deep Ensemble for Robustness to Adversariescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 13; 2026: 8.

25 papers.

Sort:

  1. 2025Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learningcited by 17
  2. 2025Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoMLcited by 6
  3. 2025The Role of Hyperparameters in Predictive Multiplicitycited by 4
  4. 2025The Anatomy of Evidence: An Investigation Into Explainable ICD Codingcited by 3
  5. 2025Rashomon perspective for measuring uncertainty in the survival predictive maintenance modelscited by 2
  6. 2025Rashomon in the Streets: Explanation Ambiguity in Scene Understandingcited by 2
  7. 2026ElliCE: Efficient and Provably Robust Algorithmic Recourse via the Rashomon Setscited by 2
  8. 2023Dynamic Interpretability for Model Comparison via Decision Rulescited by 1
  9. 2023Local Universal Explainer (LUX) -- a rule-based explainer with factual, counterfactual and visual explanationscited by 1
  10. 2024DPA: A one-stop metric to measure bias amplification in classification datasetscited by 1
  11. 2025CFIRE: A General Method for Combining Local Explanationscited by 1
  12. 2025Towards the Formalization of a Trustworthy AI for Mining Interpretable Models explOiting Sophisticated Algorithmscited by 1
13 more
  1. 2025Complete Evidence Extraction with Model Ensembles: A Case Study on Medical Codingcited by 1
  2. 2026CASHomon Sets: Efficient Rashomon Sets Across Multiple Model Classes and their Hyperparameterscited by 1
  3. 2023Is Machine Learning Unsafe and Irresponsible in Social Sciences? Paradoxes and Reconsidering from Recidivism Prediction Taskscited by 0
  4. 2025A Look into How Machine Learning is Reshaping Engineering Models: the Rise of Analysis Paralysis, Optimal yet Infeasible Solutions, and the Inevitable Rashomon Paradoxcited by 0
  5. 2025ProtoTSNet: Interpretable Multivariate Time Series Classification With Prototypical Partscited by 0
  6. 2025The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learningcited by 0
  7. 2025Translating the Rashomon Effect to Sequential Decision-Making Taskscited by 0
  8. 2026Sensivity of LLMs' Explanations to the Training Randomness:Context, Class & Task Dependenciescited by 0
  9. 2026Non-identifiability of Explanations from Model Behavior in Deep Networks of Image Authenticity Judgmentscited by 0
  10. 2026Auditing Near-Optimal Policies Can Be Exponentially Hard: Conditional Query Lower Bounds via Occupancy Rashomon Capacitycited by 0
  11. 2026Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learningcited by 0
  12. 2026The Impact of Stochasticity on the Rashomon Effect in Machine Learningcited by 0
  13. 2026Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packagescited by 0