# Performance is not enough: the story told by a Rashomon quartet

arXiv: https://arxiv.org/abs/2302.13356 · HTML: https://mlsanity.com/paper/2302.13356/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 26 Feb 2023.

## Builds on (most cited first, 8 of 8)

- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [DALEX: explainers for complex predictive models](https://mlsanity.com/paper/1806.08915/) (2018)
- [The Grammar of Interactive Explanatory Model Analysis](https://mlsanity.com/paper/2005.00497/) (2020)
- [Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process](https://mlsanity.com/paper/2109.01433/) (2021)
- [Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanations](https://mlsanity.com/paper/2105.06791/) (2021)
- [Causal quartets: Different ways to attain the same average treatment effect](https://mlsanity.com/paper/2302.12878/) (2023)

## Built on by (most cited first, 9 of 9)

- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Exploration of the Rashomon Set Assists Trustworthy Explanations for Medical Data](https://mlsanity.com/paper/2308.11446/) (2023)
- [An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification](https://mlsanity.com/paper/2405.01557/) (2024)
- [The Role of Hyperparameters in Predictive Multiplicity](https://mlsanity.com/paper/2503.13506/) (2025)
- [Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning](https://mlsanity.com/paper/2503.08636/) (2025)
- [Rashomon effect in Educational Research: Why More is Better Than One for Measuring the Importance of the Variables?](https://mlsanity.com/paper/2412.12115/) (2024)
- [Towards the Formalization of a Trustworthy AI for Mining Interpretable Models explOiting Sophisticated Algorithms](https://mlsanity.com/paper/2510.20621/) (2025)
- [VAR: Visual Analysis for Rashomon Set of Machine Learning Models' Performance](https://mlsanity.com/paper/2507.22556/) (2025)
- [Analyzing the Temporal Factors for Anxiety and Depression Symptoms with the Rashomon Perspective](https://mlsanity.com/paper/2601.20874/) (2026)

## Abstract (arXiv)

The usual goal of supervised learning is to find the best model, the one that optimizes a particular performance measure. However, what if the explanation provided by this model is completely different from another model and different again from another model despite all having similarly good fit statistics? Is it possible that the equally effective models put the spotlight on different relationships in the data? Inspired by Anscombe's quartet, this paper introduces a Rashomon Quartet, i.e. a set of four models built on a synthetic dataset which have practically identical predictive performance. However, the visual exploration reveals distinct explanations of the relations in the data. This illustrative example aims to encourage the use of methods for model visualization to compare predictive models beyond their performance.