# Rashomon Capacity: A Metric for Predictive Multiplicity in Classification

arXiv: https://arxiv.org/abs/2206.01295 · HTML: https://mlsanity.com/paper/2206.01295/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 68 references that are in the map.
- First posted on arXiv 2 Jun 2022.

## Builds on (most cited first, 17 of 17)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [On Counterfactual Explanations under Predictive Multiplicity](https://mlsanity.com/paper/2006.13132/) (2020)
- [Characterizing Fairness Over the Set of Good Models Under Selective Labels](https://mlsanity.com/paper/2101.00352/) (2021)
- [Planting Undetectable Backdoors in Machine Learning Models](https://mlsanity.com/paper/2204.06974/) (2022)
- [Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective](https://mlsanity.com/paper/2203.08124/) (2022)
- [Formalizing Generalization and Robustness of Neural Networks to Weight Perturbations](https://mlsanity.com/paper/2103.02200/) (2021)
- [Nearly Optimal Algorithms for Level Set Estimation](https://mlsanity.com/paper/2111.01768/) (2021)
- [The sample complexity of level set approximation](https://mlsanity.com/paper/2010.13405/) (2020)

## Built on by (most cited first, 30 of 47)

- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Amazing Things Come From Having Many Good Models](https://mlsanity.com/paper/2407.04846/) (2024)
- [Robust Counterfactual Explanations for Neural Networks With Probabilistic Guarantees](https://mlsanity.com/paper/2305.11997/) (2023)
- [A Path to Simpler Models Starts With Noise](https://mlsanity.com/paper/2310.19726/) (2023)
- [Exploring and Interacting with the Set of Good Sparse Generalized Additive Models](https://mlsanity.com/paper/2303.16047/) (2023)
- [Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation](https://mlsanity.com/paper/2402.00728/) (2024)
- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Recourse under Model Multiplicity via Argumentative Ensembling (Technical Report)](https://mlsanity.com/paper/2312.15097/) (2023)
- [Predictive Churn with the Set of Good Models](https://mlsanity.com/paper/2402.07745/) (2024)
- [Algorithmic Arbitrariness in Content Moderation](https://mlsanity.com/paper/2402.16979/) (2024)
- [Rashomon Sets for Prototypical-Part Networks: Editing Interpretable Models in Real-Time](https://mlsanity.com/paper/2503.01087/) (2025)
- [Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints](https://mlsanity.com/paper/2306.13738/) (2023)
- [Exploration of the Rashomon Set Assists Trustworthy Explanations for Medical Data](https://mlsanity.com/paper/2308.11446/) (2023)
- [An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification](https://mlsanity.com/paper/2405.01557/) (2024)
- [Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective](https://mlsanity.com/paper/2412.09712/) (2024)
- [Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs](https://mlsanity.com/paper/2407.04173/) (2024)
- [Asymmetry of the Relative Entropy in the Regularization of Empirical Risk Minimization](https://mlsanity.com/paper/2410.02833/) (2024)
- [Exploring the cloud of feature interaction scores in a Rashomon set](https://mlsanity.com/paper/2305.10181/) (2023)
- [X Hacking: The Threat of Misguided AutoML](https://mlsanity.com/paper/2401.08513/) (2024)
- [Counterfactual Explanations for Model Ensembles Using Entropic Risk Measures](https://mlsanity.com/paper/2503.07934/) (2025)
- [The Cost of Arbitrariness for Individuals: Examining the Legal and Technical Challenges of Model Multiplicity](https://mlsanity.com/paper/2407.13070/) (2024)
- [Fairness and Sparsity within Rashomon sets: Enumeration-Free Exploration and Characterization](https://mlsanity.com/paper/2502.05286/) (2025)
- [Practical Attribution Guidance for Rashomon Sets](https://mlsanity.com/paper/2407.18482/) (2024)
- [Unsupervised Machine Learning for Scientific Discovery: Workflow and Best Practices](https://mlsanity.com/paper/2506.04553/) (2025)
- [Welfare as a Guiding Principle for Machine Learning -- From Compass, to Lens, to Roadmap](https://mlsanity.com/paper/2502.11981/) (2025)
- [Observational Multiplicity](https://mlsanity.com/paper/2507.23136/) (2025)
- [ElliCE: Efficient and Provably Robust Algorithmic Recourse via the Rashomon Sets](https://mlsanity.com/paper/2602.07674/) (2026)
- [Using predictive multiplicity to measure individual performance within the AI Act](https://mlsanity.com/paper/2602.11944/) (2026)
- [An Epistemic and Aleatoric Decomposition of Arbitrariness to Constrain the Set of Good Models](https://mlsanity.com/paper/2302.04525/) (2023)
- [Diverse Explanations From Data-Driven and Domain-Driven Perspectives in the Physical Sciences](https://mlsanity.com/paper/2402.00347/) (2024)

## Abstract (arXiv)

Predictive multiplicity occurs when classification models with statistically indistinguishable performances assign conflicting predictions to individual samples. When used for decision-making in applications of consequence (e.g., lending, education, criminal justice), models developed without regard for predictive multiplicity may result in unjustified and arbitrary decisions for specific individuals. We introduce a new metric, called Rashomon Capacity, to measure predictive multiplicity in probabilistic classification. Prior metrics for predictive multiplicity focus on classifiers that output thresholded (i.e., 0-1) predicted classes. In contrast, Rashomon Capacity applies to probabilistic classifiers, capturing more nuanced score variations for individual samples. We provide a rigorous derivation for Rashomon Capacity, argue its intuitive appeal, and demonstrate how to estimate it in practice. We show that Rashomon Capacity yields principled strategies for disclosing conflicting models to stakeholders. Our numerical experiments illustrate how Rashomon Capacity captures predictive multiplicity in various datasets and learning models, including neural networks. The tools introduced in this paper can help data scientists measure and report predictive multiplicity prior to model deployment.