# An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification

arXiv: https://arxiv.org/abs/2405.01557 · HTML: https://mlsanity.com/paper/2405.01557/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 48 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [Rashomon Capacity: A Metric for Predictive Multiplicity in Classification](https://mlsanity.com/paper/2206.01295/) (2022)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance](https://mlsanity.com/paper/2309.13775/) (2023)
- [The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression](https://mlsanity.com/paper/2202.09101/) (2022)
- [Rank List Sensitivity of Recommender Systems to Interaction Perturbations](https://mlsanity.com/paper/2201.12686/) (2022)
- [Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation](https://mlsanity.com/paper/2402.00728/) (2024)
- [The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions](https://mlsanity.com/paper/2304.10655/) (2023)
- [To SMOTE, or not to SMOTE?](https://mlsanity.com/paper/2201.08528/) (2022)
- [One Model Many Scores: Using Multiverse Analysis to Prevent Fairness Hacking and Evaluate the Influence of Model Design Decisions](https://mlsanity.com/paper/2308.16681/) (2023)
- [Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints](https://mlsanity.com/paper/2306.13738/) (2023)
- [Performance is not enough: the story told by a Rashomon quartet](https://mlsanity.com/paper/2302.13356/) (2023)
- [The Effect of Balancing Methods on Model Behavior in Imbalanced Classification Problems](https://mlsanity.com/paper/2307.00157/) (2023)
- [Partial Resampling of Imbalanced Data](https://mlsanity.com/paper/2207.04631/) (2022)

## Built on by (most cited first, 11 of 11)

- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML](https://mlsanity.com/paper/2507.14744/) (2025)
- [The Role of Hyperparameters in Predictive Multiplicity](https://mlsanity.com/paper/2503.13506/) (2025)
- [Rashomon perspective for measuring uncertainty in the survival predictive maintenance models](https://mlsanity.com/paper/2502.15772/) (2025)
- [Explainable bank failure prediction models: Counterfactual explanations to reduce the failure risk](https://mlsanity.com/paper/2407.11089/) (2024)
- [forester: A Tree-Based AutoML Tool in R](https://mlsanity.com/paper/2409.04789/) (2024)
- [Rashomon effect in Educational Research: Why More is Better Than One for Measuring the Importance of the Variables?](https://mlsanity.com/paper/2412.12115/) (2024)
- [Predictive Multiplicity in Survival Models: A Method for Quantifying Model Uncertainty in Predictive Maintenance Applications](https://mlsanity.com/paper/2504.12156/) (2025)
- [Argumentative Ensembling for Robust Recourse under Model Multiplicity](https://mlsanity.com/paper/2506.20260/) (2025)
- [Robust Ambiguity Detection (RAD) From Model- and Feature-Space Consistency](https://mlsanity.com/paper/2608.11541/) (2026)
- [Mitigating the Multiplicity Burden: The Role of Calibration in Reducing Predictive Multiplicity of Classifiers](https://mlsanity.com/paper/2603.11750/) (2026)

## Abstract (arXiv)

Predictive models may generate biased predictions when classifying imbalanced datasets. This happens when the model favors the majority class, leading to low performance in accurately predicting the minority class. To address this issue, balancing or resampling methods are critical data-centric AI approaches in the modeling process to improve prediction performance. However, there have been debates and questions about the functionality of these methods in recent years. In particular, many candidate models may exhibit very similar predictive performance, called the Rashomon effect, in model selection, and they may even produce different predictions for the same observations. Selecting one of these models without considering the predictive multiplicity -- which is the case of yielding conflicting models' predictions for any sample -- can result in blind selection. In this paper, the impact of balancing methods on predictive multiplicity is examined using the Rashomon effect. It is crucial because the blind model selection in data-centric AI is risky from a set of approximately equally accurate models. This may lead to severe problems in model selection, validation, and explanation. To tackle this matter, we conducted real dataset experiments to observe the impact of balancing methods on predictive multiplicity through the Rashomon effect by using a newly proposed metric obscurity in addition to the existing ones: ambiguity and discrepancy. Our findings showed that balancing methods inflate the predictive multiplicity and yield varying results. To monitor the trade-off between the prediction performance and predictive multiplicity for conducting the modeling process responsibly, we proposed using the extended version of the performance-gain plot when balancing the training data.