# On Model Evaluation under Non-constant Class Imbalance

arXiv: https://arxiv.org/abs/2001.05571 · HTML: https://mlsanity.com/paper/2001.05571/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2020.

## Built on by (most cited first, 8 of 8)

- [Learning image representations for anomaly detection: application to discovery of histological alterations in drug development](https://mlsanity.com/paper/2210.07675/) (2022)
- [Financial Fine-tuning a Large Time Series Model](https://mlsanity.com/paper/2412.09880/) (2024)
- [Synthetic Data Generation for Augmenting Small Samples](https://mlsanity.com/paper/2501.18741/) (2025)
- [Benchmark of Data Preprocessing Methods for Imbalanced Classification](https://mlsanity.com/paper/2303.03094/) (2023)
- [A Modular and Adaptive System for Business Email Compromise Detection](https://mlsanity.com/paper/2308.10776/) (2023)
- [Learning Invariant Representations with a Nonparametric Nadaraya-Watson Head](https://mlsanity.com/paper/2309.13377/) (2023)
- [Active Learning Framework to Automate NetworkTraffic Classification](https://mlsanity.com/paper/2211.08399/) (2022)
- [Machine learning models for estimating counterfactuals in a single-arm inflammatory bowel disease study](https://mlsanity.com/paper/2604.23465/) (2026)

## Abstract (arXiv)

Many real-world classification problems are significantly class-imbalanced to detriment of the class of interest. The standard set of proper evaluation metrics is well-known but the usual assumption is that the test dataset imbalance equals the real-world imbalance. In practice, this assumption is often broken for various reasons. The reported results are then often too optimistic and may lead to wrong conclusions about industrial impact and suitability of proposed techniques. We introduce methods focusing on evaluation under non-constant class imbalance. We show that not only the absolute values of commonly used metrics, but even the order of classifiers in relation to the evaluation metric used is affected by the change of the imbalance rate. Finally, we demonstrate that using subsampling in order to get a test dataset with class imbalance equal to the one observed in the wild is not necessary, and eventually can lead to significant errors in classifier's performance estimate.