# When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

arXiv: https://arxiv.org/abs/2608.16147 · HTML: https://mlsanity.com/paper/2608.16147/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 15 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Builds on (most cited first, 5 of 5)

- [Balancing the Scales: A Comprehensive Study on Tackling Class Imbalance in Binary Classification](https://mlsanity.com/paper/2409.19751/) (2024)
- [Heterogeneous Graph Auto-Encoder for CreditCard Fraud Detection](https://mlsanity.com/paper/2410.08121/) (2024)
- [Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection](https://mlsanity.com/paper/2412.07437/) (2024)
- [Data Leakage and Deceptive Performance: A Critical Examination of Credit Card Fraud Detection Methodologies](https://mlsanity.com/paper/2506.02703/) (2025)
- [Improving Credit Card Fraud Detection through Transformer-Enhanced GAN Oversampling](https://mlsanity.com/paper/2509.19032/) (2025)

## Abstract (arXiv)

Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.