# Balancing the Scales: A Comprehensive Study on Tackling Class Imbalance in Binary Classification

arXiv: https://arxiv.org/abs/2409.19751 · HTML: https://mlsanity.com/paper/2409.19751/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 26 references that are in the map.
- First posted on arXiv 29 Sep 2024.

## Builds on (most cited first, 3 of 3)

- [Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification](https://mlsanity.com/paper/2208.02951/) (2022)
- [To SMOTE, or not to SMOTE?](https://mlsanity.com/paper/2201.08528/) (2022)
- [Stop Oversampling for Class Imbalance Learning: A Critical Review](https://mlsanity.com/paper/2202.03579/) (2022)

## Built on by (most cited first, 6 of 6)

- [Synthetic Data Generation for Augmenting Small Samples](https://mlsanity.com/paper/2501.18741/) (2025)
- [MoL for LLMs: Dual-Loss Optimization to Enhance Domain Expertise While Preserving General Capabilities](https://mlsanity.com/paper/2505.12043/) (2025)
- [Pixel-wise Modulated Dice Loss for Medical Image Segmentation](https://mlsanity.com/paper/2506.15744/) (2025)
- [AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles](https://mlsanity.com/paper/2507.11764/) (2025)
- [An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection](https://mlsanity.com/paper/2607.22286/) (2026)
- [When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification](https://mlsanity.com/paper/2608.16147/) (2026)

## Abstract (arXiv)

Class imbalance in binary classification tasks remains a significant challenge in machine learning, often resulting in poor performance on minority classes. This study comprehensively evaluates three widely-used strategies for handling class imbalance: Synthetic Minority Over-sampling Technique (SMOTE), Class Weights tuning, and Decision Threshold Calibration. We compare these methods against a baseline scenario of no-intervention across 15 diverse machine learning models and 30 datasets from various domains, conducting a total of 9,000 experiments. Performance was primarily assessed using the F1-score, although our study also tracked results on additional 9 metrics including F2-score, precision, recall, Brier-score, PR-AUC, and AUC. Our results indicate that all three strategies generally outperform the baseline, with Decision Threshold Calibration emerging as the most consistently effective technique. However, we observed substantial variability in the best-performing method across datasets, highlighting the importance of testing multiple approaches for specific problems. This study provides valuable insights for practitioners dealing with imbalanced datasets and emphasizes the need for dataset-specific analysis in evaluating class imbalance handling techniques.