# SoftMCC: An MCC-Brier Calibration Bridge for Threshold-Free Model Selection under Class Imbalance

arXiv: https://arxiv.org/abs/2608.08984 · HTML: https://mlsanity.com/paper/2608.08984/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 71 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 11 of 11)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning](https://mlsanity.com/paper/1609.06570/) (2016)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Appropriateness of Performance Indices for Imbalanced Data Classification: An Analysis](https://mlsanity.com/paper/2008.11752/) (2020)
- [Matthews Correlation Coefficient Loss for Deep Convolutional Networks: Application to Skin Lesion Segmentation](https://mlsanity.com/paper/2010.13454/) (2020)
- [Bridging the Gap: Unifying the Training and Evaluation of Neural Network Binary Classifiers](https://mlsanity.com/paper/2009.01367/) (2020)
- [AnyLoss: Transforming Classification Metrics into Loss Functions](https://mlsanity.com/paper/2405.14745/) (2024)

## Abstract (arXiv)

Model selection for imbalanced binary classification often uses the Matthews correlation coefficient (MCC), but thresholding makes validation rankings threshold-dependent. SoftMCC is a post-training MCC validation framework on established probability-valued confusion counts, coupling an MCC-specific calibrated identity with a tie-aware, shared-pool selection protocol. Its core score is a covariance-normalized probability-label association, reduces exactly to MCC for hard predictions, and is Pearson-bounded. Under perfect population calibration it equals the Brier skill score with identical candidate ordering; outside that regime the gap does not identify calibration error. Across 18 settings with 12 duplicate-safe grouped repeats, SoftMCC attains the best stability mean rank (2.31) and highest mean tie-corrected Kendall's W (0.659), with a significant Friedman test (p=0.007); Nemenyi analysis separates it from AUPRC and MCC@0.5, while 14-source-family sensitivity retains only the latter. Selected-model utility shows no advantage. Three of six prespecified comparisons have negative mean test-MCC differences, only F1@best survives Holm correction (p=0.014), and the dataset-level test is not significant (p=0.117). Label permutation lowers mean W to 0.092; temperature scaling shifts SoftMCC rankings (mean Spearman 0.851) whereas rank-based and threshold-optimized metrics remain invariant. SoftMCC is a calibration-sensitive MCC-family selector with bounded stability and utility evidence.