# Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters

arXiv: https://arxiv.org/abs/2003.01182 · HTML: https://mlsanity.com/paper/2003.01182/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2020.

## Built on by (most cited first, 9 of 9)

- [A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers](https://mlsanity.com/paper/2207.03277/) (2022)
- [Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities](https://mlsanity.com/paper/2212.10017/) (2022)
- [Area under the ROC Curve has the Most Consistent Evaluation for Binary Classification](https://mlsanity.com/paper/2408.10193/) (2024)
- [The impact of using biased performance metrics on software defect prediction research](https://mlsanity.com/paper/2103.10201/) (2021)
- [On the Cross-Dataset Generalization of Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2402.10974/) (2024)
- [Assessing the Early Bird Heuristic (for Predicting Project Quality)](https://mlsanity.com/paper/2105.11082/) (2021)
- [On The Effectiveness of One-Class Support Vector Machine in Different Defect Prediction Scenarios](https://mlsanity.com/paper/2202.12074/) (2022)
- [CONCORD: Towards a DSL for Configurable Graph Code Representation](https://mlsanity.com/paper/2401.17967/) (2024)
- [VulValidate: Auditing Function-Level Vulnerability Labels with Executable Evidence](https://mlsanity.com/paper/2610.05103/) (2026)

## Abstract (arXiv)

Context: There is considerable diversity in the range and design of computational experiments to assess classifiers for software defect prediction. This is particularly so, regarding the choice of classifier performance metrics. Unfortunately some widely used metrics are known to be biased, in particular F1. Objective: We want to understand the extent to which the widespread use of the F1 renders empirical results in software defect prediction unreliable. Method: We searched for defect prediction studies that report both F1 and the Matthews correlation coefficient (MCC). This enabled us to determine the proportion of results that are consistent between both metrics and the proportion that change. Results: Our systematic review identifies 8 studies comprising 4017 pairwise results. Of these results, the direction of the comparison changes in 23% of the cases when the unbiased MCC metric is employed. Conclusion: We find compelling reasons why the choice of classification performance metric matters, specifically the biased and misleading F1 metric should be deprecated.