# The impact of using biased performance metrics on software defect prediction research

arXiv: https://arxiv.org/abs/2103.10201 · HTML: https://mlsanity.com/paper/2103.10201/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 90 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 8 of 8)

- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Guidelines for including grey literature and conducting multivocal literature reviews in software engineering](https://mlsanity.com/paper/1707.02553/) (2017)
- [Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters](https://mlsanity.com/paper/2003.01182/) (2020)
- [Correction of "A Comparative Study to Benchmark Cross-project Defect Prediction Approaches"](https://mlsanity.com/paper/1707.09281/) (2017)
- [Are SonarQube Rules Inducing Bugs?](https://mlsanity.com/paper/1907.00376/) (2019)
- [A Systematic Review of Unsupervised Learning Techniques for Software Defect Prediction](https://mlsanity.com/paper/1907.12027/) (2019)
- [The Prevalence of Errors in Machine Learning Experiments](https://mlsanity.com/paper/1909.04436/) (2019)
- [On the costs and profit of software defect prediction](https://mlsanity.com/paper/1911.04309/) (2019)

## Built on by (most cited first, 4 of 4)

- [A Comprehensive Study of Machine Learning Techniques for Log-Based Anomaly Detection](https://mlsanity.com/paper/2307.16714/) (2023)
- [Graph-Based Machine Learning Improves Just-in-Time Defect Prediction](https://mlsanity.com/paper/2110.05371/) (2021)
- [HEDP: A Method for Early Forecasting Software Defects based on Human Error Mechanisms](https://mlsanity.com/paper/2110.06758/) (2021)
- [Applications and Challenges of Fairness APIs in Machine Learning Software](https://mlsanity.com/paper/2508.16377/) (2025)

## Abstract (arXiv)

Context: Software engineering researchers have undertaken many experiments investigating the potential of software defect prediction algorithms. Unfortunately, some widely used performance metrics are known to be problematic, most notably F1, but nevertheless F1 is widely used. Objective: To investigate the potential impact of using F1 on the validity of this large body of research. Method: We undertook a systematic review to locate relevant experiments and then extract all pairwise comparisons of defect prediction performance using F1 and the un-biased Matthews correlation coefficient (MCC). Results: We found a total of 38 primary studies. These contain 12,471 pairs of results. Of these, 21.95% changed direction when the MCC metric is used instead of the biased F1 metric. Unfortunately, we also found evidence suggesting that F1 remains widely used in software defect prediction research. Conclusions: We reiterate the concerns of statisticians that the F1 is a problematic metric outside of an information retrieval context, since we are concerned about both classes (defect-prone and not defect-prone units). This inappropriate usage has led to a substantial number (more than one fifth) of erroneous (in terms of direction) results. Therefore we urge researchers to (i) use an unbiased metric and (ii) publish detailed results including confusion matrices such that alternative analyses become possible.