# Evaluating prediction systems in software project estimation

arXiv: https://arxiv.org/abs/2101.05426 · HTML: https://mlsanity.com/paper/2101.05426/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2021.

## Built on by (most cited first, 6 of 6)

- [Agile Effort Estimation: Have We Solved the Problem Yet? Insights From A Replication Study](https://mlsanity.com/paper/2201.05401/) (2022)
- [Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health](https://mlsanity.com/paper/2301.06577/) (2023)
- [Old but Gold: Reconsidering the value of feedforward learners for software analytics](https://mlsanity.com/paper/2101.06319/) (2021)
- [On The Effectiveness of One-Class Support Vector Machine in Different Defect Prediction Scenarios](https://mlsanity.com/paper/2202.12074/) (2022)
- [Do Not Take It for Granted: Comparing Open-Source Libraries for Software Development Effort Estimation](https://mlsanity.com/paper/2207.01705/) (2022)
- [Trajectory-Aware Benchmark Subset Selection for Cost-Efficient Software Engineering Agent Regression Testing](https://mlsanity.com/paper/2609.24928/) (2026)

## Abstract (arXiv)

Context: Software engineering has a problem in that when we empirically evaluate competing prediction systems we obtain conflicting results. Objective: To reduce the inconsistency amongst validation study results and provide a more formal foundation to interpret results with a particular focus on continuous prediction systems. Method: A new framework is proposed for evaluating competing prediction systems based upon (1) an unbiased statistic, Standardised Accuracy, (2) testing the result likelihood relative to the baseline technique of random 'predictions', that is guessing, and (3) calculation of effect sizes. Results: Previously published empirical evaluations of prediction systems are re-examined and the original conclusions shown to be unsafe. Additionally, even the strongest results are shown to have no more than a medium effect size relative to random guessing. Conclusions: Biased accuracy statistics such as MMRE are deprecated. By contrast this new empirical validation framework leads to meaningful results. Such steps will assist in performing future meta-analyses and in providing more robust and usable recommendations to practitioners.