# Revisiting Process versus Product Metrics: a Large Scale Analysis

arXiv: https://arxiv.org/abs/2008.09569 · HTML: https://mlsanity.com/paper/2008.09569/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2020.

## Built on by (most cited first, 2 of 2)

- [FRUGAL: Unlocking SSL for Software Analytics](https://mlsanity.com/paper/2108.09847/) (2021)
- [When Less is More: On the Value of "Co-training" for Semi-Supervised Software Defect Predictors](https://mlsanity.com/paper/2211.05920/) (2022)

## Abstract (arXiv)

Numerous methods can build predictive models from software data. However, what methods and conclusions should we endorse as we move from analytics in-the-small (dealing with a handful of projects) to analytics in-the-large (dealing with hundreds of projects)? To answer this question, we recheck prior small-scale results (about process versus product metrics for defect prediction and the granularity of metrics) using 722,471 commits from 700 Github projects. We find that some analytics in-the-small conclusions still hold when scaling up to analytics in-the-large. For example, like prior work, we see that process metrics are better predictors for defects than product metrics (best process/product-based learners respectively achieve recalls of 98%/44% and AUCs of 95%/54%, median values). That said, we warn that it is unwise to trust metric importance results from analytics in-the-small studies since those change dramatically when moving to analytics in-the-large. Also, when reasoning in-the-large about hundreds of projects, it is better to use predictions from multiple models (since single model predictions can become confused and exhibit a high variance).