# Replication Can Improve Prior Results: A GitHub Study of Pull Request Acceptance

arXiv: https://arxiv.org/abs/1902.04060 · HTML: https://mlsanity.com/paper/1902.04060/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2019.

## Built on by (most cited first, 4 of 4)

- [FRUGAL: Unlocking SSL for Software Analytics](https://mlsanity.com/paper/2108.09847/) (2021)
- [DebtFree: Minimizing Labeling Cost in Self-Admitted Technical Debt Identification using Semi-Supervised Learning](https://mlsanity.com/paper/2201.10592/) (2022)
- [Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset](https://mlsanity.com/paper/2606.13468/) (2026)
- [Less, but Stronger: On the Value of Strong Heuristics in Semi-supervised Learning for Software Analytics](https://mlsanity.com/paper/2302.01997/) (2023)

## Abstract (arXiv)

Crowdsourcing and data mining can be used to effectively reduce the effort associated with the partial replication and enhancement of qualitative studies. For example, in a primary study, other researchers explored factors influencing the fate of GitHub pull requests using an extensive qualitative analysis of 20 pull requests. Guided by their findings, we mapped some of their qualitative insights onto quantitative questions. To determine how well their findings generalize, we collected much more data (170 additional pull requests from 142 GitHub projects). Using crowdsourcing, that data was augmented with subjective qualitative human opinions about how pull requests extended the original issue. The crowd's answers were then combined with quantitative features and, using data mining, used to build a predictor for whether code would be merged. That predictor was far more accurate that one built from the primary study's qualitative factors (F1=90 vs 68%), illustrating the value of a mixed-methods approach and replication to improve prior results. To test the generality of this approach, the next step in future work is to conduct other studies that extend qualitative studies with crowdsourcing and data mining.