# A large-scale study on research code quality and execution

arXiv: https://arxiv.org/abs/2103.12793 · HTML: https://mlsanity.com/paper/2103.12793/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2021.

## Built on by (most cited first, 22 of 22)

- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Paper2Agent: Reimagining Research Papers As Interactive and Reliable AI Agents](https://mlsanity.com/paper/2509.06917/) (2025)
- [PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)](https://mlsanity.com/paper/2409.12699/) (2024)
- [Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis](https://mlsanity.com/paper/2502.01853/) (2025)
- [When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP](https://mlsanity.com/paper/2303.16166/) (2023)
- [Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies](https://mlsanity.com/paper/2510.25506/) (2025)
- [Quantum Advantage Seeker with Kernels (QuASK): a software framework to speed up the research in quantum machine learning](https://mlsanity.com/paper/2206.15284/) (2022)
- [Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches](https://mlsanity.com/paper/2602.08561/) (2026)
- [Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks](https://mlsanity.com/paper/2603.04459/) (2026)
- [System Log Parsing with Large Language Models: A Review](https://mlsanity.com/paper/2504.04877/) (2025)
- [MLDev: Data Science Experiment Automation and Reproducibility Software](https://mlsanity.com/paper/2107.12322/) (2021)
- [LLM-Assisted Replication for Quantitative Social Science](https://mlsanity.com/paper/2602.18453/) (2026)
- [Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformatics](https://mlsanity.com/paper/2603.22018/) (2026)
- [An Agentic Approach Towards Replication Package Quality Evaluation](https://mlsanity.com/paper/2606.02006/) (2026)
- [How much can ChatGPT really help Computational Biologists in Programming?](https://mlsanity.com/paper/2309.09126/) (2023)
- [An End-to-End System for Reproducibility Assessment of Source Code Repositories via Their Readmes](https://mlsanity.com/paper/2310.09634/) (2023)
- [Reproducibility of Machine Learning-Based Fault Detection and Diagnosis for HVAC Systems in Buildings: An Empirical Study](https://mlsanity.com/paper/2508.00880/) (2025)
- [It's LIT! Reliability-Optimized LLMs with Inspectable Tools](https://mlsanity.com/paper/2511.14903/) (2025)
- [Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization](https://mlsanity.com/paper/2512.06006/) (2025)
- [Persistent Recursive Worlds Enable Autonomous Software Evolution](https://mlsanity.com/paper/2608.10450/) (2026)
- [Impact of Data Loss in Postprocessing on Training and Inference of Quantum Neural Networks](https://mlsanity.com/paper/2609.05060/) (2026)
- [K-Dense BYOK: An Open-Source AI Research Assistant That Runs Locally and Keeps a Hash-Chained Lab Notebook](https://mlsanity.com/paper/2610.00074/) (2026)

## Abstract (arXiv)

This article presents a study on the quality and execution of research code from publicly-available replication datasets at the Harvard Dataverse repository. Research code is typically created by a group of scientists and published together with academic papers to facilitate research transparency and reproducibility. For this study, we define ten questions to address aspects impacting research reproducibility and reuse. First, we retrieve and analyze more than 2000 replication datasets with over 9000 unique R files published from 2010 to 2020. Second, we execute the code in a clean runtime environment to assess its ease of reuse. Common coding errors were identified, and some of them were solved with automatic code cleaning to aid code execution. We find that 74% of R files crashed in the initial execution, while 56% crashed when code cleaning was applied, showing that many errors can be prevented with good coding practices. We also analyze the replication datasets from journals' collections and discuss the impact of the journal policy strictness on the code re-execution rate. Finally, based on our results, we propose a set of recommendations for code dissemination aimed at researchers, journals, and repositories.