# Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies

arXiv: https://arxiv.org/abs/2510.25506 · HTML: https://mlsanity.com/paper/2510.25506/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 80 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 30 of 36)

- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [To Believe or Not to Believe Your LLM](https://mlsanity.com/paper/2406.02543/) (2024)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)](https://mlsanity.com/paper/2304.06815/) (2023)
- [Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning](https://mlsanity.com/paper/2304.11384/) (2023)
- [Towards an Understanding of Large Language Models in Software Engineering Tasks](https://mlsanity.com/paper/2308.11396/) (2023)
- [A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice](https://mlsanity.com/paper/2410.15326/) (2024)
- [Large Language Model for Vulnerability Detection: Emerging Results and Future Directions](https://mlsanity.com/paper/2401.15468/) (2024)
- [Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores](https://mlsanity.com/paper/2410.03492/) (2024)
- [(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs](https://mlsanity.com/paper/2311.11123/) (2023)
- [A large-scale study on research code quality and execution](https://mlsanity.com/paper/2103.12793/) (2021)
- [Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors](https://mlsanity.com/paper/2306.01394/) (2023)
- [The Fundamental Principles of Reproducibility](https://mlsanity.com/paper/2011.10098/) (2020)
- [On the Replicability and Reproducibility of Deep Learning in Software Engineering](https://mlsanity.com/paper/2006.14244/) (2020)
- [Exploring ChatGPT for Toxicity Detection in GitHub](https://mlsanity.com/paper/2312.13105/) (2023)
- [Towards Evaluation Guidelines for Empirical Studies involving LLMs](https://mlsanity.com/paper/2411.07668/) (2024)
- [PyTy: Repairing Static Type Errors in Python](https://mlsanity.com/paper/2401.06619/) (2024)
- [A Reproducibility and Generalizability Study of Large Language Models for Query Generation](https://mlsanity.com/paper/2411.14914/) (2024)
- [A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutions](https://mlsanity.com/paper/2409.20303/) (2024)
- [Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPT](https://mlsanity.com/paper/2502.17450/) (2025)
- [Semantic Sleuth: Identifying Ponzi Contracts via Large Language Models](https://mlsanity.com/paper/2411.07498/) (2024)
- [Paving the Way for Mature Secondary Research: The Seven Types of Literature Review](https://mlsanity.com/paper/2205.01163/) (2022)
- [CrashTranslator: Automatically Reproducing Mobile Application Crashes Directly from Stack Trace](https://mlsanity.com/paper/2310.07128/) (2023)
- [Measuring Determinism in Large Language Models for Software Code Review](https://mlsanity.com/paper/2502.20747/) (2025)

## Built on by (most cited first, 12 of 12)

- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models](https://mlsanity.com/paper/2606.04739/) (2026)
- [Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study](https://mlsanity.com/paper/2607.02436/) (2026)
- [Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs](https://mlsanity.com/paper/2604.27006/) (2026)
- [How Much Static Structure Do Code Agents Need? A Study of Deterministic Anchoring](https://mlsanity.com/paper/2606.26979/) (2026)
- [Large Databases Need Small, Open-Weight Language Models](https://mlsanity.com/paper/2606.31808/) (2026)
- [DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making](https://mlsanity.com/paper/2607.20491/) (2026)
- [Agent-Based Software Artifact Evaluation](https://mlsanity.com/paper/2602.02235/) (2026)
- [Referential Security as a New Paradigm for AI Evaluations](https://mlsanity.com/paper/2605.25673/) (2026)
- [From Backlog Items to Security Guidance: Towards Continuous Security Compliance](https://mlsanity.com/paper/2607.27374/) (2026)
- [Model Retirement Creates Reproducibility Risk in Biomedical AI Publications](https://mlsanity.com/paper/2609.04699/) (2026)
- [How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure](https://mlsanity.com/paper/2609.30074/) (2026)

## Abstract (arXiv)

Large Language Models have gained remarkable interest in industry and academia. The increasing interest in LLMs in academia is also reflected in the number of publications on this topic over the last years. For instance, alone 78 of the around 425 publications at ICSE 2024 performed experiments with LLMs. Conducting empirical studies with LLMs remains challenging and raises questions on how to achieve reproducible results, for both researchers and practitioners. One important step towards excelling in empirical research on LLM and their application is to first understand to what extent current research results are eventually reproducible and what factors may impede reproducibility. This investigation is within the scope of our work. We contribute an analysis of the reproducibility of LLM-centric studies, provide insights into the factors impeding reproducibility, and discuss suggestions on how to improve the current state. In particular, we studied the 85 articles describing LLM-centric studies, published at ICSE 2024 and ASE 2024. Of the 85 articles, 18 provided research artefacts and used OpenAI models. We attempted to replicate those 18 studies. Of the 18 studies, only five were sufficiently complete and executable. For none of the five studies, we were able to fully reproduce the results. Two studies seemed to be partially reproducible, and three studies did not seem to be reproducible. Our results highlight not only the need for stricter research artefact evaluations but also for more robust study designs to ensure the reproducible value of future publications.