# A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering

arXiv: https://arxiv.org/abs/2501.12728 · HTML: https://mlsanity.com/paper/2501.12728/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 45 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Prompted LLMs as Chatbot Modules for Long Open-domain Conversation](https://mlsanity.com/paper/2305.04533/) (2023)
- [Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology](https://mlsanity.com/paper/1809.03006/) (2018)
- [An Extensive Comparison of Static Application Security Testing Tools](https://mlsanity.com/paper/2403.09219/) (2024)
- [Leveraging Large Language Models for Preliminary Security Risk Analysis: A Mission-Critical Case Study](https://mlsanity.com/paper/2403.15756/) (2024)
- [On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?](https://mlsanity.com/paper/2412.11698/) (2024)
- [The Evolution of Empirical Methods in Software Engineering](https://mlsanity.com/paper/1912.11512/) (2019)
- [Lessons Learnt in Conducting Survey Research](https://mlsanity.com/paper/1702.05744/) (2017)
- [Pitfalls in Experiments with DNN4SE: An Analysis of the State of the Practice](https://mlsanity.com/paper/2305.11556/) (2023)

## Built on by (most cited first, 2 of 2)

- [What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects](https://mlsanity.com/paper/2509.07763/) (2025)
- [The Invisible Hand of AI Libraries Shaping Open Source Projects and Communities](https://mlsanity.com/paper/2601.01944/) (2026)

## Abstract (arXiv)

Context: Empirical Software Engineering (ESE) drives innovation in SE through qualitative and quantitative studies. However, concerns about the correct application of empirical methodologies have existed since the 2006 Dagstuhl seminar on SE. Objective: To analyze three decades of SE research, identify mistakes in statistical methods, and evaluate experts' ability to detect and address these issues. Methods: We conducted a literature survey of ~27,000 empirical studies, using LLMs to classify statistical methodologies as adequate or inadequate. Additionally, we selected 30 primary studies and held a workshop with 33 ESE experts to assess their ability to identify and resolve statistical issues. Results: Significant statistical issues were found in the primary studies, and experts showed limited ability to detect and correct these methodological problems, raising concerns about the broader ESE community's proficiency in this area. Conclusions. Despite our study's eventual limitations, its results shed light on recurring issues from promoting information copy-and-paste from past authors' works and the continuous publication of inadequate approaches that promote dubious results and jeopardize the spread of the correct statistical strategies among researchers. Besides, it justifies further investigation into empirical rigor in software engineering to expose these recurring issues and establish a framework for reassessing our field's foundation of statistical methodology application. Therefore, this work calls for critically rethinking and reforming data analysis in empirical software engineering, paving the way for our work soon.