# Towards Evaluation Guidelines for Empirical Studies involving LLMs

arXiv: https://arxiv.org/abs/2411.07668 · HTML: https://mlsanity.com/paper/2411.07668/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2024.

## Built on by (most cited first, 15 of 15)

- [Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies](https://mlsanity.com/paper/2510.25506/) (2025)
- [Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research](https://mlsanity.com/paper/2506.12691/) (2025)
- [Evaluating the effectiveness of LLM-based interoperability](https://mlsanity.com/paper/2510.23893/) (2025)
- [OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering](https://mlsanity.com/paper/2512.15979/) (2025)
- [Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures](https://mlsanity.com/paper/2601.03988/) (2026)
- [Brick-DICL: Dynamic In-Context Learning for Automated Brick Schema Classification](https://mlsanity.com/paper/2606.17637/) (2026)
- [ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation](https://mlsanity.com/paper/2505.17632/) (2025)
- [Empowering Computing Education Researchers Through LLM-Assisted Content Analysis](https://mlsanity.com/paper/2508.18872/) (2025)
- [LLM Analysis of 150+ years of German Parliamentary Debates on Migration Reveals Shift from Post-War Solidarity to Anti-Solidarity in the Last Decade](https://mlsanity.com/paper/2509.07274/) (2025)
- [Designing Empirical Studies on LLM-Based Code Generation: Towards a Reference Framework](https://mlsanity.com/paper/2510.03862/) (2025)
- [Taking a Pulse on How Generative AI is Reshaping the Software Engineering Research Landscape](https://mlsanity.com/paper/2604.11184/) (2026)
- [From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data](https://mlsanity.com/paper/2607.16845/) (2026)
- [Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews](https://mlsanity.com/paper/2607.24991/) (2026)
- [Antipatterns in AI-assisted Qualitative Data Analysis: A Catalog of Temptations and Pitfalls for Software Engineering Researchers](https://mlsanity.com/paper/2608.27927/) (2026)
- [After the Party: Growth, Governance, and Security Scanning in the OpenClaw Agent Skill Ecosystem](https://mlsanity.com/paper/2609.17274/) (2026)

## Abstract (arXiv)

In the short period since the release of ChatGPT, large language models (LLMs) have changed the software engineering research landscape. While there are numerous opportunities to use LLMs for supporting research or software engineering tasks, solid science needs rigorous empirical evaluations. However, so far, there are no specific guidelines for conducting and assessing studies involving LLMs in software engineering research. Our focus is on empirical studies that either use LLMs as part of the research process or studies that evaluate existing or new tools that are based on LLMs. This paper contributes the first set of holistic guidelines for such studies. Our goal is to start a discussion in the software engineering research community to reach a common understanding of our standards for high-quality empirical studies involving LLMs.