# Guidelines for Empirical Studies in Software Engineering involving Large Language Models

arXiv: https://arxiv.org/abs/2508.15503 · HTML: https://mlsanity.com/paper/2508.15503/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2025.

## Built on by (most cited first, 21 of 21)

- [Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications](https://mlsanity.com/paper/2511.19933/) (2025)
- [Agentic Business Process Management: A Research Manifesto](https://mlsanity.com/paper/2603.18916/) (2026)
- [EdgeFlow: Edge-Map Augmented VLM-Based Flowchart Processing for Industrial Requirements Engineering](https://mlsanity.com/paper/2605.27332/) (2026)
- [Do programming languages still matter to your AI coding agent teammate? Evidence at scale from chess engines](https://mlsanity.com/paper/2606.13763/) (2026)
- [MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models](https://mlsanity.com/paper/2606.24815/) (2026)
- [Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations](https://mlsanity.com/paper/2607.09172/) (2026)
- [Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report](https://mlsanity.com/paper/2608.11965/) (2026)
- [The Software Engineering Simulations Lab: Agentic AI for RE Quality Simulations](https://mlsanity.com/paper/2511.17762/) (2025)
- [Instruction-Tuning Open-Weight Language Models for BPMN Model Generation](https://mlsanity.com/paper/2512.12063/) (2025)
- [3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse](https://mlsanity.com/paper/2607.07980/) (2026)
- [Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews](https://mlsanity.com/paper/2607.24991/) (2026)
- [Can LLMs Test Terminal User Interfaces?](https://mlsanity.com/paper/2608.03743/) (2026)
- [Detecting Soft Skills in ML Engineering Roles CVs](https://mlsanity.com/paper/2608.10046/) (2026)
- [Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation](https://mlsanity.com/paper/2608.11513/) (2026)
- [Antipatterns in AI-assisted Qualitative Data Analysis: A Catalog of Temptations and Pitfalls for Software Engineering Researchers](https://mlsanity.com/paper/2608.27927/) (2026)
- [Model Retirement Creates Reproducibility Risk in Biomedical AI Publications](https://mlsanity.com/paper/2609.04699/) (2026)
- [When Who You Are Can Change the Code You Get: A Study of Persona-Induced Bias in LLM Code Generation](https://mlsanity.com/paper/2609.22102/) (2026)
- [How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure](https://mlsanity.com/paper/2609.30074/) (2026)
- [A Framework for the Systematic Review of ML Assets in AI Registries](https://mlsanity.com/paper/2610.09551/) (2026)
- [One Skill Too Many: How Co-Installed Skills Conflict in Coding Agents](https://mlsanity.com/paper/2610.11647/) (2026)
- [DTScribe: Generating Models for Digital Twin Reporting using Local Large Language Models](https://mlsanity.com/paper/2610.18635/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommendations (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the system and prompt design beyond the model; (4) report session traces, i.e., interaction logs and runtime traces; (5) use suitable baselines, benchmarks, and metrics; (6) include an open LLM as a baseline; (7) validate LLM outputs against human judgment; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines.org).