# Designing Empirical Studies on LLM-Based Code Generation: Towards a Reference Framework

arXiv: https://arxiv.org/abs/2510.03862 · HTML: https://mlsanity.com/paper/2510.03862/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Builds on (most cited first, 9 of 9)

- [Bias Testing and Mitigation in LLM-based Code Generation](https://mlsanity.com/paper/2309.14345/) (2023)
- [On the Effectiveness of Large Language Models in Domain-Specific Code Generation](https://mlsanity.com/paper/2312.01639/) (2023)
- [SALLM: Security Assessment of Generated Code](https://mlsanity.com/paper/2311.00889/) (2023)
- [Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study](https://mlsanity.com/paper/2310.02059/) (2023)
- [RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code](https://mlsanity.com/paper/2409.15154/) (2024)
- [From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining](https://mlsanity.com/paper/2309.15606/) (2023)
- [Towards Evaluation Guidelines for Empirical Studies involving LLMs](https://mlsanity.com/paper/2411.07668/) (2024)
- [Analyzing Prominent LLMs: An Empirical Study of Performance and Complexity in Solving LeetCode Problems](https://mlsanity.com/paper/2508.03931/) (2025)
- [GPT in Data Science: A Practical Exploration of Model Selection](https://mlsanity.com/paper/2311.11516/) (2023)

## Abstract (arXiv)

The rise of large language models (LLMs) has introduced transformative potential in automated code generation, addressing a wide range of software engineering challenges. However, empirical evaluation of LLM-based code generation lacks standardization, with studies varying widely in goals, tasks, and metrics, which limits comparability and reproducibility. In this paper, we propose a theoretical framework for designing and reporting empirical studies on LLM-based code generation. The framework is grounded in both our prior experience conducting such experiments and a comparative analysis of key similarities and differences among recent studies. It organizes evaluation around core components such as problem sources, quality attributes, and metrics, supporting structured and systematic experimentation. We demonstrate its applicability through representative case mappings and identify opportunities for refinement. Looking forward, we plan to evolve the framework into a more robust and mature tool for standardizing LLM evaluation across software engineering contexts.