# Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

arXiv: https://arxiv.org/abs/2505.08903 · HTML: https://mlsanity.com/paper/2505.08903/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2025.

## Built on by (most cited first, 17 of 17)

- [Towards Enforcing Company Policy Adherence in Agentic Workflows](https://mlsanity.com/paper/2507.16459/) (2025)
- [Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead](https://mlsanity.com/paper/2512.00020/) (2025)
- [Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents](https://mlsanity.com/paper/2602.22764/) (2026)
- [Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents](https://mlsanity.com/paper/2603.02239/) (2026)
- [Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs](https://mlsanity.com/paper/2604.20211/) (2026)
- [Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings](https://mlsanity.com/paper/2506.03013/) (2025)
- [Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMs](https://mlsanity.com/paper/2511.02197/) (2025)
- [SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements](https://mlsanity.com/paper/2607.27409/) (2026)
- [AgentExecutor: Partial Code Execution via Agentic Context Generation](https://mlsanity.com/paper/2608.05959/) (2026)
- [DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models](https://mlsanity.com/paper/2603.23514/) (2026)
- [Toward Executable Repository-Level Code Generation via Environment Alignment](https://mlsanity.com/paper/2604.03622/) (2026)
- [An empirical study of LoRA-based fine-tuning of large language models for automated test case generation](https://mlsanity.com/paper/2604.06946/) (2026)
- [A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2605.25536/) (2026)
- [All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application](https://mlsanity.com/paper/2606.22475/) (2026)
- [Reliable and Developer-Aligned Evaluation of Agents for Software Engineering](https://mlsanity.com/paper/2607.06713/) (2026)
- [ReqGenX: An Empirical Study of Atomic Decomposition, Artifact Regeneration, and Reconstruction for Legacy SRS Documents](https://mlsanity.com/paper/2607.16564/) (2026)
- [When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answering](https://mlsanity.com/paper/2607.17063/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are gaining increasing popularity in software engineering (SE) due to their unprecedented performance across various applications. These models are increasingly being utilized for a range of SE tasks, including requirements engineering and design, code analysis and generation, software maintenance, and quality assurance. As LLMs become more integral to SE, evaluating their effectiveness is crucial for understanding their potential in this field. In recent years, substantial efforts have been made to assess LLM performance in various SE tasks, resulting in the creation of several benchmarks tailored to this purpose. This paper offers a thorough review of 291 benchmarks, addressing three main aspects: what benchmarks are available, how benchmarks are constructed, and the future outlook for these benchmarks. We begin by examining SE tasks such as requirements engineering and design, coding assistant, software testing, AIOPs, software maintenance, and quality management. We then analyze the benchmarks and their development processes, highlighting the limitations of existing benchmarks. Additionally, we discuss the successes and failures of LLMs in different software tasks and explore future opportunities and challenges for SE-related benchmarks. We aim to provide a comprehensive overview of benchmark research in SE and offer insights to support the creation of more effective evaluation tools.