Reliable and Developer-Aligned Evaluation of Agents for Software Engineering
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Large language models are rapidly moving towards closing the development cycle, transitioning from simple assistive companions to autonomous contributors deeply embedded into collaborative development environments. Despite their accelerated adoption, existing evaluation techniques are limited due to their fragmented nature and distorted projection of true model capabilities, often obtained from hypothetical syntactic scenarios. This research aims to bridge this gap by providing a comprehensive evaluation methodology for LLM-powered agents that is grounded in real-world software development practice. Our evaluation approach focuses on contamination-awareness, in-the-wild agentic behavior assessment, and trajectory-aware benchmarks and metrics capturing realistic coding contexts, human-aligned behavior, and model failure modes.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 7 of its 13 references that are in the map. First posted on arXiv 7 Jul 2026.
Builds on
7 of 13 references.
Closest ancestors · 1
Same direction, newest first
Other references · 6
- 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 93
- 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
- 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
- 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62
- 2025Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Taskscited by 17
- 2025A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metricscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
- 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
- 2025LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
- 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
- 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture
- 2025From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering
- 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
- 2026LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities