# Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering

arXiv: https://arxiv.org/abs/2604.01437 · HTML: https://mlsanity.com/paper/2604.01437/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 13 of 13)

- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios](https://mlsanity.com/paper/2503.12374/) (2025)
- [CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building](https://mlsanity.com/paper/2505.21069/) (2025)
- [A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs](https://mlsanity.com/paper/2411.07098/) (2024)
- [Are Autonomous Web Agents Good Testers?](https://mlsanity.com/paper/2504.01495/) (2025)
- [An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software](https://mlsanity.com/paper/2509.13471/) (2025)
- [An LLM-based multi-agent framework for agile effort estimation](https://mlsanity.com/paper/2509.14483/) (2025)
- [LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems](https://mlsanity.com/paper/2501.11864/) (2025)
- [Agents in the Sandbox: End-to-End Crash Bug Reproduction for Minecraft](https://mlsanity.com/paper/2503.20036/) (2025)
- [Agentic Specification Generator for Move Programs](https://mlsanity.com/paper/2509.24515/) (2025)
- [Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent](https://mlsanity.com/paper/2512.14990/) (2025)

## Built on by (most cited first, 1 of 1)

- [Reversa: A Reverse Documentation Engineering Framework for Converting Legacy Software into Operational Specifications for AI Agents](https://mlsanity.com/paper/2605.18684/) (2026)

## Abstract (arXiv)

With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design description frequently renders the reproduction of results infeasible. To synthesize current evaluation practices for Agentic AI in SE, this study analyzes 18 papers on the topic, published or accepted by ICSE 2026, ICSE 2025, FSE 2025, ASE 2025, and ISSTA 2025. The analysis identifies prevailing approaches and their limitations in evaluating Agentic AI for SE, both in current research and potential future studies. To address these shortcomings, this position paper proposes a set of guidelines and recommendations designed to empower reproducible, explainable, and effective evaluations of Agentic AI in software engineering. In particular, we recommend that Agentic AI researchers make their Thought-Action-Result (TAR) trajectories and LLM interaction data, or summarized versions of these artifacts, publicly accessible. Doing so will enable subsequent studies to more effectively analyze the strengths and weaknesses of different Agentic AI approaches. To demonstrate the feasibility of such comparisons, we present a proof-of-concept case study that illustrates how TAR trajectories can support systematic analysis across approaches.