# AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking

arXiv: https://arxiv.org/abs/2604.23581 · HTML: https://mlsanity.com/paper/2604.23581/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 26 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 14 of 14)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [Where LLM Agents Fail and How They can Learn From Failures](https://mlsanity.com/paper/2509.25370/) (2025)

## Built on by (most cited first, 9 of 9)

- [From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws](https://mlsanity.com/paper/2606.06324/) (2026)
- [Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions](https://mlsanity.com/paper/2607.21635/) (2026)
- [Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges](https://mlsanity.com/paper/2608.12097/) (2026)
- [Terminal Agents: A Survey of AI Agents in Command-Line Environments](https://mlsanity.com/paper/2608.20485/) (2026)
- [Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning](https://mlsanity.com/paper/2609.38147/) (2026)
- [ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents](https://mlsanity.com/paper/2608.30685/) (2026)
- [Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts](https://mlsanity.com/paper/2609.38670/) (2026)
- [Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills](https://mlsanity.com/paper/2610.01833/) (2026)
- [Lightweight, Rubric-Guided Trajectory Evaluation for Production AI Agents](https://mlsanity.com/paper/2610.03315/) (2026)

## Abstract (arXiv)

Agentic systems that chain reasoning, tool use, and synthesis into multi-step workflows are entering production, yet prevailing evaluation practices like end-to-end outcome checks and ad-hoc trace inspection systematically mask the intermediate failures that dominate real-world error budgets. We present AgentEval, a framework that formalizes agent executions as evaluation directed acyclic graphs (DAGs), where each node carries typed quality metrics assessed by a calibrated LLM judge (GPT-4o), classified through a hierarchical failure taxonomy (3 levels, 21 subcategories), and linked to upstream dependencies for automated root cause attribution. An ablation study isolates the impact of DAG-based dependency modeling: it alone contributes +22 percentage points to failure detection recall and +34 pp to root cause accuracy over flat step-level evaluation with identical judges and rubrics. Across three production workflows (450 test cases, two agent model families, predominantly sequential architectures with a 12% non-DAG trace rate), AgentEval achieves 2.17x higher failure detection recall than end-to-end evaluation (0.89 vs. 0.41), Cohen's kappa = 0.84 agreement with human experts, and 72% root cause accuracy against an 81% human ceiling. Cross-system evaluation on tau-bench and SWE-bench traces confirms transferability (failure detection recall >= 0.78) without taxonomy or rubric modification. A 4-month pilot with 18 engineers detected 23 pre-release regressions through CI/CD-integrated regression testing, reducing median root-cause identification time from 4.2 hours to 22 minutes and driving measurable failure rate reductions in two workflows.