# Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents

arXiv: https://arxiv.org/abs/2511.08242 · HTML: https://mlsanity.com/paper/2511.08242/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 40 references that are in the map.
- First posted on arXiv 11 Nov 2025.

## Builds on (most cited first, 5 of 5)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites](https://mlsanity.com/paper/2504.11543/) (2025)
- [STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarking](https://mlsanity.com/paper/2507.03674/) (2025)
- [A2Perf: Real-World Autonomous Agents Benchmark](https://mlsanity.com/paper/2503.03056/) (2025)

## Built on by (most cited first, 2 of 2)

- [ProActor: Timing-Aware Reinforcement Learning for Proactive Task Scheduling Agents](https://mlsanity.com/paper/2605.24900/) (2026)
- [Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks](https://mlsanity.com/paper/2609.11018/) (2026)

## Abstract (arXiv)

As AI agents proliferate across industries and applications, evaluating their performance based solely on infrastructural metrics such as latency, time-to-first-token, or token throughput is proving insufficient. These metrics fail to capture the quality of an agent's decisions, its operational autonomy, or its ultimate business value. This white paper proposes a novel, comprehensive framework of eleven outcome-based, task-agnostic performance metrics for AI agents that transcend domain boundaries. These metrics are designed to enable organizations to evaluate agents based on the quality of their decisions, their degree of autonomy, their adaptability to new challenges, and the tangible business value they deliver, regardless of the underlying model architecture or specific use case. We introduce metrics such as Goal Completion Rate (GCR), Autonomy Index (AIx), Multi-Step Task Resilience (MTR), and Business Impact Efficiency (BIE). Through a large-scale simulated experiment involving four distinct agent architectures (ReAct, Chain-of-Thought, Tool-Augmented, Hybrid) across five diverse domains (Healthcare, Finance, Marketing, Legal, and Customer Service), we demonstrate the framework's efficacy. Our results reveal significant performance trade-offs between different agent designs, highlighting the Hybrid Agent as the most consistently high-performing model across the majority of our proposed metrics, achieving an average Goal Completion Rate of 88.8% and the highest Return on Investment (ROI). This work provides a robust, standardized methodology for the holistic evaluation of AI agents, paving the way for more effective development, deployment, and governance.