# What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment

arXiv: https://arxiv.org/abs/2510.08847 · HTML: https://mlsanity.com/paper/2510.08847/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 58 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 16 of 16)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023)
- [Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2503.09572/) (2025)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories](https://mlsanity.com/paper/2504.08942/) (2025)
- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [PlanGenLLMs: A Modern Survey of LLM Planning Capabilities](https://mlsanity.com/paper/2502.11221/) (2025)
- [Evaluating Step-by-step Reasoning Traces: A Survey](https://mlsanity.com/paper/2502.12289/) (2025)
- [Get Experience from Practice: LLM Agents with Record & Replay](https://mlsanity.com/paper/2505.17716/) (2025)
- [Plancraft: an evaluation dataset for planning with LLM agents](https://mlsanity.com/paper/2412.21033/) (2024)
- [Evaluating the Goal-Directedness of Large Language Models](https://mlsanity.com/paper/2504.11844/) (2025)

## Built on by (most cited first, 8 of 8)

- [From Plan to Action: How Well Do Agents Follow the Plan?](https://mlsanity.com/paper/2604.12147/) (2026)
- [Fantastic Adaptive Taxonomies and How to Use Them](https://mlsanity.com/paper/2607.16387/) (2026)
- [Plover: Steering GUI Agents through Plan-Centric Interaction](https://mlsanity.com/paper/2607.15193/) (2026)
- [Agent Seer: Synthesizing Scenarios from Specification Understanding](https://mlsanity.com/paper/2608.26133/) (2026)
- [ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance](https://mlsanity.com/paper/2609.09458/) (2026)
- [Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution](https://mlsanity.com/paper/2609.38108/) (2026)
- [Consistent Plan-Act for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2609.38891/) (2026)
- [SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents](https://mlsanity.com/paper/2610.13157/) (2026)

## Abstract (arXiv)

We introduce the Agent GPA (Goal-Plan-Action) framework, driven by the fundamental insight that critical agent failures emerge at the intersections of setting goals, devising plans, and executing actions. We operationalize the framework with a factorized suite of LLM judges designed to measure distinct elements of Goal-Plan-Act alignment. To make this methodology scalable and generalizable across diverse agent architectures and datasets, we use state-of-the-art automated prompt optimization techniques to systematically generate domain-specific evaluation criteria. We validate this approach across three benchmarks: a multi-agent research setting (TRAIL/GAIA), a single coding agent setting (TRAIL/SWE-bench), and a private, enterprise data-agent setting (Snowflake Intelligence). Extensive evaluation on TRAIL/GAIA demonstrates the core validity of the framework, which identifies a broad range of agent failures (95% of human-annotated errors), localizes errors to enable targeted debugging (86% of human-annotated errors), and exhibits strong agreement with human evaluators. Crucially, by applying our automated methodology to both public datasets, we demonstrate that our GPA judges generally achieve the highest error coverage (ranging from 76% to 86%) in comparison to manual prompting approaches. We also leverage an evolutionary coding agent to improve judge consistency by up to 38% through iterative refinement of evaluation rubrics. Overall, Agent GPA provides a rigorous and generalizable paradigm for targeted agent evaluation.