# CORE: Full-Path Evaluation of LLM Agents Beyond Final State

arXiv: https://arxiv.org/abs/2509.20998 · HTML: https://mlsanity.com/paper/2509.20998/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 22 of 22)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [MolmoAct: Action Reasoning Models that can Reason in Space](https://mlsanity.com/paper/2508.07917/) (2025)
- [BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments](https://mlsanity.com/paper/2108.03332/) (2021)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation](https://mlsanity.com/paper/2402.08191/) (2024)
- [Evaluating the World Model Implicit in a Generative Model](https://mlsanity.com/paper/2406.03689/) (2024)
- [Learning to Model the World with Language](https://mlsanity.com/paper/2308.01399/) (2023)
- [WorldGym: World Model as An Environment for Policy Evaluation](https://mlsanity.com/paper/2506.00613/) (2025)
- [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](https://mlsanity.com/paper/2506.18123/) (2025)
- [AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World](https://mlsanity.com/paper/2503.24278/) (2025)
- [GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots](https://mlsanity.com/paper/2404.15500/) (2024)
- [Multi-Agent Geospatial Copilots for Remote Sensing Workflows](https://mlsanity.com/paper/2501.16254/) (2025)
- [Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge](https://mlsanity.com/paper/2407.06939/) (2024)

## Built on by (most cited first, 9 of 9)

- [Agentic AI in Remote Sensing: Foundations, Taxonomy, and Emerging Systems](https://mlsanity.com/paper/2601.01891/) (2026)
- [Persuasion Propagation: Does Persuasion Change AI Agent Behavior?](https://mlsanity.com/paper/2602.00851/) (2026)
- [Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems](https://mlsanity.com/paper/2606.07805/) (2026)
- [GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence](https://mlsanity.com/paper/2606.17246/) (2026)
- [ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration](https://mlsanity.com/paper/2608.08605/) (2026)
- [Testing and Evaluation of Agentic AI Systems In Military Command and Control](https://mlsanity.com/paper/2608.20597/) (2026)
- [SimCRAFT: Distilling Remote Sensing Agents via Synthetic Trajectories and Contextual Retrieval-Augmented Fine-Tuning](https://mlsanity.com/paper/2608.30277/) (2026)
- [Deploying and Evaluating a Smart-Agriculture Agentic Engine for Full-Season Soybean Farm Operations](https://mlsanity.com/paper/2609.00106/) (2026)
- [AgentStripes: An Evidence-Graded Maturity Model for AI Agents](https://mlsanity.com/paper/2610.18603/) (2026)

## Abstract (arXiv)

Evaluating AI agents that solve real-world tasks through function-call sequences remains an open challenge. Existing agentic benchmarks often reduce evaluation to a binary judgment of the final state, overlooking critical aspects such as safety, efficiency, and intermediate correctness. We propose a framework based on deterministic finite automata (DFAs) that encodes tasks as sets of valid tool-use paths, enabling principled assessment of agent behavior in diverse world models. Building on this foundation, we introduce CORE, a suite of five metrics, namely Path Correctness, Path Correctness - Kendall's tau Composite, Prefix Criticality, Harmful-Call Rate, and Efficiency, that quantify alignment with expected execution patterns. Across diverse worlds, our method reveals important performance differences between agents that would otherwise appear equivalent under traditional final-state evaluation schemes.