# EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

arXiv: https://arxiv.org/abs/2609.02783 · HTML: https://mlsanity.com/paper/2609.02783/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 38 of its 44 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 38)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [Anchor Points: Benchmarking Models with Much Fewer Examples](https://mlsanity.com/paper/2309.08638/) (2023)
- [The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution](https://mlsanity.com/paper/2510.25726/) (2025)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Commit0: Library Generation from Scratch](https://mlsanity.com/paper/2412.01769/) (2024)
- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [LongCodeZip: Compress Long Context for Code Language Models](https://mlsanity.com/paper/2510.00446/) (2025)
- [SWE-QA: Can Language Models Answer Repository-level Code Questions?](https://mlsanity.com/paper/2509.14635/) (2025)
- [Fluid Language Model Benchmarking](https://mlsanity.com/paper/2509.11106/) (2025)
- [Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks](https://mlsanity.com/paper/2511.04689/) (2025)
- [Efficient Evaluation of LLM Performance with Statistical Guarantees](https://mlsanity.com/paper/2601.20251/) (2026)
- [LastingBench: Defend Benchmarks Against Knowledge Leakage](https://mlsanity.com/paper/2506.21614/) (2025)
- [Stop-RAG: Value-Based Retrieval Control for Iterative RAG](https://mlsanity.com/paper/2510.14337/) (2025)
- [Inference-Time Budget Control for LLM Search Agents](https://mlsanity.com/paper/2605.05701/) (2026)
- [BAGEN: Are LLM Agents Budget-Aware?](https://mlsanity.com/paper/2606.00198/) (2026)
- [Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments](https://mlsanity.com/paper/2505.17616/) (2025)
- [Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings](https://mlsanity.com/paper/2510.26384/) (2025)
- [Agent psychometrics: Task-level performance prediction in agentic coding benchmarks](https://mlsanity.com/paper/2604.00594/) (2026)
- [Efficient Benchmarking of AI Agents](https://mlsanity.com/paper/2603.23749/) (2026)
- [CaRT: Teaching LLM Agents to Know When They Know Enough](https://mlsanity.com/paper/2510.08517/) (2025)
- [AgentStop: Terminating Local AI Agents Early to Save Energy in Consumer Devices](https://mlsanity.com/paper/2605.15206/) (2026)
- [Submodular Benchmark Selection](https://mlsanity.com/paper/2605.02209/) (2026)

## Built on by (most cited first, 3 of 3)

- [DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agents](https://mlsanity.com/paper/2609.14637/) (2026)
- [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026)
- [Disentangling Task Difficulty from Run-Level Failure in Agent Failure Prediction](https://mlsanity.com/paper/2610.05572/) (2026)

## Abstract (arXiv)

Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.