# Agents' Last Exam

arXiv: https://arxiv.org/abs/2606.05405 · HTML: https://mlsanity.com/paper/2606.05405/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 50 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Video Processing from Electro-optical Sensors for Object Detection and Tracking in Maritime Environment: A Survey](https://mlsanity.com/paper/1611.05842/) (2016)
- [Gym-Anything: Turn any Software into an Agent Environment](https://mlsanity.com/paper/2604.06126/) (2026)
- [Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways](https://mlsanity.com/paper/2302.11283/) (2023)
- [OneMillion-Bench: How Far are Language Agents from Human Experts?](https://mlsanity.com/paper/2603.07980/) (2026)

## Built on by (most cited first, 30 of 31)

- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization](https://mlsanity.com/paper/2606.19930/) (2026)
- [EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?](https://mlsanity.com/paper/2609.04280/) (2026)
- [Locating Hidden Failures Makes Long-Horizon Agents More Reliable](https://mlsanity.com/paper/2609.17930/) (2026)
- [CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR](https://mlsanity.com/paper/2608.07796/) (2026)
- [What is Missing from AI Post-Training AI: An Empirical Analysis](https://mlsanity.com/paper/2608.19072/) (2026)
- [BrainPilot: Automating Brain Discovery with Agentic Research](https://mlsanity.com/paper/2607.15079/) (2026)
- [FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverables](https://mlsanity.com/paper/2608.04077/) (2026)
- [Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence](https://mlsanity.com/paper/2608.11341/) (2026)
- [StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows](https://mlsanity.com/paper/2608.17800/) (2026)
- [FrontierChallenge: Evaluating Scientific Workflow Completion](https://mlsanity.com/paper/2608.24979/) (2026)
- [CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design](https://mlsanity.com/paper/2609.16251/) (2026)
- [CheatBench: Measuring Reward Gaming in AI Agents](https://mlsanity.com/paper/2609.36308/) (2026)
- [Finding the Right Fit: Model-Harness Interactions across Agent Tasks](https://mlsanity.com/paper/2610.00917/) (2026)
- [MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement](https://mlsanity.com/paper/2610.11959/) (2026)
- [READY or Not: Reliable Enterprise Agent Deployment](https://mlsanity.com/paper/2609.02095/) (2026)
- [xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems](https://mlsanity.com/paper/2609.07784/) (2026)
- [Finishing the Task Is Not Enough: Evaluating Agent Resilience and Considerate Participation under Accumulating Challenge](https://mlsanity.com/paper/2609.10724/) (2026)
- [Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models](https://mlsanity.com/paper/2609.13005/) (2026)
- [MintAct: A Unified Visual Agent for Digital Environments](https://mlsanity.com/paper/2609.22083/) (2026)
- [HM-ROUTER: Joint Model and Harness Routing for Agentic Systems](https://mlsanity.com/paper/2609.32213/) (2026)
- [CompoWorld: Compositional Environment Scaling for General Agents](https://mlsanity.com/paper/2609.33665/) (2026)
- [Timeline-Bench: Evaluating Agents on Realistic Video-Editing Tasks, from Raw Footage to Final Cut](https://mlsanity.com/paper/2609.35143/) (2026)
- [TCSAlgBench: Benchmarking Automated Proving for Research-Level Theoretical Computer Science](https://mlsanity.com/paper/2609.35606/) (2026)
- [Scaling Verifiable Environments for Long-horizon Work Agents](https://mlsanity.com/paper/2610.04906/) (2026)
- [A Trust Layer for Agent Evaluation](https://mlsanity.com/paper/2610.07274/) (2026)
- [AgentTime: Can Agents Estimate and Control Their Own Runtime?](https://mlsanity.com/paper/2610.09944/) (2026)
- [SciExam for ENSO: Can AI Agents Build Climate Models?](https://mlsanity.com/paper/2610.10513/) (2026)
- [Hybrid Reasoning Agents](https://mlsanity.com/paper/2610.18861/) (2026)

## Abstract (arXiv)

Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.