# OneMillion-Bench: How Far are Language Agents from Human Experts?

arXiv: https://arxiv.org/abs/2603.07980 · HTML: https://mlsanity.com/paper/2603.07980/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations](https://mlsanity.com/paper/2506.13651/) (2025)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction](https://mlsanity.com/paper/2508.11987/) (2025)
- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claims](https://mlsanity.com/paper/2506.02064/) (2025)
- [LiveTradeBench: Seeking Real-World Alpha with Large Language Models](https://mlsanity.com/paper/2511.03628/) (2025)
- [Evaluation Framework for AI Systems in "the Wild"](https://mlsanity.com/paper/2504.16778/) (2025)

## Built on by (most cited first, 13 of 13)

- [Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents](https://mlsanity.com/paper/2604.06132/) (2026)
- [Agents' Last Exam](https://mlsanity.com/paper/2606.05405/) (2026)
- [Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies](https://mlsanity.com/paper/2605.03596/) (2026)
- [JobBench: Aligning Agent Work With Human Will](https://mlsanity.com/paper/2605.26329/) (2026)
- [OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation](https://mlsanity.com/paper/2604.10866/) (2026)
- [Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields](https://mlsanity.com/paper/2606.11042/) (2026)
- [AlphaEval: Evaluating Agents in Production](https://mlsanity.com/paper/2604.12162/) (2026)
- [VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild](https://mlsanity.com/paper/2605.27882/) (2026)
- [StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows](https://mlsanity.com/paper/2608.17800/) (2026)
- [Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models](https://mlsanity.com/paper/2604.16593/) (2026)
- [Learning with a Single Rollout via Pass@k Policy Optimization](https://mlsanity.com/paper/2606.25451/) (2026)
- [xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems](https://mlsanity.com/paper/2609.07784/) (2026)
- [UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactions](https://mlsanity.com/paper/2610.11638/) (2026)

## Abstract (arXiv)

As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce OneMillion-Bench (OMB), a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focusing on expert-level problems to ensure meaningful differentiation across agents. Together, OMB provides a unified testbed for assessing agentic reliability, professional depth, and an indicator of practical readiness in domain-intensive scenarios.