mlsanityData to 7 Oct 2026

OneMillion-Bench: How Far are Language Agents from Human Experts?

arXiv 2603.07980 ↗First posted 9 Mar 2026cs.LG cs.AI cs.CLcited by 13

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce OneMillion-Bench (OMB), a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focusing on expert-level problems to ensure meaningful differentiation across agents. Together, OMB provides a unified testbed for assessing agentic reliability, professional depth, and an indicator of practical readiness in domain-intensive scenarios.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 40 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

21 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claims1 Jun
  2. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents20 Feb
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  4. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb

Other references · 17

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  6. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  7. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  9. 2025Humanity's Last Examcited by 467
  10. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  11. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  12. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
5 more
  1. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  2. 2025FutureX: An Advanced Live Benchmark for LLM Agents in Future Predictioncited by 55
  3. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 18
  4. 2025LiveTradeBench: Seeking Real-World Alpha with Large Language Modelscited by 13
  5. 2025Evaluation Framework for AI Systems in "the Wild"cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  2. 2026Agents' Last Examcited by 25
  3. 2026Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependenciescited by 20
  4. 2026JobBench: Aligning Agent Work With Human Willcited by 10
  5. 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulationcited by 6
  6. 2026Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fieldscited by 4
  7. 2026AlphaEval: Evaluating Agents in Productioncited by 1
  8. 2026VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wildcited by 1
  9. 2026StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflowscited by 1
  10. 2026Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Modelscited by 0
  11. 2026Learning with a Single Rollout via Monte Carlo Pass@k Criticcited by 0
  12. 2026xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problemscited by 0
1 more
  1. 2026UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
  2. 2023AgentBench: Evaluating LLMs as Agents
  3. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  4. 2025SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations
  5. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  6. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  7. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  8. 2026Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents