JobBench: Aligning Agent Work With Human Will
Not on the map. Why?
Abstract (arXiv)
Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 17 of its 45 references that are in the map. First posted on arXiv 25 May 2026.
Builds on
17 of 45 references.
References · 17
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
5 more
- 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
- 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
- 2025Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforcecited by 33
- 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
- 2026OneMillion-Bench: How Far are Language Agents from Human Experts?cited by 13
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026Kimi K3: Open Frontier Intelligencecited by 99
- 2026RRSI: Regularized Recursive Self-Improvement of Agent Harnessescited by 3
- 2026Agent Skills Should Go Beyond Text: The Case for Visual Skillscited by 1
- 2026GDPevo: Evaluating Agent Self-Evolution on Real Business Taskscited by 1
- 2026FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverablescited by 1
- 2026Atria Dawn: The Dawn of Agentic Superintelligencecited by 0
- 2026Raven: The Harness of Harnesses for Composable Agentic Intelligencecited by 0
- 2026WorkGenesis: Building the Worlds That Teach Agents to Workcited by 0
- 2026REMORY: Learning Residual Memory for Context Compactioncited by 0
- 2026MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvementcited by 0
- 2026MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement8 Oct 2026
- 2026REMORY: Learning Residual Memory for Context Compaction8 Oct 2026
- 2026WorkGenesis: Building the Worlds That Teach Agents to Work30 Sep 2026
- 2026Raven: The Harness of Harnesses for Composable Agentic Intelligence27 Sep 2026
- 2026RRSI: Regularized Recursive Self-Improvement of Agent Harnesses21 Sep 2026
- 2026Atria Dawn: The Dawn of Agentic Superintelligence14 Sep 2026
- 2026FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverables4 Aug 2026
- 2026GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks4 Aug 2026
- 2026Kimi K3: Open Frontier Intelligence27 Jul 2026
- 2026Agent Skills Should Go Beyond Text: The Case for Visual Skills31 May 2026
Similar, not cited
Close in wording, no citation either way
- 2025UpBench: A Dynamically Evolving Real-World Labor-Market Agentic Benchmark Framework Built for Human-Centric AI
- 2026Agentic AI and Occupational Displacement: A Multi-Regional Task Exposure Analysis of Emerging Labor Market Disruption
- 2026Who Delegates to AI? Evidence from Agent Configurations in Github
- 2026The Organizational Behavior of Agentic AI: Collective Intelligence in Human-Agent Workflows
- 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting
- 2025HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
- 2025DoubleAgents: Human-Agent Alignment in a Socially Embedded Workflow
- 2026Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values