How Well Does Agent Development Reflect Real-World Work?
Not on the map. Why?
Abstract (arXiv)
AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 1 Mar 2026.
Builds on
52 of 67 references.
References · 52
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
- 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
- 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
38 more
- 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
- 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
- 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
- 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
- 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
- 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
- 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
- 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
- 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
- 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
- 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
- 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
- 2025Towards a Science of Scaling Agent Systemscited by 125
- 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
- 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
- 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
- 2025Inducing Programmatic Skills for Agentic Taskscited by 103
- 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 99
- 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 97
- 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
- 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 89
- 2025Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversationscited by 88
- 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
- 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
- 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 66
- 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
- 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
- 2025Levels of Autonomy for AI Agentscited by 45
- 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 43
- 2025Measuring Agents in Productioncited by 35
- 2025Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforcecited by 33
- 2025WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Taskscited by 29
- 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
- 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 18
- 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 17
- 2024VibeCheck: Discover and Quantify Qualitative Differences in Large Language Modelscited by 15
- 2025Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environmentscited by 15
- 2025The AI Agent Indexcited by 14
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 36
- 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 24
- 2026LLMs Corrupt Your Documents When You Delegatecited by 9
- 2026AlphaEval: Evaluating Agents in Productioncited by 1
- 2026BlueFin: Benchmarking LLM Agents on Financial Spreadsheetscited by 1
- 2026Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoningcited by 0
- 2026Economic Evaluations of Language Modelscited by 0
- 2026ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledgecited by 0
- 2026Developing a Roadmap to an AI-first Organization: A Case Study in Embedded Software Developmentcited by 0
- 2026GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesiscited by 0
- 2026GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis30 Sep 2026
- 2026Developing a Roadmap to an AI-first Organization: A Case Study in Embedded Software Development25 Sep 2026
- 2026ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge11 Sep 2026
- 2026Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning17 Jul 2026
- 2026Economic Evaluations of Language Models26 Jun 2026
- 2026BlueFin: Benchmarking LLM Agents on Financial Spreadsheets29 May 2026
- 2026SWE-chat: Coding Agent Interactions From Real Users in the Wild22 Apr 2026
- 2026LLMs Corrupt Your Documents When You Delegate17 Apr 2026
- 2026AlphaEval: Evaluating Agents in Production14 Apr 2026
- 2026Gym-Anything: Turn any Software into an Agent Environment7 Apr 2026