mlsanityData to 7 Oct 2026

How Well Does Agent Development Reflect Real-World Work?

arXiv 2603.01203 ↗First posted 1 Mar 2026cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 1 Mar 2026.

Builds on

52 of 67 references.

References · 52

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  7. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  8. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  9. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  10. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  11. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  12. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
38 more
  1. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  2. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  3. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  4. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  5. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  6. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  7. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  8. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  9. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  10. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
  11. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  12. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  13. 2025Towards a Science of Scaling Agent Systemscited by 125
  14. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  15. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  16. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  17. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  18. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 99
  19. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 97
  20. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  21. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 89
  22. 2025Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversationscited by 88
  23. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  24. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  25. 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 66
  26. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  27. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  28. 2025Levels of Autonomy for AI Agentscited by 45
  29. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 43
  30. 2025Measuring Agents in Productioncited by 35
  31. 2025Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforcecited by 33
  32. 2025WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Taskscited by 29
  33. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  34. 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 18
  35. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 17
  36. 2024VibeCheck: Discover and Quantify Qualitative Differences in Large Language Modelscited by 15
  37. 2025Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environmentscited by 15
  38. 2025The AI Agent Indexcited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.