mlsanityData to 7 Oct 2026

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

arXiv 2412.14161 ↗First posted 18 Dec 2024cs.CLcited by 306

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at accelerating or even autonomously performing work-related tasks? The answer to this question has important implications both for industry looking to adopt AI into their workflows and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that the most competitive agent can complete 30% of tasks autonomously. This paints a nuanced picture on task automation with LM agents--in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. We release code, data, environment, and experiments on https://the-agent-company.com.

Key facts. Cited by 306 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 26 of its 67 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

26 of 67 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments4 Nov
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  3. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents18 Oct

Other references · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  9. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  10. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  11. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  12. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
11 more
  1. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  2. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  3. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  4. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  5. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  6. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  7. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  8. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  9. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  10. 2024Beyond Browsing: API-Based Web Agentscited by 77
  11. 2024VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Taskscited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 98; 2026: 208.

306 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2025Survey on Evaluation of LLM-based Agentscited by 190
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  6. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  7. 2025Towards a Science of Scaling Agent Systemscited by 125
  8. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  9. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 73
  10. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  11. 2026Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scalecited by 65
  12. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
38 more
  1. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  2. 2025Self-Challenging Language Model Agentscited by 56
  3. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 54
  4. 2025CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactionscited by 46
  5. 2025Levels of Autonomy for AI Agentscited by 45
  6. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 43
  7. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  8. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agentscited by 41
  9. 2026Code as Agent Harnesscited by 40
  10. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  11. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 36
  12. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 35
  13. 2026MiniMax Sparse Attentioncited by 34
  14. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
  15. 2025ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineeringcited by 30
  16. 2025BEARCUBS: A benchmark for computer-using web agentscited by 29
  17. 2025WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Taskscited by 29
  18. 2025A Rosetta Stone for AI Benchmarkscited by 29
  19. 2026WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluationcited by 29
  20. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  21. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 28
  22. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  23. 2026SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?cited by 28
  24. 2026ClawBench: Can AI Agents Complete Everyday Online Tasks?cited by 28
  25. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  26. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  27. 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systemscited by 23
  28. 2025Creativity in LLM-based Multi-Agent Systems: A Surveycited by 23
  29. 2025Large Language Model-based Data Science Agent: A Surveycited by 22
  30. 2025How malicious AI swarms can threaten democracy: The fusion of agentic AI and LLMs marks a new frontier in information warfarecited by 20
  31. 2026Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependenciescited by 20
  32. 2025Agentic AI Needs a Systems Theorycited by 19
  33. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 19
  34. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 19
  35. 2026PhysicianBench: Evaluating LLM Agents in Real-World EHR Environmentscited by 19
  36. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?cited by 19
  37. 2025SEW: Self-Evolving Agentic Workflows for Automated Code Generationcited by 18
  38. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 18

50 most cited of 306 All 306 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023AgentBench: Evaluating LLMs as Agents
  2. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
  3. 2024Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?
  4. 2026EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents
  5. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research
  6. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents
  7. 2024Towards a Realistic Long-Term Benchmark for Open-Web Research Agents
  8. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents