mlsanityData to 7 Oct 2026

WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting

arXiv 2405.00823 ↗First posted 1 May 2024cs.CL cs.AI cs.MAcited by 47

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce WorkBench: a benchmark dataset for evaluating agents' ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents' errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents' ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/olly-styles/WorkBench.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 25 of its 32 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

25 of 32 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  10. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  12. 2023Large Language Modelscited by 774
13 more
  1. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  2. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  3. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  4. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  5. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  6. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  7. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  8. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  9. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  10. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  11. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  12. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  13. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 30.

47 papers.

Sort:

By area:

  1. 2024AI Agents That Mattercited by 199
  2. 2025Towards a Science of Scaling Agent Systemscited by 125
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  4. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  5. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  6. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  7. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Modelscited by 48
  8. 2025CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactionscited by 46
  9. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  10. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  11. 2025Benchmarking Deep Search over Heterogeneous Enterprise Datacited by 23
  12. 2026Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependenciescited by 20
35 more
  1. 2025Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environmentscited by 15
  2. 2026EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settingscited by 14
  3. 2026World of Workflows: A Benchmark for Bringing World Models to Enterprise Systemscited by 10
  4. 2026CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agentscited by 10
  5. 2025AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprisecited by 8
  6. 2025LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactionscited by 6
  7. 2025CRMWeaver: Building Powerful Business Agent via Agentic RL and Shared Memoriescited by 5
  8. 2026EvoMAS: Evolutionary Generation of Multi-Agent Systemscited by 4
  9. 2026SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environmentscited by 4
  10. 2026TacoMAS: Test-Time Co-Evolution of Topology and Capability in LLM-based Multi-Agent Systemscited by 4
  11. 2026MemToolAgent: Leveraging Memory for Tool Using Agents Based on Environment and User Feedbackcited by 4
  12. 2026CoAgent: Concurrency Control for Multi-Agent Systemscited by 4
  13. 2026Safeguarding LLM Agents from Misalignment through Provenance Analysiscited by 4
  14. 2025TheMCPCompany: Creating General-purpose Agents with Task-specific Toolscited by 3
  15. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  16. 2026Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agentscited by 2
  17. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Trainingcited by 2
  18. 2025ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Executioncited by 1
  19. 2026Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Callscited by 1
  20. 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Groundingcited by 1
  21. 2026GDPevo: Evaluating Agent Self-Evolution on Real Business Taskscited by 1
  22. 2026Lost in Compaction: Evaluating Side-Constraint Loss under Context Compactioncited by 1
  23. 2026From Version Conflicts to Decision Conflicts: Selective Revalidation for Long-Running AI Agentscited by 1
  24. 2024Towards Evaluating Large Language Models for Graph Query Generationcited by 0
  25. 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathycited by 0
  26. 2026Agents that Matter: Optimizing Multi-Agent LLMs via Removal-Based Attributioncited by 0
  27. 2026WorkBench Revisited: Workplace Agents Two Years Oncited by 0
  28. 2026AI Snitches Get Glitches: Towards Evading Agentic Surveillancecited by 0
  29. 2026When Do Multi-Agent Systems Help? An Information Bottleneck Perspectivecited by 0
  30. 2026What Could the Agent See at 19:05? Generating Temporal Enterprise Scenarios from Real Research and Replaying Them to Evaluate Agentscited by 0
  31. 2026DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agentscited by 0
  32. 2026OptiMAS: Automatically Optimize Multi-Agent Systemcited by 0
  33. 2026DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflowscited by 0
  34. 2026CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Basescited by 0
  35. 2026Topological Coherence for Self-evolving Multi-agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
  2. 2026ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software
  3. 2026ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
  4. 2025UpBench: A Dynamically Evolving Real-World Labor-Market Agentic Benchmark Framework Built for Human-Centric AI
  5. 2026SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries
  6. 2026JobBench: Aligning Agent Work With Human Will
  7. 2026WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
  8. 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation