mlsanityData to 14 Oct 2026

OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation

arXiv 2407.19056 ↗First posted 26 Jul 2024cs.CLcited by 70

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Office automation significantly enhances human productivity by automatically finishing routine tasks in the workflow. Beyond the basic information extraction studied in much of the prior document AI literature, the office automation research should be extended to more realistic office tasks which require to integrate various information sources in the office system and produce outputs through a series of decision-making processes. We introduce OfficeBench, one of the first office automation benchmarks for evaluating current LLM agents' capability to address office tasks in realistic office workflows. OfficeBench requires LLM agents to perform feasible long-horizon planning, proficiently switch between applications in a timely manner, and accurately ground their actions within a large combined action space, based on the contextual demands of the workflow. Applying our customized evaluation methods on each task, we find that GPT-4 Omni achieves the highest pass rate of 47.00%, demonstrating a decent performance in handling office tasks. However, this is still far below the human performance and accuracy standards required by real-world office workflows. We further observe that most issues are related to operation redundancy and hallucinations, as well as limitations in switching between multiple applications, which may provide valuable insights for developing effective agent frameworks for office automation.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 29 of its 38 references that are in the map. First posted on arXiv 26 Jul 2024.

Builds on

29 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents24 Jan

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  11. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  12. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
16 more
  1. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  2. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  3. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  4. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 188
  5. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  6. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  7. 2021LayoutReader: Pre-training of Text and Layout for Reading Order Detectioncited by 80
  8. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
  9. 2021Document AI: Benchmarks, Models and Applicationscited by 62
  10. 2020LAMBERT: Layout-Aware (Language) Modeling for information extractioncited by 53
  11. 2023LMDX: Language Model-based Document Information Extraction and Localizationcited by 37
  12. 2022VRDU: A Benchmark for Visually-rich Document Understandingcited by 35
  13. 2020DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understandingcited by 31
  14. 2022Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Frameworkcited by 10
  15. 2022MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understandingcited by 9
  16. 2024DOCMASTER: A Unified Platform for Annotation, Training, & Inference in Document Question-Answeringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 19; 2026: 46.

70 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  2. 2026Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skillscited by 187
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  4. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  5. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agentscited by 136
  6. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 57
  7. 2025Simulating Environments with Reasoning Models for Agent Trainingcited by 57
  8. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  9. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  10. 2025LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automationcited by 40
  11. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  12. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 38
38 more
  1. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  2. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 34
  3. 2024WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Modelscited by 30
  4. 2024Medchain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequencecited by 17
  5. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  6. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agentscited by 10
  7. 2025Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflowscited by 10
  8. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  9. 2026Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instabilitycited by 10
  10. 2026TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agentscited by 9
  11. 2025Memento No More: Coaching AI Agents to Master Multiple Tasks via Hints Internalizationcited by 8
  12. 2026OSExpert: Computer-Use Agents Learning Professional Skills via Explorationcited by 7
  13. 2026PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Taskscited by 7
  14. 2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agentscited by 6
  15. 2026Demystifying Agent Skills: Why They Work-Until They Don'tcited by 5
  16. 2026Agentic Tool Use in Large Language Modelscited by 4
  17. 2026AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoningcited by 4
  18. 2025DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboardscited by 3
  19. 2025UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliabilitycited by 3
  20. 2026ARM: Role-Conditioned Neuron Transplantation for Training-Free Generalist LLM Agent Mergingcited by 3
  21. 2026FileGram: Grounding Agent Personalization in File-System Behavioral Tracescited by 3
  22. 2026PREPING: Zero-Task Agent Memory Construction via Self-Practicecited by 3
  23. 2025PAACE: A Plan-Aware Automated Agent Context Engineering Frameworkcited by 2
  24. 2026OS-Marathon: Benchmarking Computer-Use Agents on Vast-Horizon, Repetitive Taskscited by 2
  25. 2026FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applicationscited by 2
  26. 2026Simulate to Generalize: Scaling Stateful Supervision for API-calling Agents using LLM World Modelscited by 2
  27. 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Groundingcited by 2
  28. 2026Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agentscited by 2
  29. 2025Learning API Functionality from In-Context Demonstrations for Tool-based Agentscited by 1
  30. 2025ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Executioncited by 1
  31. 2025Reason-Plan-ReAct: A Reasoner-Planner Supervising a ReAct Executor for Complex Enterprise Taskscited by 1
  32. 2026GNNVerifier: Graph-based Verifier for LLM Task Planningcited by 1
  33. 2026ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agentscited by 1
  34. 2026PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflowscited by 1
  35. 2026DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operationscited by 1
  36. 2026Software Engineering for and with GUI Agentcited by 1
  37. 2026DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?cited by 1
  38. 2026UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrationscited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
  2. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting
  3. 2026ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
  4. 2026Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
  5. 2025MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
  6. 2026AutomationBench
  7. 2026ParseBench: A Document Parsing Benchmark for AI Agents
  8. 2026OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning